bugs – Composite

That Zarro Boogs feeling

April 20, 2016 Liz

This is my third Firefox release as release manager, and the fifth that I’ve followed closely from the beginning to the end of the release cycle. (31 and 36 as QA lead; 39, 43, and 46 as release manager.) This time I felt more than usually okay with things, even while there was a lot of change in our infrastructure and while we started triaging and following even more bugs than usual. No matter how on top of things I get, there is still chaos and things still come up at the last minute. Stuff breaks, and we never stop finding new issues!

I’m not going into all the details because that would take forever and would mostly be me complaining or blaming myself for things. Save it for the post-mortem meeting. This post is to record my feeling of accomplishment from today.

During the approximately 6 week beta cycle of Firefox development we release around 2 beta versions per week. I read through many bugs nominated as possibly important regressions, and many that need review and assessment to decide if the benefit of backporting warrants the risk of breaking something else.

During this 7 week beta cycle I have made some sort of decision about at least 480 bugs. That usually means that I’ve read many more bugs, since figuring out what’s going on in one may mean reading through its dependencies, duplicates, and see-alsos, or whatever someone randomly mentions in comment 45 of 96.

And today I got to a point I’ve never been at near the end of a beta cycle: Zarro Boogs found!

This is what Bugzilla says when you do a query and it returns 0. I think everyone likes saying (and seeing) “Zarro Boogs”. Its silliness expresses the happy feeling you get when you have burned down a giant list of bugs.

This particular query is for bugs that anyone at all has nominated for the release management team to pay attention to.

Here is the list of requests for uplift (or backporting, same thing) to the mozilla-beta repo:

Yes!! Also zarro boogs.

Since we build our release candidate a week (or a few days) from the mozilla-release repo, I check up on requests to uplift there too:

PEAK ZARRO BOOGS.

For the bugs that are unresolved and that I’m still tracking into the 46 release next week, it’s down to 4: Two fairly high volume crashes that may not be actionable yet, one minor issue in a system addon that will be resolved in a planned out-of-band upgrade, and one web compatibility issue that should be resolved soon by an external site. Really not bad!

Our overall regression tracking has a release health dashboard on displays in many Mozilla offices. Blockers, 0. Known new regressions that we are still working on and haven’t explicitly decided to wontfix: 1. (But this will be fixed by the system addon update once 46 ships.) Carryover regressions: 41; about 15 of them are actually fixed but not marked up correctly yet. The rest are known regressions we shipped with already that still aren’t fixed. Some of those are missed uplift opportunities. We will do better in the next release!

In context, I approved 196 bugs for uplift during beta, and 329 bugs for aurora. And, we fix several thousands of issues in every release during the approx. 12 week development cycle. Which ones of those should we pay the most attention to, and which of those can be backported? Release managers act as a sort of Maxwell’s Demon to let in only particular patches …

Will this grim activity level for the past 7 weeks and my current smug feeling of being on top of regression burndown translate to noticeably better “quality”… for Firefox users? That is hard to tell, but I feel hopeful that it will over time. I like the feeling of being caught up, even temporarily.

Here I am with drink in hand on a sunny afternoon, toasting all the hard working developers, QA testers, beta users, release engineers, PMs, managers and product folks who did most of the actual work to fix this stuff and get it firmly into place in this excellent, free, open source browser. Cheers!

FFUntriaged number games

September 5, 2013 Liz

As some of you may know we have over 900,000 thousand bug reports in bugzilla.mozilla.org these days. Around 120K of those are open bugs.

I keep an eye on the incoming bugs, which are still around 350-550 for any 24 hour period and peck away at those. Many people in QA and various engineering teams also keep watch over the incoming bugs so that problems are caught quickly, escalated appropriately, and stuff gets fixed and released as soon as possible.

But the incoming are just one thing among many. Lately I’ve been working on a fairly arbitrary goal. That is to bring down a specific number, the bugs filed for Firefox that are in the “Untriaged” component, where the last comment was by the person who reported the bug. I have it on my todo list as “FFUntriaged”, so I think of it that way. FFUN! (I’m sure that is unconvincing…)

Bugs on a laptop

When I started putting half an hour to an hour a day into this, the count was well over 500 bugs. Now it is closer to 400. I answer some from the front of the queue and some from the tail end, which right now goes back to February 2012. The older, 2012 bugs are mostly obsolete at this point, but I have caught a few that are still valid, and that are now categorized in a product and component that brought them to the notice of the developers who may already be working on similar issues.

Of the bugs that I end up closing, a bunch of them probably should have been support questions in the first place. I resolve them INVALID if they are reallky support questions because they aren’t and weren’t bugs in Firefox. If I can’t tell what was going on, and the reporter doesn’t answer my “needinfo” query, I can resolve the bug INCOMPLETE since there was never enough information to tell if it was really a bug or not.

A few of these long-untriaged Firefox bugs ended up in the RESOLVED WORKSFORME status, which I think of as: when the bug was reported, no one was able to reproduce it, I can’t reproduce it now, and maybe also the original reporter can’t reproduce it. Maybe it got fixed along the way. It doesn’t seem important enough to anyone to pin down exactly what fixed it. It just works now. Resolving it seems ok to me, since that doesn’t erase any history: the bug report is still findable if it comes up again.

The FFUNtriaged bugs project has been fairly satisfying just to watch the number come down over time. Pretty soon we will have it down to something reasonable, like under 30, and actually recent bugs instead of cruft from a year ago. Then I’ll pick a new little project!

And now a digression about duplicate bugs.

A few bugs from FFUNtriaged end up being marked as duplicates. I catch a few, but more often someone more experienced notices the duplicates after I do something else with them, which sends bugmail or puts a report into a new product or component. Some reports just sound like they must have been reported before. DUPing them is a good way to establish connections and direct the original bug reporter to where the action or discussion is. There is good advice in Screening duplicate bugs article on MDN.

There is not only a Most Frequently Reported Bugs list, there is also now a whole dashboard which can show most duped bugs by product. The one I have been looking at a lot recently is the most duped list for Core::Layout bugs. The product dashboard doesn’t let you limit by time though. So I still think the main Most Frequently Reported Bugs page is more useful; you can change its query to limit it by product, or view it as a regular (sortable) bug list.

In theory the better we get (collectively) at duping bugs, the more useful the lists of most-duped bugs will be. It may be a self perpetuating cycle though, to where we learn the most-duped ones, then dup more bugs to them. I have thought before it might be fruitful to hunt after (or ask someone for a lead to) closely related bugs and sort through them to see if any are obvious dupes.

Things I know about automatically as dupes are: anything involving shortcut keys. Layout complains about tables and images. Anything to do with bookmarks. All those are worth a search and a quick scan of a list of bugs with similar words in the summary and then a bit of digging!

Sometimes people are a bit upset that “their” report gets
duped to an already existing bug. I don’t have enough experience (after 8 months triaging) to really have a sense in the patterns of what gets fixed and why. But when a bug is duped to an older one, the people who get bugmail on that older bug are going to get a poke of some kind, so at least that brings the issue to possible attention. And over time, it may affect how teams or engineers set priorities or figure out what to fix or escalate. So I think it it likely useful.

FFUntriage number games

September 5, 2013 Liz

As some of you may know we have over 900,000 thousand bug reports in bugzilla.mozilla.org these days. Around 120K of those are open bugs.

I keep an eye on the incoming bugs, which are still around 350-550 for any 24 hour period and peck away at those. Many people in QA and various engineering teams also keep watch over the incoming bugs so that problems are caught quickly, escalated appropriately, and stuff gets fixed and released as soon as possible.

But the incoming are just one thing among many. Lately I’ve been working on a fairly arbitrary goal. That is to bring down a specific number, the bugs filed for Firefox that are in the “Untriaged” component, where the last comment was by the person who reported the bug. I have it on my todo list as “FFUntriaged”, so I think of it that way. FFUN!

Bugs on a laptop

When I started putting half an hour to an hour a day into this, the count was well over 500 bugs. Now it is closer to 400. I answer some from the front of the queue and some from the tail end, which right now goes back to February 2012. The older, 2012 bugs are mostly obsolete at this point, but I have caught a few that are still valid, and that are now categorized in a product and component that brought them to the notice of the developers who may already be working on similar issues.

Of the bugs that I end up closing, a bunch of them probably should have been support questions in the first place. I resolve them INVALID if they are reallky support questions because they aren’t and weren’t bugs in Firefox. If I can’t tell what was going on, and the reporter doesn’t answer my “needinfo” query, I can resolve the bug INCOMPLETE since there was never enough information to tell if it was really a bug or not.

A few of these long-untriaged Firefox bugs ended up in the RESOLVED WORKSFORME status, which I think of as: when the bug was reported, no one was able to reproduce it, I can’t reproduce it now, and maybe also the original reporter can’t reproduce it. Maybe it got fixed along the way. It doesn’t seem important enough to anyone to pin down exactly what fixed it. It just works now. Resolving it seems ok to me, since that doesn’t erase any history: the bug report is still findable if it comes up again.

The FFUNtriaged bugs project has been fairly satisfying just to watch the number come down over time. Pretty soon we will have it down to something reasonable, like under 30, and actually recent bugs instead of cruft from a year ago. Then I’ll pick a new little project!

And now a digression about duplicate bugs.

A few bugs from FFUNtriaged end up being marked as duplicates. I catch a few, but more often someone more experienced notices the duplicates after I do something else with them, which sends bugmail or puts a report into a new product or component. Some reports just sound like they must have been reported before. DUPing them is a good way to establish connections and direct the original bug reporter to where the action or discussion is. There is good advice in Screening duplicate bugs article on MDN.

There is not only a Most Frequently Reported Bugs list, there is also now a whole dashboard which can show most duped bugs by product. The one I have been looking at a lot recently is the most duped list for Core::Layout bugs. The product dashboard doesn’t let you limit by time though. So I still think the main Most Frequently Reported Bugs page is more useful; you can change its query to limit it by product, or view it as a regular (sortable) bug list.

In theory the better we get (collectively) at duping bugs, the more useful the lists of most-duped bugs will be. It may be a self perpetuating cycle though, to where we learn the most-duped ones, then dup more bugs to them. I have thought before it might be fruitful to hunt after (or ask someone for a lead to) closely related bugs and sort through them to see if any are obvious dupes.

Things I know about automatically as dupes are: anything involving shortcut keys. Layout complains about tables and images. Anything to do with bookmarks. All those are worth a search and a quick scan of a list of bugs with similar words in the summary and then a bit of digging!

Sometimes people are a bit upset that “their” report gets
duped to an already existing bug. I don’t have enough experience (after 8 months triaging) to really have a sense in the patterns of what gets fixed and why. But when a bug is duped to an older one, the people who get bugmail on that older bug are going to get a poke of some kind, so at least that brings the issue to possible attention. And over time, it may affect how teams or engineers set priorities or figure out what to fix or escalate. So I think it it likely useful.

File a bug: the missing manual, now with unicorns

April 9, 2013 Liz

At countless conference talks I have heard standard advice on “how to get involved in an open source project”. It goes something like this:

Step 1. File a bug!
Step 2. Submit a patch! (repeat steps 1 and 2 for a while)
Step 3. Now you are ready to write some new features and stuff! Fly and be free!

I always thought that was interesting because it is an attempt to reassure people that you don’t have to leap into immediate coding. Just file a bug, that is the first step. This results in people coming into projects and wondering vaguely how to find bugs.

Part of what I want to do as bugmaster for Mozilla is to put in another step — look at the bugs already reported, since there are a squizillion of them, and see what you can do to improve the meta information of those bugs.

Today on Planet Mozilla I noticed some really good advice from Jason Smith on how to find bugs: WebRTC testing: Try out conversat.io and file bugs. It is really sensible and practical. Jason’s blog has a few posts like this that advise focus in a particular area, like WebRTC or Desktop web apps, by incorporating use of those tools in your daily life. Our “get involved in open source” sequence now looks more like this,

Step 1. Find some feature that could use testing.
Step 2. Figure out how to use it regularly.
Step 3. Use it.
Step 4. Encounter behavior you think is a bug.
Step 5. FILE A BUG (BUT HOW?)

There is a lot of background knowledge necessary to actually file a bug in the complicated system that is bugzilla.mozilla.org (or BMO).

So let’s take the WebRTC example. Say you’ve followed Jason’s advice, used conversat.io for a while, and found A BUG. Jason helpfully provides a link directly into Bugzilla to the enter_bug form, with the Product and Component pre-filled out to be for a bug with WebRTC (the component) and Core (the product.) Bugs in BMO are categorized according to Product, like Firefox Desktop, Firefox for Android, FirefoxOS, Thunderbird, etc. “Core” is the product for the code that is shared between many other products. If you were looking to file a bug with WebRTC from scratch you would probably not know which product to file it under, though you have to choose one. So it’s great that Jason gives a link to the right product and component!

But wait. There is so much more background, or context, to understand. You don’t have to, but it is very good to understand it!

First of all you have to have a bugzilla account for the link to work properly. If you do that, you will be a new bugzilla user, and some of the bug entry forms will look different to you — you’ll be automatically directed to a “guided bug entry form” which is broken into several steps, rather than the form that shows you the whole “advanced view” with several dozen fields and dropdown menus.

Second, how about looking at the list of all the components in the Core product. This is a good part of the background knowledge – a little piece of the map or geography of Bugzilla. As you can see, there are a lot of components that are part of Core. Scrolling down to the WebRTC bit, you can see that there are several sub categories: WebRTC, WebRTC:Audio/Video, WebRTC:Networking, and WebRTC:Signaling. Click on the general WebRTC component to see a list of open WebRTC bugs. This is where your geography lesson gets useful.

Right now there are 113 open bugs for WebRTC. You might look over them simply to get an idea of what kinds of bugs others have found. More about this later!

The important thing at this moment is: Is your bug already reported? Depending on how many bugs there are in this list, and your levels of interest and patience, you might want to either quickly read through the summaries (the title of the bug) or do a search down the page for words that might be in the bug you’re about to file.

If you find something in that list that you think is your bug, take a closer look at it. Read it and the comments and try to understand what they’re talking about. If it is the same as your bug, you may want to leave a comment describing what you saw happen.

But let’s say you don’t find your bug on that list. Aha! Here we use the File a bug link from the original blog post, link to file a bug with WebRTC. If you are me, or a user of Bugzilla who has made more than 25 edits or comments to bugs, you will see the advanced bug entry form, which is huge (you can see it from space) and looks like this:

Enter bug webrtc advanced

If you are a relatively new user of Bugzilla, you’ll come first to a guided entry form, broken into several screens. (At any point, you can switch to the advanced entry form with a link at the bottom of the page, if clicking through multiple screens annoys you.) The first screen for guided bug entry would normally be for selecting the product and component. Since you have these already chosen in Jason’s convenient link, you start on Screen 2, where you can enter the summary for your bug. In this case I am reporting a sort of unicorn invasion:

Enterbug webrtc screen2

After you enter a summary you will see a list pop up underneath the summary field, of bugs that may be similar. It is worth reading through those to see if anyone else has reported unicorns invading their conversat.io screens in Firefox. In this case, definitely not.

Enterbug webrtc screen2 list

Since no one else has reported this bug, I click the “My issue is not listed” button, and advance to screen 3, which suggests how I can describe my actions or steps to reproduce the issue, exactly what happened that I think is a bug, and what I think should have happened instead.

Enter bug screen3 unicorn jpg

Great, we have filed a bug! Back to that list of “how to contribute to an open source project”:

Step 1. Find some feature that could use testing.
Step 2. Figure out how to use it regularly.
Step 3. Use it.
Step 4. Encounter behavior you think is a bug.
Step 4.5 Make a bugzilla.mozilla.org account.
Step 4.6 Confirm it with the email confirmation.
Step 4.7 Log in to bugzilla.mozilla.org.
Step 5. FILE A BUG (which we now know how to do!!!)

But wait, there’s more — or there can be if you want to get your bearings on that map of Bugzilla. Take a look back at the list of all the general open WebRTC bugs. What can we understand from this list?

As I look over the current list it is pretty mysterious. From the language in the summaries, I would guess some of the bugs are notes by the development team for their own to-do list, and some of them look like bugs discovered by general users of the software. My first impulse is to sort the page a few different ways to see what that reveals. Sorting by Status shows the UNCONFIRMED bugs at the top and the NEW bugs listed just underneath. There is one titled getUserMedia freeze all system that isn’t confirmed yet and may be a good example.

Here is an interesting one, No event when remote peer disappeared. My view of this bug is going to be different from yours, if you are new to Bugzilla, because I have more magic powers, ie, canconfirm and editbugs permissions, as well as some extra admin stuff. There is a lot of stuff on the page. It’s extremely “busy” with text! You have to learn to skim it and parse it mentally so that you can see and notice the stuff that is important to you at the moment. Here is what this bug looks like for a new Bugzilla user.

Example webrtc bug2

What I can see from reading this bug is that there is at least one person actively looking at newly filed bugs, triaging them, and working on the project. And in fact as I click around and read more of the bugs for WebRTC I can see Jason is actively engaged with most of them, which is not a big surprise since he is blogging about the subject.

Jason’s blog looks like a quite useful place to find out areas that welcome testers and bug-finders. You can also look at the QMO quality assurance and testing pages which explain how to run nightly builds and participate in QA’s bug test and triage days.

My bigger point here, though, is that to start contributing to an open source project, aside from reporting one-off bugs you accidentally discover, it is super helpful to learn the landscape of the project. Adopt the project and poke around to learn about it. If you are reporting a bug, look at the other bugs. Look at who is commenting and working on those bugs. Join their IRC channel and read their wiki pages and (usually more formal than the wiki pages…) developer documentation. Or simply google the project to learn more about what’s going on with it. In this case I found that conversat.io is quite new and was developed partly to show off what WebRTC is and what it can do.

It was really apparent, from my morning of poking around, how much transparency there is at Mozilla, and the amazing technical depths you can get to from half a day’s reading and bug surfing. As a society we really have yet to realize the implications for education and educational institutions. It is a major cultural shift I am happy to be part of.

Screen reader and accessibility bug day

March 11, 2013 Liz

Tomorrow Mozilla is hosting a screen reader and general accessibility bug day.

Len Burns and I have invited screen reader users of Firefox and other Mozilla products to join us in sorting through existing accessibility bugs. Some folks from the Mozilla Accessibility team will be on hand to talk with us.

I’m pretty sure we will also collect some new bugs along the way!

I hope that people will make new connections, and that we can attract a wider accessibility bugmaster team to do ongoing work with Mozilla’s existing developers and a11y experts.

The screen reader landscape for web access is fairly complicated. For example, here are the Firefox Gecko docs for Windows accessibility vendors which explain the relationship between the DOM tree and Microsoft’s accessibility API. Common screen reader software includes NVDA, Window-Eyes, JAWS, and Orca.

Orca2 sm

If you would like a quick overview of common web access issues, look through Aaron Leventhal’s presentation. I like it because he includes some political dimensions and context for accessibility.

So far, the “bugmaster” bug days have been combined with QA’s efforts. I’m hoping to also hold focused bug days like this one, in cooperation with various teams across Mozilla. As we gather more templates for bug managing and triage, I hope we’ll coalesce bugmaster teams with expertise in particular areas. And we can repeat topic-focused bug days periodically.

If you are interested in web accessibility or if you use a screen reader, please drop by on Tuesday and say hello in the #accessibility channel on irc.mozilla.org.

You can also add the name you go by, your irc handle, your contact infomation, or anything else you use to identify yourself, on the wiki page for the bug day under Participants: https://wiki.mozilla.org/Bugmasters/Bug_days/a11y#Participants .

The goals of the screen reader bug day are to improve everyone’s experience of Firefox and other Mozilla tools. We would like for everyone to be able to access the web smoothly. Through collecting more information on accessibility bugs, we hope to connect committed technical users with accessibility developers, and make our community better and more powerful. Our bugmaster work should help to make developers’ work easier. That way they can spend more time fixing stuff.

If you have any questions or feedback, please feel free to email me at lhenry@mozilla.org.

In setting up this event I tried to make sure that the tools we are using are as accessible as possible. Etherpad, which Mozilla teams often use for bug triage events, is not useable for screen readers. The wiki.mozilla.org pages seem readable though editing may be more of a challenge. IRC feels like our best bet for good communication. I also went through about 100 screen-reader-related bugs and emailed the bug reporters and commenters to send them invitations to participate.

Len is particularly interested in developing a plan for Thunderbird and screen reader vendors. If you share this interest I recommend joining the mailing list tb-planning.

Here is what Len has to say on Thunderbird accessibility:

It is complicated, because the issues are really between the two major screen reader vendors and Mozilla. Meaning, that the solution would need to be a cooperative one. Because the screen reader vendors perceive, right or wrong, that little more than security bugs are being updated in Thunderbird, they do not seem terribly motivated. I am not quite sure where to take it.

Unless I could convince the vendors that solving these issues are worthy of their time, I am a bit stuck.

I would definitely be willing to raise the specifics with both vendors if I could give them some reason to encourage a belief that there is a mutual interest in improving things. When I have raised several with GW Micro I hear things like: This has been filed for over a year with no response, and the like.

Those of us using screen readers are currently in quite a pickle regarding email. The choices are quite limited. A lot of us have been using Thunderbird for some time, but when things reach a point where you are spending too much time trying to send a simple email, something must give.

I can also tell you that what finally tipped me were problems between the composition screen, and other open apps on-screen. If I were going to compose an email in TBird right now, I would have to be sure that Skype was minimized, MirandaIM was minimized, etc. If I did not, I would be likely to encounter a range of strange behaviors in the edit window such as being unable to read back the text I am writing, inaability to use my backspace, format distortion, and more.

What has been slowing me down on these issues was a lack of knowing avenues of pursuit. The challenge will be convincing vendors that investing time is a benefit. My position is that I am not sure, but, we have a good chance of catalyzing and contributing to change and possible strengthened relationships.

Len has been a professional system administrator, coder and web accessibility consultant since the Internet was a kinder and gentler place. He makes his living these days free lancing as a web accessibility consultant for colleges and universities and coding the back-end glue of web sites for small to mid-sized businesses.

Thanks to Len for his insights on web dev, email, and access in the last couple of weeks as well as his outreach efforts to talk with vendors, software users, and developers!