AdaCamp Portland was an amazing conference for feminist women in open source tech and culture. Not all, but many of the conference attendees are developers, system administrators, or do other technical work in open source software. I gave an informal talk meant to be an overview of some things I currently do at Mozilla. Lots of people came to the session! We all introduced ourselves going around the room.
To start off with, I showed a sample bug to talk about the process of reporting a bug, using Bugzilla, and practicing the skill of reading and understanding a bug report.
We looked first at Bug 926292.
Let’s look at the life of this bug!
This bug was reported in October 2013 for Firefox 24 by someone new to bugzilla.mozilla.org. New users have basic permissions to file and comment on bugs. For around their first 25 bugs filed or commented on, they are marked “New to Bugzilla” to anyone with more permissions on the system. This helps more experienced users to know when they’re in conversation with people who are relatively new to the system. And, bugs reported by new users are automatically entered into Bugzilla with a status of “UNCONFIRMED”.
Our bug reporter was answered the same day by a community bug triager who used the “needinfo” checkbox to ask the bug reporter more questions. A bit later, in Comment 2, I was able to confirm the bug; I marked it NEW. Community members often jump in to do this from Bug Triage bug days, from our One and Done community taskboard, or because they watch the “Firefox::Untriaged” component. (Yes . . . you too can sign up to get email from Bugzilla every time a new bug is filed!)
Francesca Ciceri is currently working on bug triage and verification with our team as part of the GNOME-OPW internship program, doing similiar work to Tiziana Selitto who was an OPW intern last year! Both their blogs have good insights into what it’s like to approach QA in a huge and somewhat chaotic system like Mozilla’s.
In our example bug, I took a guess as to which product and component to add to the bug. This is like putting the bug into the right place where developers who work in a particular area will be likely to see it, and pay attention to it. I moved it from “Firefox” to “Core” and thought it may be something to do with the CSS Object Model. Picking the right product and component is tricky. Sometimes I look for similar bugs, to see what component they’re in. Sometimes I use Bugzilla’s Browse pages to skim or search through the descriptions of components for Firefox, Core, and Toolkit. Even after doing this for a year and a half, I get it wrong. Here, a developer moved the bug to what he thought was a better component for it, Core::Layout. (Developers also sometimes guess wrong, and keep passing a bug around to each others’ components like a hot potato.)
At this point a few developers explored the bug, and went back and forth with each other and the bug reporter about whether it had been fixed or not, exactly what the bug was, whether it is a Mac issue or a Firefox issue, and how to fix it. It was resolved as a duplicate of another bug in October, but the bug reporter came back to reopen it in February 2014. The bug reporter was polite but persistent in explaining their view, giving more details of the browser behavior, trying to find the bug in the very latest developer build (Nightly), giving a test case and comparing the behavior in different browsers. A developer submitted a patch, asked for code review. Related bugs were mentioned and linked. At least two new bugs were filed.
One important thing to note is that people working on QA and development tend to move very fluidly between using various Firefox versions. One of the best things you can do to get involved with helping out is to set up all four “channels” of Firefox with the capability to run them all at once with different profiles, and to start with new, clean profiles. In fact, we need better and more up to date documentation of how to do that on different operating systems, with screenshots! Here are some links that may help you set that up:
* http://www.callum-macdonald.com/about/faq/multiple-firefox-instances/
* https://developer.mozilla.org/en-US/docs/Mozilla/Multiple_Firefox_Profiles
* https://support.mozilla.org/en-US/kb/profile-manager-create-and-remove-firefox-profiles
OK, back to bug 926292!
Since I had worked on the bug and added myself to the cc field, I got bugmail about all these changes, and more or less followed a long. I often think that the collaboration that happens in bug fixing is very beautiful, and even fairly efficient!
In comment 29 you can see that code got committed to a mercurial repository, to “inbound”. From there, it goes through automated tests and is merged by one of the “sheriffs” into another hg repository, mozilla-central, where it will go into the next build of Nightly, which at that point in April, was Firefox 31.
Comment 30 suggests uplifting the patch to versions that will soon be released, to Aurora and Beta. Release managers started to get involved, commenting and asking the developers to formally nominate the bug for uplift.
At this point in my talk I explained a little bit about the “trains”.
The versions of Firefox under development advance on a 6 week cycle, from Nightly to Aurora to Beta to the main release of Firefox. In this rapid release schedule, Firefox 31 was Nightly, so Aurora was 30, 29 was Beta, and the release version most folks use was 28. The uplift request was refused so the patch “rode the train”. That means, if you were using Firefox 31 any time after the patch was merged into mozilla-central, you will see its effect. (It would also be fixed for Firefox 32 and 33 which are currently in use as Aurora and Nightly, since 31 is currently Beta.)
Our bug was marked “FIXED” when the patch was merged into mozilla-central. You can see near the end of its comments that I tagged the bug “verifyme” to put it into the queue of bugs that need verifying for Firefox 31. Many people see that list and work on verifying bugs including community members in our Bug Verification test days. I hope the story of this particular bug is over. I don’t have the number immediately to hand but I believe that over 1000 bugs are fixed for each version of Firefox over its release cycle. We can’t verify them all, but it’s amazing what we do get done as a team!
Other tools we looked at in my talk and the ensuing discussion:
Datazilla, which tests and measures Firefox performace: https://datazilla.mozilla.org/
Mozmill, a UI automation framework for Mozilla apps including Firefox and Thunderbird: https://github.com/mozilla/mozmill
Socorro, or crash-stats, where QA and other teams keep track of crashes in Firefox and other Mozilla products: https://crash-stats.mozilla.com
The ftp directories where Firefox builds and build candidates are stored: ftp://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/
The mercurial repositories or “the tree”: http://hg.mozilla.org/
DXR, a nifty tool to search Mozilla’s code: http://dxr.mozilla.org/mozilla-central/source/
TBPL which shows the test results for every commit that’s merged into different branches https://tbpl.mozilla.org/
And a quick view into Mozilla’s Jenkins continuous integration dashboard which you can only see from our VPN, just to give an idea of the work we do when Firefox is in Beta. As a particular version of Firefox advances through rapid release, QA pays more attention to particular areas and uses different tools. We have to know a little bit about everything, be able to reproduce a user’s bug on many different possible platforms, figure out which developers may be able to fix a bug (or whose commit may have caused a regression or crash).
It was a lot to cover in an hour long talk! I wanted to pilot this informally as a test for doing a more formal talk with slides.
It represented fairly well that QA covers quite a lot of territory; it’s complicated and interesting work.