Captain's log, stardate 2016-10-26

Once again I resolve to write about my work at Mozilla as a Firefox release manager. It’s hard to do, because even the smallest thing could fill LONG paragraphs with many links! Since I keep daily notes on what I work on, let me try translating that in brief. When moved, maybe I’ll go into depth.

This week we are coming into the home stretch of a 7 week release cycle. “My” release for right now is Firefox 49, which was released on I’m still juggling problems and responses and triaging for that every day. In a week and a half, we were scheduled to release Firefox 50. Today after some discussion we pushed back that schedule by a week.

Meanwhile, I am also helping a new release manager (Gerry) to go through tracked bugs, new regressions, top crash reports, and uplift requests for Aurora/Developer Edition (Firefox 51). I’m going through uplift requests for Firefox 45.5.0esr, the extended support release. There’s still more – I paid some attention to our “update orphaning” project to bring users stuck on older versions of Firefox forward to the current, better and safer versions.

As usual, this means talking to developers and managers across pretty much all the teams at Mozilla, so it is never boring. Our goal is to get fixes and improvements as fast as possible while making sure, as best we can, that those fixes aren’t causing worse problems. We also have the interesting challenges of working across many time zones around the world.

Firefox stuffed animal  installed

Today I had a brief 1:1 meeting with my manager and went to the Firefox Product cross-functional meeting. I always find useful as it brings together many teams. There was a long Firefox team all hands discussion and then I skipped going to another hour long triage meeting with the platform/firefox engineering managers. Whew! We had a lively discussion over the last couple of days about a performance regression (Bug 1304434). The issues are complicated to sort out. Everyone involved is super smart and the discussions have a collegiate quality. No one is “yelling at each other”, while we regularly challenge each other’s assumptions and are free to disagree – usually in public view on a mailing list or in our bug tracker. This is part of why I really love Mozilla. While we can get a bit heated and stressed, overall, the culture is good. YMMV of course.

By that time (11am) I had been working since 7:30am, setting many queries in bugs, and on IRC, and in emails into motion and made a lot of small but oddly difficult decisions. Often this meant exercising my wontfix powers on bugs — deferring uplift (aka “backport” ) to 50, 51, or leaving a fix in 52 to ride the trains to release some time next year. As I’m feeling pretty good I headed out to have lunch and work from a cafe downtown (Mazarine – the turkey salad sandwich was very good!)

This afternoon I’m focusing on ESR 45, and Aurora 51, doing a bit more bug triage. There are a couple of ESR uplifts stressing me out — seriously, I was having kittens over these patches — but now that we have an extra week until we release, it feels like a better position for asking for a 2nd code review, a bit more time for QA, and so on.

Heading out soon for drinks with friends across the street from this cafe, and then to the Internet Archive’s 20th anniversary party. Yay, Internet Archive!

That Zarro Boogs feeling

This is my third Firefox release as release manager, and the fifth that I’ve followed closely from the beginning to the end of the release cycle. (31 and 36 as QA lead; 39, 43, and 46 as release manager.) This time I felt more than usually okay with things, even while there was a lot of change in our infrastructure and while we started triaging and following even more bugs than usual. No matter how on top of things I get, there is still chaos and things still come up at the last minute. Stuff breaks, and we never stop finding new issues!

I’m not going into all the details because that would take forever and would mostly be me complaining or blaming myself for things. Save it for the post-mortem meeting. This post is to record my feeling of accomplishment from today.

During the approximately 6 week beta cycle of Firefox development we release around 2 beta versions per week. I read through many bugs nominated as possibly important regressions, and many that need review and assessment to decide if the benefit of backporting warrants the risk of breaking something else.

During this 7 week beta cycle I have made some sort of decision about at least 480 bugs. That usually means that I’ve read many more bugs, since figuring out what’s going on in one may mean reading through its dependencies, duplicates, and see-alsos, or whatever someone randomly mentions in comment 45 of 96.

And today I got to a point I’ve never been at near the end of a beta cycle: Zarro Boogs found!

list of zero bugs

This is what Bugzilla says when you do a query and it returns 0. I think everyone likes saying (and seeing) “Zarro Boogs”. Its silliness expresses the happy feeling you get when you have burned down a giant list of bugs.

This particular query is for bugs that anyone at all has nominated for the release management team to pay attention to.

Here is the list of requests for uplift (or backporting, same thing) to the mozilla-beta repo:

more zero pending requests

Yes!! Also zarro boogs.

Since we build our release candidate a week (or a few days) from the mozilla-release repo, I check up on requests to uplift there too:

list of zero pending requests

PEAK ZARRO BOOGS.

For the bugs that are unresolved and that I’m still tracking into the 46 release next week, it’s down to 4: Two fairly high volume crashes that may not be actionable yet, one minor issue in a system addon that will be resolved in a planned out-of-band upgrade, and one web compatibility issue that should be resolved soon by an external site. Really not bad!

Our overall regression tracking has a release health dashboard on displays in many Mozilla offices. Blockers, 0. Known new regressions that we are still working on and haven’t explicitly decided to wontfix: 1. (But this will be fixed by the system addon update once 46 ships.) Carryover regressions: 41; about 15 of them are actually fixed but not marked up correctly yet. The rest are known regressions we shipped with already that still aren’t fixed. Some of those are missed uplift opportunities. We will do better in the next release!

In context, I approved 196 bugs for uplift during beta, and 329 bugs for aurora. And, we fix several thousands of issues in every release during the approx. 12 week development cycle. Which ones of those should we pay the most attention to, and which of those can be backported? Release managers act as a sort of Maxwell’s Demon to let in only particular patches …

Will this grim activity level for the past 7 weeks and my current smug feeling of being on top of regression burndown translate to noticeably better “quality”… for Firefox users? That is hard to tell, but I feel hopeful that it will over time. I like the feeling of being caught up, even temporarily.

liz in sunglasses with a drink in hand

Here I am with drink in hand on a sunny afternoon, toasting all the hard working developers, QA testers, beta users, release engineers, PMs, managers and product folks who did most of the actual work to fix this stuff and get it firmly into place in this excellent, free, open source browser. Cheers!

A useful Bugzilla trick

At the beginning of February I changed teams within Mozilla and am now working as a release manager. It follows naturally from a lot of the work I’ve already been doing at Mozilla and I’m excited to join the team working with Lukas, Lawrence, and Sylvestre!

I just learned a cool trick for dealing with several bugzilla.mozilla.org bugs at once, on MacOS X.

1) Install Bugzilla Services.

2) Add a keyboard shortcut as Alex Keybl describes in the blog post above. (I am using Control-Command-B)

3) Install the Tree Style Tab addon.

Now, from any text, whether in email, a desktop text file, or anywhere in the browser, I can highlight a bunch of text and bug number will be parsed out of the text. For example, from an email this morning:

Bug 1137050 - Startup up Crash - patch should land soon, potentially risky
David Major seems to think it is risky for the release.

Besides that, we are going to take:
Bug 1137469 - Loop exception - patch waiting for review
Bug 1136855 - print preferences - patch approved
Bug 1137141 - Fx account + hello - patch waiting for review
Bug 1136300 - Hello + share buttons - Mike  De Boer will work on a patch today

And maybe a fix for the ANY query (bug 1093983) if we have one...

I highlighted the entire email and hit the “open in bugzilla” keystroke. This resulted in a Bugzilla list view for the 6 bugs mentioned in the email.

Bugzilla list view example

With BugzillaJS installed, I have an extra option at the bottom of the page, “Open All in Tabs”, so if I wanted to triage these bugs, I can open them all at once. The tabs show up in my sidebar, indented from their parent tab. This is handy if I want to collapse this group of tabs, or close the parent tab and all its children at once (The original list view of these 6 bugs, and each of its individual tabs.) Tree Style Tab is my new favorite thing!

Tree style tabs bugzilla

In this case, after I had read each bug from this morning and closed the tabs, my coworker Sylvestre asked me to make sure I cc-ed myself into all of them to keep an eye on them later today and over the weekend so that when fixes are checked in, I can approve them for release.

Here I did not want to open up every bug in its own tab but instead went for “Change Several Bugs at Once” which is also at the bottom of the page.

Bugzilla batch edit

This batch edit view of bugs is a bit scarily powerful since it will result in bugmail to many people for each bug’s changes. When you need it, it’s a great feature. I added myself to the cc: field all in one swoop instead of having to click each tab open, click around several times in each bug to add myself and save and close the tab again.

It was a busy day yesterday at work but I had a nice time working from the office rather than at home. Here is the view from the SF Mozilla office 7th floor deck where I was working and eating cake in the sun. Cannot complain about life, really.
Mozilla bridge view

Automated test harnesses for Firefox, zooming out a bit

In December I was going through some of the steps to be able to change, fix, and interpret the results of a small subset of the gazillion automated tests that Mozilla runs on Firefox builds. My last post arrived at the point of being able to run mochitests locally on my laptop on the latest Firefox code. Over the holidays I dug further into the underlying situation and broaded my perspective. I wanted to make sure that I didn’t end up grubbing away at something that no one cared about or wasn’t missing the big picture.

My question was, how can I, or QA in general, understand where Firefox is with e10s (Electrolysis, the code name for the multiprocess Firefox project). Can I answer the question, are we ready to have e10s turned on by default in Firefox — for the Developer Edition (formerly known as Aurora), for Beta, and for a new release? What criteria are we judging by? Complicated. And given those things, how can we help move the project along and ensure good quality; Firefox that works as well as or, we hope, better than, Firefox with e10s not enabled? My coworker Juan and I boiled it down to basically, stability (lowering the crash rate) and automated test coverage. To answer my bigger questions about how to improve automated test coverage I had to kind of zoom out, and look from another angle. For Firefox developers a lot of what I am about to describe is basic knowledge. It seems worth explaining, since it took me significant time to figure out.

First of all let’s look at treeherder. Treeherder is kind of the new TBPL, which is the old new Tinderbox. It lets us monitor the current state of the code repositories and the tests that run against them. It is a window into Mozilla’s continuous integration setup. A battery of tests are poised to run against Firefox builds on many different platforms. Have a look!

Current view of mozilla-central on treeherder

Treeherder2

Edward Tufte would have a cow. Luckily, this is not for Tufte to enjoy. And I love it. You can just keep digging around in there, and it will keep telling you things. What a weird, complicated gold mine.

Digression! When I first started working for Mozilla I went to the Automation and Tools team work week where they all came up with Treeherder. We wanted to name it something about Ents, because it is about the Tree(s) of the code repos. I explained the whole thing to my son, I think from the work week, which awesomely was in London. He was 11 or 12 at the time and he suggested the name “Yggdrazilla” keeping the -zilla theme and in reference to Yggdrasil, the World-Tree from Norse mythology. My son is pretty awesome. We had to reject that name because we can’t have more -zilla names and also no one would be able to spell Yggdrasil. Alas! So, anyway, treeherder.

The left side of the screen describes the latest batch of commits that were merged into mozilla-central. (The “tree” of code that is used to build Nightly.) On the right, there are a lot of operating systems/platforms listed. Linux opt (optimized version of Firefox for release) is at the top, along with a string of letters and numbers which we hope are green. Those letter and numbers represent batches of tests. You can hover over them to see a description. The tests marked M (1 2 3 etc) are mochitests, bc1 is mochitest-browser-chrome, dt are developer tools tests, and so on. For linux-opt you can see that there are some batches of tests with e10s in the name. We need the mochitest-plain tests to run on Firefox if it has e10s not enabled, or enabled. So the tests are duplicated, possibly changed to work under e10s, and renamed. We have M(1 2 3 . . .) tests, and also M-e10s(1 2 3 . . . ). Tests that are green are all passing. Orange means they aren’t passing. I am not quite sure what red (busted) means (bustage in the tree! red alert!) but let’s just worry about orange. (If you want to read more about the war on orange and what all this means, read Let’s have more green trees from Vaibhav’s blog. )

I kept asking, in order to figure out what needed doing that I could usefully do within the scope of As Soon As Possible, “So, what controls what tests are in which buckets? How do I know how many there are and what they are? Where are they in the codebase? How can I turn them on and off in a way that doesn’t break everything, or breaks it productively?” Good questions. Therefore the answers are long.

There are many other branches of the code other than mozilla-central. Holly is a branch where the builds for all the platforms have e10s enabled. (Many of these repos or branches or twigs or whatever, are named after different kinds of tree.) The tests are the standard set of tests, not particularly tweaked to allow for e10s. We can see what is succeeded and failing on treeherder’s view of holly. A lot of tests are orange on holly! Have a look at holly by clicking through on the link above. Here is a picture of the current state of holly.

Treeherder holly1

If you click a batch of tests where there are some failures — an orange one — then a new panel will open up in treeherder! I will pick a juicy looking one. Right now, for MacOS 10.6 opt, M(2) is orange. Clicking it gives me a ton of info. Scrolling down a bit in the bottom left panel tells me this:

mochitest-plain-chunked 164546/12/14136

The first number is how many tests ran. Scary. Really? 164546 tests ran? Kind of. This is counting assertions, in other words, “is” statements from SimpleTest. The first number lists how many assertions passed. The second number is for assertion failures and the third is for “todo” statements.

The batches of tests running on holly are all running against an e10s build of Firefox. Anything that’s consistently green on holly, we can move over to mozilla-central by making some changes in mozharness. I asked a few people how to do this, and Jim Matthies helpfully pointed me at a past example in Bug 1061014. I figured I could make a stab at adding some of the newly passing tests in Bug 1122901.

As I looked at how to do this, I realized I needed commit access level 2 so I filed a bug to ask for that. And, I also tried merging mozilla-central to holly. That was ridiculously exciting though I hadn’t fixed anything yet. I was just bringing the branch up to date. On my first try doing this, I immediately got a ping on IRC from one of the sheriffs (who do merges and watch the state of the “tree” or code repository) asking me why I had done something irritating and wrong. It took us a bit to figure out what had happened. When I set up mercurial, the setup process and docs told me to install a bunch of Mozilla specific hg extensions. So, I had an extension set up to post to bugzilla every time I updated something. Since I was merging several weeks of one branch to another this touched hundreds of bugs, sending bugmail to untold numbers of people. Mercifully, Bugzilla cut this off after some limit was reached. It was so embarrassing I could feel myself turning beet red as I thought of how many people just saw my mistake and wondered what the heck I was doing. And yet just had to forge onwards, fix my config file, and try it again. Super nicely, Clint Talbert told me that the first time he tried pushing some change he broke all branches of every product and had no idea what had happened. Little did he know I would blog about his sad story to make myself feel less silly….. That was years ago and I think Mozilla was still using cvs at that point! I merged mozilla-central to holly again, did not break anything this time, and watched the tests run and gradually appear across the screen. Very cool.

I also ended up realizing that the changes to mozharness to turn these batches of tests on again were not super obvious and to figure it out I needed to read a 2000-line configuration file which has somewhat byzantine logic. I’m not judging it, it is clearly something that has grown organically over time and someone else probably in release engineering is an expert on it and can tweak it casually to do whatever is needed.

Back to our story. For a batch of tests on holly that have failures and thus are showing up on treeherder as orange, it should be possible to go through the logs for the failing tests, figure out how to turn them off with some skip-if statements, filing bugs for each skipped failing test. Then, keep doing that till a batch of tests is green and it is ready to be moved over.

That seems like a reasonable plan for improving the automated test landscape, which should help developers to know that their code works in Firefox whether e10s is enabled or not. In effect, having the tests should mean that many problems are prevented from ever becoming bugs. The effect of this test coverage is hard to measure. How do you prove something didn’t happen? Perhaps by looking at which e10s tests fail on pushes to the try server. Another issue here is that there are quite a lot of tests that have been around for many years. It is hard too know how many of them are useful, whether there are a lot of redundant tests, in short whether there is a lot of cruft and there probably is. With a bit more experience in the code and fixing and writing tests it would be easier to judge the usefulness of these tests.

I can also see that, despite this taking a while to figure out (and to even begin to explain) it is a good entry point to contributing to Firefox. It has more or less finite boundaries. If you can follow what I just described in this post and my last post, and you can read and follow a little python and javascript, then you can do this. And, if you were to go through many of the tests, over time you would end up understanding more about how the codebase is structured.

As usual when I dive into anything technical at Mozilla, I think it’s pretty cool that most of this work happens in the open. It is a great body of data for academics to study, it’s an example of how this work actually happens for anyone interested in the field, and it’s something that anyone can contribute to if they have the time and interest to put in some effort.

This post seems very plain with only some screenshots of Treeherder for illustration. Here, have a photo of me making friends with a chicken.

Liz with chicken

Thoughts on working at Mozilla and the Firefox release process

Some thoughts on my life at Mozilla as I head into our company-wide work week here in Portland! My first year at Mozilla I spent managing the huge volume of bugs, updating docs on how to triage incoming bugs and helping out with Bugzilla itself. For my second year I’ve been more closely tied into the Firefox release process, and I switched from being on the A-Team (Automation and Tools) to desktop Firefox QA, following Firefox 31 and then Firefox 34 through their beginning to release.

picard saying if it's not in bugzilla it doesn't exist

I also spent countless hours fooling around with Socorro, filing bugs on the highest volume crash signatures for Firefox, and then updating the bugs and verifying fixes, especially for startup crashes or crashes associated with “my” releases.

Over this process I’ve really enjoyed working with everyone I’ve met online and in person. The constant change of Mozilla environments and the somewhat anarchic processes are completely fascinating. Though sometimes unnerving. I have spent these two years becoming more of a generalist. I have to talk with end users, developers on every Mozilla engineering team, project or product managers if they exist, other QA teams, my own Firefox/Desktop QA team, the people maintaining all the tools that all of us use, release management, and release engineering. Much of the actual QA has been done by our Romanian team so I have coordinated a lot with them and hope to meet them all some day. When I need to dig into a bug or feature and figure out how it works, it means poking around in documentation or in the code or talking with people, then documenting whatever it is.

This is a job (and a workplace) suited for people who can cope with rapid change and shifting ground. Without a very specific focus, expertise is difficult. People complain a lot about this. But I kind of like it! And I’m constantly impressed by how well our processes work and what we produce. There are specific people who I think of as total rock stars of knowing things. Long time experts like bz or dbaron. Or dmajor who does amazing crash investigations. I am in their secret fan club. If there were bugzilla “likes” I would be liking up a storm on there! It’s really amazing how well the various engineering teams work together. As we scramble around to improve things and as we nitpick I want to keep that firmly in mind. And, as we are all a bit burned out from dealing with the issues from the 33 Firefox release and many point releases, then the 10th anniversary release, then last minute scrambling to incorporate surprise/secret changes to Firefox 34 because of our new deal with Yahoo.

I was in a position to see some of the hard work of the execs, product and design people, and engineers as well as going through some rounds of iterating very quickly with them these last 2 weeks (and seeing gavin and lmandel keep track of that rapid pace) Then seeing just part of what the release team needed to do, and knowing that from their perspective they also could see what IT and infrastructure teams had to do in response. So many ripples of teamwork and people thinking things through, discussing, building an ever-changing consensus reality.

I find that for every moment I feel low for not knowing something, or making a mistake (in public no less) there are more moments when I know something no one else knows in a particular context or am able to add something productive because I’m bringing a generalist perspective that includes my years of background as a developer.

As an author and editor I am reminded of what it takes to edit an anthology. My usual role is as a general editor with a vision: putting out a call, inviting people to contribute, working back and forth with authors, tracking all the things necessary (quite a lot to track, more than you’d imagine!) and shifting back and forth from the details of different versions of a story or author bios, to how the different pieces of the book fit together. And in that equation you also, if you’re lucky, have a brilliant and meticulous copy editor. At Aqueduct Press I worked twice with Kath Wilham. I would have gone 10 rounds of nitpick and Kath would still find things wrong. My personal feeling for my past year’s work is that my job as test plan lead on a release has been 75% “editor” and 25% copy editor.

If I had a choice I would always go with the editor and “glue” work rather than the final gatekeeper work. That last moment of signing off on any of the stages of a release freak me out. I’m not enough of a nitpicker and in fact, have zero background in QA — instead, 20 years as a lone wolf (or nearly so) developer, a noisy, somewhat half-assed one who has never *had* QA to work with much less *doing* QA. It’s been extremely interesting, and it has also been part of my goal for working in big teams. The other thing I note from this stressful last 6 weeks is that, consistent with other situations for me, in a temporary intense “crisis” mode my skills shine out. Like with the shifting-every-30-minutes disaster relief landscape, where I had great tactical and logistical skills and ended up as a good leader. My problem is that I can’t sustain that level of awareness, productivity, or activeness, both physically and mentally, for all that long. I go till I drop, crash & burn. The trick is knowing that’s going to happen, communicating it beforehand, and having other people to back you up. The real trick which I hope to improve at is knowing my limits and not crashing and burning at all. On the other hand, I like knowing that in a hard situation I have this ability to tap into. It just isn’t something I can expect to do all the time, and isn’t sustainable. One thing I miss about my “old” life is building actual tools. These last 2 years have been times of building human and institutional infrastructure for me much more than making something more obvious and concrete. I would like to write another book and to build a useful open source tool of some sort. Either for Mozilla, for an anti-harassment tool suite, or for Ingress…. 🙂 Alternatively I have a long term idea about open source hardware project for mobility scooters and powerchairs. That may never happen, so at the least, I’m resolving to write up my outline of what should happen, in case someone else has the energy and time to do it.

In 2013 and 2014 I mentored three interns for the GNOME-OPW project at Mozilla. Thanks so much to Tiziana Selitto, Maja Frydrychowicz, and Francesca Ciceri for being awesome to work with! As well as the entire GNOME-OPW team at Mozilla and beyond. I spoke about the OPW projects this summer at Open Source Bridge and look forward to more work as a mentor and guide in the future.

Meanwhile! I started a nonprofit in my “spare time”! It’s Double Union, a feminist, women-only hacker and maker space in San Francisco and it has around 150 members. I’m so proud of everything that DU has become. And I continue my work on the advisory boards of two other feminist organizations, Ada Initiative and GimpGirl as well as work in the backchannels of Geekfeminism.org. I wrote several articles for Model View Culture this year and advised WisCon on security issues and threat modeling. I read hundreds of books, mostly science fiction, fantasy, and history. I followed the awesome work my partner Danny does with EFF, as well. These things are not just an important part of my life, they also make my work at Mozilla more valuable because I am bringing perspectives from these communities to the table in all my work.

The other day I thought of another analogy for my last year’s work at Mozilla that made me laugh pretty hard. I feel personally like the messy “glue” Perl scripts it used to be my job to write to connect tools and data. Part of that coding landscape was because we didn’t have very good practices or design patterns but part of it I see as inevitable in our field. We need human judgement and routing and communication to make complicated systems work, as well as good processes.

I think for 2015 I will be working more closely with the e10s team as well as keeping on with crash analysis and keeping an eye on the release train.

Mad respect and appreciation to everyone at Mozilla!!

Firefox launch party liz larissa in fox ears

How to test new features in Firefox 34 Aurora

If you’re a fan of free and open source software and would like to contribute to Firefox, join me for some Firefox feature testing!

There are some nifty features under development right now for Firefox 34 including translation in the browser, making voice or video calls (a feature called “Hello” or “Loop”), debugging information for web developers in the Dev Tools Inspector, and recent improvements to HTML5 gaming.

I’ve written step by step instructions on these
ways to test Firefox 34. If you would like to see what it’s like to improve a popular open source project, trying out these tasks is a good introduction.

Aurora

First, Install the Aurora version of Firefox. It is best to set it up to use multiple profiles. That ensures you don’t use your everyday version of Firefox for testing, so you won’t risk losing your usual profile information. It also makes it easy to restart Firefox with a new, clean profile with all the default settings, very useful for testing. Sometimes I realize I’m running 5 different versions of Firefox at once!

To test “Hello”, try making some voice or video calls from Firefox Aurora. You will need a friend to test with. Or, use two computers that you control. This is a good task to try while joining our chat channels, #qa or #testday on irc.mozilla.org; ask if anyone there wants to test Hello with you. The goal here is mostly to find and report new bugs.

If you test the translation infobar in Aurora you may find some new bugs. This is a fun feature to test. I like trying it on Wikipedia in many different languages, and also looking at newspapers!

If you’re a web developer, you may use Developer Tools in Firefox. I’m asking Aurora users to go through some unconfirmed bug reports, to help improve the Developer Tools Inspector.

If you like games you can test HTML5 web-based games in Firefox Aurora. This helps us improve Firefox and also helps the independent game developers. We have a list of demo games so you can play them, report glitches, and feel like a virtuous open source citizen all at once. Along the way you have opportunities to learn some interesting stuff about how graphics on the web can work (or not work).

Monster madness

These testing tasks are all set up in One and Done, Mozilla QA’s site to start people along the path to joining our open source community. This site was developed with a lot of community contribution including the design and concept by long-time community member Parul and a lot of code by two interns this summer, Pankaj and Maja.

Testing gives a great view into the development process for people who may not (yet) be programmers. I especially love how transparent Mozilla’s process can be. Anyone can report a bug, visible to the entire world in bugzilla.mozilla.org. There are many people watching that incoming stream of bug reports, confirming them and routing them to developer teams, sometimes tagging them as good first bugs for new contributors. Developers who may or may not be Mozilla employees show up in the bugs, like magic . . . if you think of bugmail notifications as magic . . .

It is amazing to see this very public and somewhat anarchic collaboration process at work. Of course, it can also be extremely satisfying to see a bug you discovered and reported, your pet bug, finally get fixed.

Mozilla's bug reporting, QA, and release processes

AdaCamp Portland was an amazing conference for feminist women in open source tech and culture. Not all, but many of the conference attendees are developers, system administrators, or do other technical work in open source software. I gave an informal talk meant to be an overview of some things I currently do at Mozilla. Lots of people came to the session! We all introduced ourselves going around the room.

To start off with, I showed a sample bug to talk about the process of reporting a bug, using Bugzilla, and practicing the skill of reading and understanding a bug report.

We looked first at Bug 926292.

Bugzilla couple

Let’s look at the life of this bug!

This bug was reported in October 2013 for Firefox 24 by someone new to bugzilla.mozilla.org. New users have basic permissions to file and comment on bugs. For around their first 25 bugs filed or commented on, they are marked “New to Bugzilla” to anyone with more permissions on the system. This helps more experienced users to know when they’re in conversation with people who are relatively new to the system. And, bugs reported by new users are automatically entered into Bugzilla with a status of “UNCONFIRMED”.

Our bug reporter was answered the same day by a community bug triager who used the “needinfo” checkbox to ask the bug reporter more questions. A bit later, in Comment 2, I was able to confirm the bug; I marked it NEW. Community members often jump in to do this from Bug Triage bug days, from our One and Done community taskboard, or because they watch the “Firefox::Untriaged” component. (Yes . . . you too can sign up to get email from Bugzilla every time a new bug is filed!)

Francesca Ciceri is currently working on bug triage and verification with our team as part of the GNOME-OPW internship program, doing similiar work to Tiziana Selitto who was an OPW intern last year! Both their blogs have good insights into what it’s like to approach QA in a huge and somewhat chaotic system like Mozilla’s.

In our example bug, I took a guess as to which product and component to add to the bug. This is like putting the bug into the right place where developers who work in a particular area will be likely to see it, and pay attention to it. I moved it from “Firefox” to “Core” and thought it may be something to do with the CSS Object Model. Picking the right product and component is tricky. Sometimes I look for similar bugs, to see what component they’re in. Sometimes I use Bugzilla’s Browse pages to skim or search through the descriptions of components for Firefox, Core, and Toolkit. Even after doing this for a year and a half, I get it wrong. Here, a developer moved the bug to what he thought was a better component for it, Core::Layout. (Developers also sometimes guess wrong, and keep passing a bug around to each others’ components like a hot potato.)

At this point a few developers explored the bug, and went back and forth with each other and the bug reporter about whether it had been fixed or not, exactly what the bug was, whether it is a Mac issue or a Firefox issue, and how to fix it. It was resolved as a duplicate of another bug in October, but the bug reporter came back to reopen it in February 2014. The bug reporter was polite but persistent in explaining their view, giving more details of the browser behavior, trying to find the bug in the very latest developer build (Nightly), giving a test case and comparing the behavior in different browsers. A developer submitted a patch, asked for code review. Related bugs were mentioned and linked. At least two new bugs were filed.

One important thing to note is that people working on QA and development tend to move very fluidly between using various Firefox versions. One of the best things you can do to get involved with helping out is to set up all four “channels” of Firefox with the capability to run them all at once with different profiles, and to start with new, clean profiles. In fact, we need better and more up to date documentation of how to do that on different operating systems, with screenshots! Here are some links that may help you set that up:
* http://www.callum-macdonald.com/about/faq/multiple-firefox-instances/
* https://developer.mozilla.org/en-US/docs/Mozilla/Multiple_Firefox_Profiles
* https://support.mozilla.org/en-US/kb/profile-manager-create-and-remove-firefox-profiles

OK, back to bug 926292!

Since I had worked on the bug and added myself to the cc field, I got bugmail about all these changes, and more or less followed a long. I often think that the collaboration that happens in bug fixing is very beautiful, and even fairly efficient!

In comment 29 you can see that code got committed to a mercurial repository, to “inbound”. From there, it goes through automated tests and is merged by one of the “sheriffs” into another hg repository, mozilla-central, where it will go into the next build of Nightly, which at that point in April, was Firefox 31.

Comment 30 suggests uplifting the patch to versions that will soon be released, to Aurora and Beta. Release managers started to get involved, commenting and asking the developers to formally nominate the bug for uplift.

At this point in my talk I explained a little bit about the “trains”.

Trains

The versions of Firefox under development advance on a 6 week cycle, from Nightly to Aurora to Beta to the main release of Firefox. In this rapid release schedule, Firefox 31 was Nightly, so Aurora was 30, 29 was Beta, and the release version most folks use was 28. The uplift request was refused so the patch “rode the train”. That means, if you were using Firefox 31 any time after the patch was merged into mozilla-central, you will see its effect. (It would also be fixed for Firefox 32 and 33 which are currently in use as Aurora and Nightly, since 31 is currently Beta.)

Our bug was marked “FIXED” when the patch was merged into mozilla-central. You can see near the end of its comments that I tagged the bug “verifyme” to put it into the queue of bugs that need verifying for Firefox 31. Many people see that list and work on verifying bugs including community members in our Bug Verification test days. I hope the story of this particular bug is over. I don’t have the number immediately to hand but I believe that over 1000 bugs are fixed for each version of Firefox over its release cycle. We can’t verify them all, but it’s amazing what we do get done as a team!

Other tools we looked at in my talk and the ensuing discussion:

Datazilla, which tests and measures Firefox performace: https://datazilla.mozilla.org/

Mozmill, a UI automation framework for Mozilla apps including Firefox and Thunderbird: https://github.com/mozilla/mozmill

Socorro, or crash-stats, where QA and other teams keep track of crashes in Firefox and other Mozilla products: https://crash-stats.mozilla.com

The ftp directories where Firefox builds and build candidates are stored: ftp://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/

The mercurial repositories or “the tree”: http://hg.mozilla.org/

DXR, a nifty tool to search Mozilla’s code: http://dxr.mozilla.org/mozilla-central/source/

TBPL which shows the test results for every commit that’s merged into different branches https://tbpl.mozilla.org/

And a quick view into Mozilla’s Jenkins continuous integration dashboard which you can only see from our VPN, just to give an idea of the work we do when Firefox is in Beta. As a particular version of Firefox advances through rapid release, QA pays more attention to particular areas and uses different tools. We have to know a little bit about everything, be able to reproduce a user’s bug on many different possible platforms, figure out which developers may be able to fix a bug (or whose commit may have caused a regression or crash).

It was a lot to cover in an hour long talk! I wanted to pilot this informally as a test for doing a more formal talk with slides.

It represented fairly well that QA covers quite a lot of territory; it’s complicated and interesting work.

FirefoxOS phone beta testing

I got my FirefoxOS phone and am doing some testing on FirefoxOS 1.1 beta. I’m about to switch over to using it as my primary phone, but so far I’ve just used its apps and web connection. It seems very responsive and fast.

green and orange FirefoxOS phones

My friend Tiziana and I compared phones. Hers is the orange one, a ZTE phone running OPEN_US_DEV_FFOS_V1.0.0B1. Mine is green and is from Alcatel, running FirefoxOS 1.1.0.0-prerelease. I definitely notice the difference in the new version of the OS in that it is more responsive and I feel more sure in control of the touchscreen. Typing works well. I miss having something to swipe across the letters so I hope an open source version of type-by-swipe is something in the works.

I have installed a few art and drawing apps as well as games. The only game I’ve tried so far is Little Alchemy, which was very buggy in earlier versions this spring, but works beautifully now.

Email setup worked great, browsing is good, I’ve been taking photos and emailing them. I’ve used the Wikipedia app to look things up, and the “I’m thinking of…” search feature to listen to music on the bus. The Music app comes pre-loaded with some albums but I haven’t figured out a good way to load my own music onto the phone. On the other hand I haven’t needed to since “I’m thinking of…” gets me decently fast connections to YouTube and other music sources.

One nice touch with this was that when I bring up my “I’m thinking of” Janelle Monae search the wallpaper background of my search screen changes to a giant aweseome photo of Monae in a tuxedo and I am offered the things on my phone to do with the search on top, then under a hairline divider, searches on Monae in many other categories like free music, Grooveshark, Google, Twitter, her official website, tickets, Amazon, and so on. Here’s what it looks like:

firefoxOS thinking of search screen

That’s different from my usual mental model of “search”. The result is interesting and, I think, powerful and useful. I’ll be exploring it over the next few days at the Mozilla Summit. And of course reporting bugs, but for now, I just wanted to share that the phone not only quite usable as a smartphone, it’s exciting to explore.

File a bug: the missing manual, now with unicorns

At countless conference talks I have heard standard advice on “how to get involved in an open source project”. It goes something like this:

Step 1. File a bug!
Step 2. Submit a patch! (repeat steps 1 and 2 for a while)
Step 3. Now you are ready to write some new features and stuff! Fly and be free!

I always thought that was interesting because it is an attempt to reassure people that you don’t have to leap into immediate coding. Just file a bug, that is the first step. This results in people coming into projects and wondering vaguely how to find bugs.

Part of what I want to do as bugmaster for Mozilla is to put in another step — look at the bugs already reported, since there are a squizillion of them, and see what you can do to improve the meta information of those bugs.

Today on Planet Mozilla I noticed some really good advice from Jason Smith on how to find bugs: WebRTC testing: Try out conversat.io and file bugs. It is really sensible and practical. Jason’s blog has a few posts like this that advise focus in a particular area, like WebRTC or Desktop web apps, by incorporating use of those tools in your daily life. Our “get involved in open source” sequence now looks more like this,

Step 1. Find some feature that could use testing.
Step 2. Figure out how to use it regularly.
Step 3. Use it.
Step 4. Encounter behavior you think is a bug.
Step 5. FILE A BUG (BUT HOW?)

There is a lot of background knowledge necessary to actually file a bug in the complicated system that is bugzilla.mozilla.org (or BMO).

So let’s take the WebRTC example. Say you’ve followed Jason’s advice, used conversat.io for a while, and found A BUG. Jason helpfully provides a link directly into Bugzilla to the enter_bug form, with the Product and Component pre-filled out to be for a bug with WebRTC (the component) and Core (the product.) Bugs in BMO are categorized according to Product, like Firefox Desktop, Firefox for Android, FirefoxOS, Thunderbird, etc. “Core” is the product for the code that is shared between many other products. If you were looking to file a bug with WebRTC from scratch you would probably not know which product to file it under, though you have to choose one. So it’s great that Jason gives a link to the right product and component!

But wait. There is so much more background, or context, to understand. You don’t have to, but it is very good to understand it!

First of all you have to have a bugzilla account for the link to work properly. If you do that, you will be a new bugzilla user, and some of the bug entry forms will look different to you — you’ll be automatically directed to a “guided bug entry form” which is broken into several steps, rather than the form that shows you the whole “advanced view” with several dozen fields and dropdown menus.

Second, how about looking at the list of all the components in the Core product. This is a good part of the background knowledge – a little piece of the map or geography of Bugzilla. As you can see, there are a lot of components that are part of Core. Scrolling down to the WebRTC bit, you can see that there are several sub categories: WebRTC, WebRTC:Audio/Video, WebRTC:Networking, and WebRTC:Signaling. Click on the general WebRTC component to see a list of open WebRTC bugs. This is where your geography lesson gets useful.

Right now there are 113 open bugs for WebRTC. You might look over them simply to get an idea of what kinds of bugs others have found. More about this later!

The important thing at this moment is: Is your bug already reported? Depending on how many bugs there are in this list, and your levels of interest and patience, you might want to either quickly read through the summaries (the title of the bug) or do a search down the page for words that might be in the bug you’re about to file.

If you find something in that list that you think is your bug, take a closer look at it. Read it and the comments and try to understand what they’re talking about. If it is the same as your bug, you may want to leave a comment describing what you saw happen.

But let’s say you don’t find your bug on that list. Aha! Here we use the File a bug link from the original blog post, link to file a bug with WebRTC. If you are me, or a user of Bugzilla who has made more than 25 edits or comments to bugs, you will see the advanced bug entry form, which is huge (you can see it from space) and looks like this:

Enter bug webrtc advanced

If you are a relatively new user of Bugzilla, you’ll come first to a guided entry form, broken into several screens. (At any point, you can switch to the advanced entry form with a link at the bottom of the page, if clicking through multiple screens annoys you.) The first screen for guided bug entry would normally be for selecting the product and component. Since you have these already chosen in Jason’s convenient link, you start on Screen 2, where you can enter the summary for your bug. In this case I am reporting a sort of unicorn invasion:

Enterbug webrtc screen2

After you enter a summary you will see a list pop up underneath the summary field, of bugs that may be similar. It is worth reading through those to see if anyone else has reported unicorns invading their conversat.io screens in Firefox. In this case, definitely not.

Enterbug webrtc screen2 list

Since no one else has reported this bug, I click the “My issue is not listed” button, and advance to screen 3, which suggests how I can describe my actions or steps to reproduce the issue, exactly what happened that I think is a bug, and what I think should have happened instead.

Enter bug screen3 unicorn jpg

Great, we have filed a bug! Back to that list of “how to contribute to an open source project”:

Step 1. Find some feature that could use testing.
Step 2. Figure out how to use it regularly.
Step 3. Use it.
Step 4. Encounter behavior you think is a bug.
Step 4.5 Make a bugzilla.mozilla.org account.
Step 4.6 Confirm it with the email confirmation.
Step 4.7 Log in to bugzilla.mozilla.org.
Step 5. FILE A BUG (which we now know how to do!!!)

But wait, there’s more — or there can be if you want to get your bearings on that map of Bugzilla. Take a look back at the list of all the general open WebRTC bugs. What can we understand from this list?

As I look over the current list it is pretty mysterious. From the language in the summaries, I would guess some of the bugs are notes by the development team for their own to-do list, and some of them look like bugs discovered by general users of the software. My first impulse is to sort the page a few different ways to see what that reveals. Sorting by Status shows the UNCONFIRMED bugs at the top and the NEW bugs listed just underneath. There is one titled getUserMedia freeze all system that isn’t confirmed yet and may be a good example.

Here is an interesting one, No event when remote peer disappeared. My view of this bug is going to be different from yours, if you are new to Bugzilla, because I have more magic powers, ie, canconfirm and editbugs permissions, as well as some extra admin stuff. There is a lot of stuff on the page. It’s extremely “busy” with text! You have to learn to skim it and parse it mentally so that you can see and notice the stuff that is important to you at the moment. Here is what this bug looks like for a new Bugzilla user.

Example webrtc bug2

What I can see from reading this bug is that there is at least one person actively looking at newly filed bugs, triaging them, and working on the project. And in fact as I click around and read more of the bugs for WebRTC I can see Jason is actively engaged with most of them, which is not a big surprise since he is blogging about the subject.

Jason’s blog looks like a quite useful place to find out areas that welcome testers and bug-finders. You can also look at the QMO quality assurance and testing pages which explain how to run nightly builds and participate in QA’s bug test and triage days.

My bigger point here, though, is that to start contributing to an open source project, aside from reporting one-off bugs you accidentally discover, it is super helpful to learn the landscape of the project. Adopt the project and poke around to learn about it. If you are reporting a bug, look at the other bugs. Look at who is commenting and working on those bugs. Join their IRC channel and read their wiki pages and (usually more formal than the wiki pages…) developer documentation. Or simply google the project to learn more about what’s going on with it. In this case I found that conversat.io is quite new and was developed partly to show off what WebRTC is and what it can do.

It was really apparent, from my morning of poking around, how much transparency there is at Mozilla, and the amazing technical depths you can get to from half a day’s reading and bug surfing. As a society we really have yet to realize the implications for education and educational institutions. It is a major cultural shift I am happy to be part of.

Shorter posts with more worklogs and book reviews

While I love to go on at length and be thorough sometimes it’s been stopping me from recording interesting stuff lately. I’ll be at a conference and take great notes, which years ago I would have posted unedited. Now I tend to procrastinate posting about something “until I can do justice to it” which often results in “never”. Have I posted about Kiwicon? NO! Argh. Fuck that, I need to just post.

So I’m resolving to write more frequently about smaller topics. They may not turn into comprehensive book reviews but at least there will be something here.

mozilla roof

At work I am organizing a Bugzilla bug day and preparing to go to Toronto next week for a community building work week.

Not-at-work, lots of people are rumbling about wanting another feminist hackers meeting and a hackability wheelchair/access device hack day. I have Noisebridge stuff going on and AdaCamp is coming up in June. I forgot to actually sign up for WisCon panels but in theory am going to WisCon. There is a lot of “event to-do list” stuff here!

Notable books I read in the last few days: The Brontës Went to Woolworth’s by Rachel Ferguson, which was fantastic; The Diary of Elizabeth Pepys by Dale Spender, which I adored but which was very depressing as you can imagine if you have read Pepys; and Japanese Inn (my boring-book for bedtime) by Oliver Statler, which functioned perfectly as a boring-book and which was good but very colonialish and patronizing in the way you might expect from a book from 1960 and which if you are not trying to fall asleep at night would just make you wonder why you are bothering and realize it would be better to read some actual work of Japanese history or a primary source by one of the people referred to. Though I did enjoy reading Isabella Bird’s travels.

I am feeling more energy lately and less pain, which I attribute to my 2 months of Enbrel injections and perhaps also Tramadol, which is great as an occasional painkiller.

Here is a photo of the fabulous glistening Minecraft block cake I made for Milo’s 13th birthday party (which was at the musee mecanique again)

Minecraft cake

There really need to be square cupcake pans (well, cubical) Maybe there already are! Then it would be easy to make little Minecraft block cakes and frost them all different colors and build a hilarious structure which could be easily (if stickily) disassembled.