work – Composite

Bad tests sink ships, or something

November 3, 2016 Liz

As a connoisseur of the caution sign, I really enjoyed these gorgeous, strange, slightly disturbing safety posters from WWII-era Britain. There are stylized warnings about driving at night, about putting an eye out with that thing, about what happens if you are very imprudent with boxes, giant boards, and ladders, and then a jolly man who is ominously having way too much fun with a sort of compressed air power gun.

http://www.slate.com/blogs/the_vault/2016/09/16/modernist_british_safety_posters_from_the_middle_of_the_twentieth_century.html

Here is my new version of the “I didn’t protect my eyes” poster specially for Firefox developers,

I Didnt push to try

And another very silly new poster for developers – based on World War II’s finest modernist art –

Make sure it has tests

Really, I do love caution signs. They’re so earnest. They try very hard to be persuasive! Here are some from my Flickr collection of signs. This beautiful, detailed image of someone falling through rotten boards on a pier and then drowning helplessly is from the long pier at Aquatic Park in San Francisco.

Caution dont fall off the pier

This last one is from an old artillery testing range in Essex that you have to drive through before you can get to the incredibly dangerous and fascinating place, The Broomway. The Broomway is a submerged and mostly unmarked 6 mile long path through ocean quicksand, only usable at low tide. Did I mention the unexploded ordnance, the rapidly advancing tide, and the frequent heavy fog? Also, did I mention there is a tour that takes you over The Broomway on a giant tractor?! Sign me up!

Do not approach or touch any object or debris it may explode and kill you

Captain's log, stardate 2016-10-26

October 26, 2016 Liz

Once again I resolve to write about my work at Mozilla as a Firefox release manager. It’s hard to do, because even the smallest thing could fill LONG paragraphs with many links! Since I keep daily notes on what I work on, let me try translating that in brief. When moved, maybe I’ll go into depth.

This week we are coming into the home stretch of a 7 week release cycle. “My” release for right now is Firefox 49, which was released on I’m still juggling problems and responses and triaging for that every day. In a week and a half, we were scheduled to release Firefox 50. Today after some discussion we pushed back that schedule by a week.

Meanwhile, I am also helping a new release manager (Gerry) to go through tracked bugs, new regressions, top crash reports, and uplift requests for Aurora/Developer Edition (Firefox 51). I’m going through uplift requests for Firefox 45.5.0esr, the extended support release. There’s still more – I paid some attention to our “update orphaning” project to bring users stuck on older versions of Firefox forward to the current, better and safer versions.

As usual, this means talking to developers and managers across pretty much all the teams at Mozilla, so it is never boring. Our goal is to get fixes and improvements as fast as possible while making sure, as best we can, that those fixes aren’t causing worse problems. We also have the interesting challenges of working across many time zones around the world.

Firefox stuffed animal installed

Today I had a brief 1:1 meeting with my manager and went to the Firefox Product cross-functional meeting. I always find useful as it brings together many teams. There was a long Firefox team all hands discussion and then I skipped going to another hour long triage meeting with the platform/firefox engineering managers. Whew! We had a lively discussion over the last couple of days about a performance regression (Bug 1304434). The issues are complicated to sort out. Everyone involved is super smart and the discussions have a collegiate quality. No one is “yelling at each other”, while we regularly challenge each other’s assumptions and are free to disagree – usually in public view on a mailing list or in our bug tracker. This is part of why I really love Mozilla. While we can get a bit heated and stressed, overall, the culture is good. YMMV of course.

By that time (11am) I had been working since 7:30am, setting many queries in bugs, and on IRC, and in emails into motion and made a lot of small but oddly difficult decisions. Often this meant exercising my wontfix powers on bugs — deferring uplift (aka “backport” ) to 50, 51, or leaving a fix in 52 to ride the trains to release some time next year. As I’m feeling pretty good I headed out to have lunch and work from a cafe downtown (Mazarine – the turkey salad sandwich was very good!)

This afternoon I’m focusing on ESR 45, and Aurora 51, doing a bit more bug triage. There are a couple of ESR uplifts stressing me out — seriously, I was having kittens over these patches — but now that we have an extra week until we release, it feels like a better position for asking for a 2nd code review, a bit more time for QA, and so on.

Heading out soon for drinks with friends across the street from this cafe, and then to the Internet Archive’s 20th anniversary party. Yay, Internet Archive!

That Zarro Boogs feeling

April 20, 2016 Liz

This is my third Firefox release as release manager, and the fifth that I’ve followed closely from the beginning to the end of the release cycle. (31 and 36 as QA lead; 39, 43, and 46 as release manager.) This time I felt more than usually okay with things, even while there was a lot of change in our infrastructure and while we started triaging and following even more bugs than usual. No matter how on top of things I get, there is still chaos and things still come up at the last minute. Stuff breaks, and we never stop finding new issues!

I’m not going into all the details because that would take forever and would mostly be me complaining or blaming myself for things. Save it for the post-mortem meeting. This post is to record my feeling of accomplishment from today.

During the approximately 6 week beta cycle of Firefox development we release around 2 beta versions per week. I read through many bugs nominated as possibly important regressions, and many that need review and assessment to decide if the benefit of backporting warrants the risk of breaking something else.

During this 7 week beta cycle I have made some sort of decision about at least 480 bugs. That usually means that I’ve read many more bugs, since figuring out what’s going on in one may mean reading through its dependencies, duplicates, and see-alsos, or whatever someone randomly mentions in comment 45 of 96.

And today I got to a point I’ve never been at near the end of a beta cycle: Zarro Boogs found!

This is what Bugzilla says when you do a query and it returns 0. I think everyone likes saying (and seeing) “Zarro Boogs”. Its silliness expresses the happy feeling you get when you have burned down a giant list of bugs.

This particular query is for bugs that anyone at all has nominated for the release management team to pay attention to.

Here is the list of requests for uplift (or backporting, same thing) to the mozilla-beta repo:

Yes!! Also zarro boogs.

Since we build our release candidate a week (or a few days) from the mozilla-release repo, I check up on requests to uplift there too:

PEAK ZARRO BOOGS.

For the bugs that are unresolved and that I’m still tracking into the 46 release next week, it’s down to 4: Two fairly high volume crashes that may not be actionable yet, one minor issue in a system addon that will be resolved in a planned out-of-band upgrade, and one web compatibility issue that should be resolved soon by an external site. Really not bad!

Our overall regression tracking has a release health dashboard on displays in many Mozilla offices. Blockers, 0. Known new regressions that we are still working on and haven’t explicitly decided to wontfix: 1. (But this will be fixed by the system addon update once 46 ships.) Carryover regressions: 41; about 15 of them are actually fixed but not marked up correctly yet. The rest are known regressions we shipped with already that still aren’t fixed. Some of those are missed uplift opportunities. We will do better in the next release!

In context, I approved 196 bugs for uplift during beta, and 329 bugs for aurora. And, we fix several thousands of issues in every release during the approx. 12 week development cycle. Which ones of those should we pay the most attention to, and which of those can be backported? Release managers act as a sort of Maxwell’s Demon to let in only particular patches …

Will this grim activity level for the past 7 weeks and my current smug feeling of being on top of regression burndown translate to noticeably better “quality”… for Firefox users? That is hard to tell, but I feel hopeful that it will over time. I like the feeling of being caught up, even temporarily.

Here I am with drink in hand on a sunny afternoon, toasting all the hard working developers, QA testers, beta users, release engineers, PMs, managers and product folks who did most of the actual work to fix this stuff and get it firmly into place in this excellent, free, open source browser. Cheers!

A useful Bugzilla trick

February 27, 2015 Liz

At the beginning of February I changed teams within Mozilla and am now working as a release manager. It follows naturally from a lot of the work I’ve already been doing at Mozilla and I’m excited to join the team working with Lukas, Lawrence, and Sylvestre!

I just learned a cool trick for dealing with several bugzilla.mozilla.org bugs at once, on MacOS X.

1) Install Bugzilla Services.

2) Add a keyboard shortcut as Alex Keybl describes in the blog post above. (I am using Control-Command-B)

3) Install the Tree Style Tab addon.

Now, from any text, whether in email, a desktop text file, or anywhere in the browser, I can highlight a bunch of text and bug number will be parsed out of the text. For example, from an email this morning:

Bug 1137050 - Startup up Crash - patch should land soon, potentially risky
David Major seems to think it is risky for the release.

Besides that, we are going to take:
Bug 1137469 - Loop exception - patch waiting for review
Bug 1136855 - print preferences - patch approved
Bug 1137141 - Fx account + hello - patch waiting for review
Bug 1136300 - Hello + share buttons - Mike  De Boer will work on a patch today

And maybe a fix for the ANY query (bug 1093983) if we have one...

I highlighted the entire email and hit the “open in bugzilla” keystroke. This resulted in a Bugzilla list view for the 6 bugs mentioned in the email.

Bugzilla list view example

With BugzillaJS installed, I have an extra option at the bottom of the page, “Open All in Tabs”, so if I wanted to triage these bugs, I can open them all at once. The tabs show up in my sidebar, indented from their parent tab. This is handy if I want to collapse this group of tabs, or close the parent tab and all its children at once (The original list view of these 6 bugs, and each of its individual tabs.) Tree Style Tab is my new favorite thing!

Tree style tabs bugzilla

In this case, after I had read each bug from this morning and closed the tabs, my coworker Sylvestre asked me to make sure I cc-ed myself into all of them to keep an eye on them later today and over the weekend so that when fixes are checked in, I can approve them for release.

Here I did not want to open up every bug in its own tab but instead went for “Change Several Bugs at Once” which is also at the bottom of the page.

Bugzilla batch edit

This batch edit view of bugs is a bit scarily powerful since it will result in bugmail to many people for each bug’s changes. When you need it, it’s a great feature. I added myself to the cc: field all in one swoop instead of having to click each tab open, click around several times in each bug to add myself and save and close the tab again.

It was a busy day yesterday at work but I had a nice time working from the office rather than at home. Here is the view from the SF Mozilla office 7th floor deck where I was working and eating cake in the sun. Cannot complain about life, really.
Mozilla bridge view

Automated test harnesses for Firefox, zooming out a bit

February 7, 2015 Liz

In December I was going through some of the steps to be able to change, fix, and interpret the results of a small subset of the gazillion automated tests that Mozilla runs on Firefox builds. My last post arrived at the point of being able to run mochitests locally on my laptop on the latest Firefox code. Over the holidays I dug further into the underlying situation and broaded my perspective. I wanted to make sure that I didn’t end up grubbing away at something that no one cared about or wasn’t missing the big picture.

My question was, how can I, or QA in general, understand where Firefox is with e10s (Electrolysis, the code name for the multiprocess Firefox project). Can I answer the question, are we ready to have e10s turned on by default in Firefox — for the Developer Edition (formerly known as Aurora), for Beta, and for a new release? What criteria are we judging by? Complicated. And given those things, how can we help move the project along and ensure good quality; Firefox that works as well as or, we hope, better than, Firefox with e10s not enabled? My coworker Juan and I boiled it down to basically, stability (lowering the crash rate) and automated test coverage. To answer my bigger questions about how to improve automated test coverage I had to kind of zoom out, and look from another angle. For Firefox developers a lot of what I am about to describe is basic knowledge. It seems worth explaining, since it took me significant time to figure out.

First of all let’s look at treeherder. Treeherder is kind of the new TBPL, which is the old new Tinderbox. It lets us monitor the current state of the code repositories and the tests that run against them. It is a window into Mozilla’s continuous integration setup. A battery of tests are poised to run against Firefox builds on many different platforms. Have a look!

Current view of mozilla-central on treeherder

Treeherder2

Edward Tufte would have a cow. Luckily, this is not for Tufte to enjoy. And I love it. You can just keep digging around in there, and it will keep telling you things. What a weird, complicated gold mine.

Digression! When I first started working for Mozilla I went to the Automation and Tools team work week where they all came up with Treeherder. We wanted to name it something about Ents, because it is about the Tree(s) of the code repos. I explained the whole thing to my son, I think from the work week, which awesomely was in London. He was 11 or 12 at the time and he suggested the name “Yggdrazilla” keeping the -zilla theme and in reference to Yggdrasil, the World-Tree from Norse mythology. My son is pretty awesome. We had to reject that name because we can’t have more -zilla names and also no one would be able to spell Yggdrasil. Alas! So, anyway, treeherder.

The left side of the screen describes the latest batch of commits that were merged into mozilla-central. (The “tree” of code that is used to build Nightly.) On the right, there are a lot of operating systems/platforms listed. Linux opt (optimized version of Firefox for release) is at the top, along with a string of letters and numbers which we hope are green. Those letter and numbers represent batches of tests. You can hover over them to see a description. The tests marked M (1 2 3 etc) are mochitests, bc1 is mochitest-browser-chrome, dt are developer tools tests, and so on. For linux-opt you can see that there are some batches of tests with e10s in the name. We need the mochitest-plain tests to run on Firefox if it has e10s not enabled, or enabled. So the tests are duplicated, possibly changed to work under e10s, and renamed. We have M(1 2 3 . . .) tests, and also M-e10s(1 2 3 . . . ). Tests that are green are all passing. Orange means they aren’t passing. I am not quite sure what red (busted) means (bustage in the tree! red alert!) but let’s just worry about orange. (If you want to read more about the war on orange and what all this means, read Let’s have more green trees from Vaibhav’s blog. )

I kept asking, in order to figure out what needed doing that I could usefully do within the scope of As Soon As Possible, “So, what controls what tests are in which buckets? How do I know how many there are and what they are? Where are they in the codebase? How can I turn them on and off in a way that doesn’t break everything, or breaks it productively?” Good questions. Therefore the answers are long.

There are many other branches of the code other than mozilla-central. Holly is a branch where the builds for all the platforms have e10s enabled. (Many of these repos or branches or twigs or whatever, are named after different kinds of tree.) The tests are the standard set of tests, not particularly tweaked to allow for e10s. We can see what is succeeded and failing on treeherder’s view of holly. A lot of tests are orange on holly! Have a look at holly by clicking through on the link above. Here is a picture of the current state of holly.

Treeherder holly1

If you click a batch of tests where there are some failures — an orange one — then a new panel will open up in treeherder! I will pick a juicy looking one. Right now, for MacOS 10.6 opt, M(2) is orange. Clicking it gives me a ton of info. Scrolling down a bit in the bottom left panel tells me this:

mochitest-plain-chunked 164546/12/14136

The first number is how many tests ran. Scary. Really? 164546 tests ran? Kind of. This is counting assertions, in other words, “is” statements from SimpleTest. The first number lists how many assertions passed. The second number is for assertion failures and the third is for “todo” statements.

The batches of tests running on holly are all running against an e10s build of Firefox. Anything that’s consistently green on holly, we can move over to mozilla-central by making some changes in mozharness. I asked a few people how to do this, and Jim Matthies helpfully pointed me at a past example in Bug 1061014. I figured I could make a stab at adding some of the newly passing tests in Bug 1122901.

As I looked at how to do this, I realized I needed commit access level 2 so I filed a bug to ask for that. And, I also tried merging mozilla-central to holly. That was ridiculously exciting though I hadn’t fixed anything yet. I was just bringing the branch up to date. On my first try doing this, I immediately got a ping on IRC from one of the sheriffs (who do merges and watch the state of the “tree” or code repository) asking me why I had done something irritating and wrong. It took us a bit to figure out what had happened. When I set up mercurial, the setup process and docs told me to install a bunch of Mozilla specific hg extensions. So, I had an extension set up to post to bugzilla every time I updated something. Since I was merging several weeks of one branch to another this touched hundreds of bugs, sending bugmail to untold numbers of people. Mercifully, Bugzilla cut this off after some limit was reached. It was so embarrassing I could feel myself turning beet red as I thought of how many people just saw my mistake and wondered what the heck I was doing. And yet just had to forge onwards, fix my config file, and try it again. Super nicely, Clint Talbert told me that the first time he tried pushing some change he broke all branches of every product and had no idea what had happened. Little did he know I would blog about his sad story to make myself feel less silly….. That was years ago and I think Mozilla was still using cvs at that point! I merged mozilla-central to holly again, did not break anything this time, and watched the tests run and gradually appear across the screen. Very cool.

I also ended up realizing that the changes to mozharness to turn these batches of tests on again were not super obvious and to figure it out I needed to read a 2000-line configuration file which has somewhat byzantine logic. I’m not judging it, it is clearly something that has grown organically over time and someone else probably in release engineering is an expert on it and can tweak it casually to do whatever is needed.

Back to our story. For a batch of tests on holly that have failures and thus are showing up on treeherder as orange, it should be possible to go through the logs for the failing tests, figure out how to turn them off with some skip-if statements, filing bugs for each skipped failing test. Then, keep doing that till a batch of tests is green and it is ready to be moved over.

That seems like a reasonable plan for improving the automated test landscape, which should help developers to know that their code works in Firefox whether e10s is enabled or not. In effect, having the tests should mean that many problems are prevented from ever becoming bugs. The effect of this test coverage is hard to measure. How do you prove something didn’t happen? Perhaps by looking at which e10s tests fail on pushes to the try server. Another issue here is that there are quite a lot of tests that have been around for many years. It is hard too know how many of them are useful, whether there are a lot of redundant tests, in short whether there is a lot of cruft and there probably is. With a bit more experience in the code and fixing and writing tests it would be easier to judge the usefulness of these tests.

I can also see that, despite this taking a while to figure out (and to even begin to explain) it is a good entry point to contributing to Firefox. It has more or less finite boundaries. If you can follow what I just described in this post and my last post, and you can read and follow a little python and javascript, then you can do this. And, if you were to go through many of the tests, over time you would end up understanding more about how the codebase is structured.

As usual when I dive into anything technical at Mozilla, I think it’s pretty cool that most of this work happens in the open. It is a great body of data for academics to study, it’s an example of how this work actually happens for anyone interested in the field, and it’s something that anyone can contribute to if they have the time and interest to put in some effort.

This post seems very plain with only some screenshots of Treeherder for illustration. Here, have a photo of me making friends with a chicken.

Liz with chicken

Running mochitests for Firefox

January 2, 2015 Liz

I’m experimenting with automated testing for Firefox and figured it may be useful to record what I learned. I had a look at the Mochitest page on MDN as well as the main page on Automated testing at Mozilla. It is hard to know how to even begin to explain this. Mochitests are a huge ball of tests for Firefox. They run every time a change is pushed to mozilla-central, which is the sort of tip of the current state of our code and is used every day to build the Nightly version of Firefox. They’re run automatically for changes on other code repositories too. And, you can run them locally on your own version of Firefox.

This is going to have ridiculous levels of detail and jargon. Warning!

The first thing to do is to download the current code from mozilla-central and build it on my laptop. Here are the Firefox build instructions!

As usual I need to do several other things before I can do those things. This means hours of twiddling around on the command line, installing things, trying different configurations, fixing directory permissions and so on. Here are a few of the sometimes non-trivial things I ended up doing:

* updated Xcode and command line tools
* ran brew doctor and brew update, fixed all errors with much help from Stack Overflow, ended up doing a hard reset of brew
* Also, if you need to install a specific version of a utility, for example, autoconf: brew tap homebrew/versions; brew install autoconf213
* re-installed mercurial and git since they were screwed up somehow from a move from one Mac to another
* tried two different sample .mozconfig files, read through other Mac build config files, several layers deep (very confusing)
* updating my Firefox mozilla-central directory (hg pull -u)
* filed a bug for a build error and fixed some minor points on MDN

The build takes around an hour the first time. After that, pulling the changes from mozilla-central and reticulating the splines takes much less time.

Firefox build

Now I’m to the point where I can have a little routine every morning:
* brew doctor
* brew upgrade
* cd mozilla-central, hg pull -u
* ./mach build

Then I’m set up to run tests. Running all the mochitest-plain tests takes a long time. Running a single test may fail because it has dependencies on other tests it expects to have run first. You can also run all the tests in a particular directory, which may work out better than single tests.

Here is an example of running a single test.

./mach mochitest-browser browser/base/content/test/general/browser_aboutHome.js

Your Nightly or Nightly-debug browser will open and run through some tests. There will be a ton of output. Nifty.

Here is that same test, run with e10s enabled.

./mach mochitest-browser —e10s browser/base/content/test/general/browser_aboutHome.js

BTW if you add “2>&1 | tee -a test.log” to those commands they will pipe the output into a log file.

Back to testing. I poked around to see if I could find a super easy to understand test. The first few, I read through the test code, the associated bugs, and some other stuff. A bit overwhelming. My coworker Juan and I then talked to Joel Maher who walked us through some of the details of how mochitests work and are organized. The tests are scattered throughout the “tree” of directories in the code repository. It is useful generally to use DXR to search but I also ended up just bouncing around and getting familiar with some of the structure of where things are. For example, scarily, I now know my way to testing/mochitest/tests/SimpleTest/. Just by trying different things and looking around you start to get familiar.

Meanwhile, my goal was still to find something easy enough to grasp in an afternoon and run through as much of the process to fix a simple bug as I could manage. I looked around for tests that are known not to work under e10s, and are marked in manifest files that they should be skipped if you’re testing with e10s on. I tried turning some of these tests on and off and reading through their bugs.

Also meanwhile I asked for commit access level 1 (for the try server) so when I start changing and fixing things I can at least throw them at a remote test server as well as my own local environment’s tests.

Then, hurray, Joel lobbed me a very easy test bug.

From reading his description and looking at the html file it links to, I got that I could try this test, but it might not fail. The failure was caused by god knows what other test. Here is how to try it on its own:

./mach mochitest-chrome browser/devtools/webide/test/test_zoom.html

I could see the test open up Nightly-debug and then try to zoom in and out. Joel had also described how the test opens a window, zooms, closes the window, then opens another window, zooms, but doesn’t close. I have not really looked at any JavaScript for several years and it was never my bag. But I can fake it. Hurray, this really is the simplest possible example. Not like the 600-line things I was ending up in at random. Danny looked at the test with me and walked through the JavaScript a bit. If you look at the test file, it is first loading up the SimpleTest harness and some other stuff. I did not really read through the other stuff. *handwave* Then in the main script, it says that when the window is loaded, first off wait till we really know SimpleTest is done because the script tells us so (Maybe in opposition to something like timing out.) Then a function opens the WebIDE interface, and (this is the main bit Danny explained) the viewer= winQuerInterface etc. bit is an object that shows you the state of the viewer. *more handwaving, I do not need to know* More handwaving about “yield” but I get the idea it is giving control over partly to the test window and partly keeping it. Then it zooms in a few times, then this bit is actually the meat of the test:

 is(roundZoom, 0.6, "Reach min zoom");

Which is calling the “is” function in SimpleTest.js, Which I had already been reading, and so going back to it to think about what “is” was doing was useful. Way back several paragraphs ago I mentioned looking in testing/mochitest/tests/SimpleTest/. That is where this function lives. I also felt I did not entirely need to know the details of what the min and max zoom should be. Then, we close the window. Then open a new one. Now we see the other point of this test. We are checking to see that when you open a WebIDE window, then zoom to some zoomy state, then close the window, then re-open it, it should stay zoomed in or out to the state you left it in.

OK, now at this point I need to generate a patch with my tiny one line change. I went back to MDN to check how to do this in whatever way is Mozilla style. Ended up at How to Submit a Patch, then at How can I generate a patch for somebody else to check in for me?, and then messing about with mq which is I guess like Quilt. (Quilt is a nice name, but, welcome to hell.) I ended up feeling somewhat unnerved by mq and unsure of what it was doing. mq, or qnew, did not offer me a way to put a commit message onto my patch. After a lot of googling… not sure where I even found the answer to this, but after popping and then re-pushing and flailing some more, and my boyfriend watching over my shoulder and screeching “You’re going to RUIN IT ALL” (and desperately quoting Kent Beck at me) as I threatened to hand-edit the patch file, here is how I added a commit message:

hg qrefresh -m "Bug 1116802: closes the WebIDE a second time"

Should I keep using mq? Why add another layer into mercurial? Worth it?

To make sure my patch didn’t cause something shocking to happen (It was just one line and very simple, but, famous last words….) I ran the chrome tests that were in the same directory as my buggy test. (Piping the output into a log file and then looked for anything that mentioned test_zoom. ) The test output is a study in itself but not my focus right now as long as nothing says FAIL.

./mach mochitest-chrome browser/devtools/webide/test/

Then I exported the patch still using mq commands which I cannot feel entirely sure of.

 hg export qtip > ~/bug-1116802-fix.patch

That looked fine and, yay, had my commit message. I attached it to the bug and asked Joel to have a look.

I still don’t have a way to make the test fail but it seems logical you would want to close the window.

That is a lot of setup to get to the point where I could make a useful one line change in a file! I feel very satisfied that I got to that point.

Only 30,000 more tests to go, many of which are probably out of date. As I contemplate the giant mass of tests I wonder how many of them are useful and what the maintenance cost is and how to ever keep up or straighten them out. It’s very interesting!

You can have a look at the complicated nature of the automated tests that run constantly to test Firefox at treeherder. For any batch of commits merged into mozilla-central, huge numbers of tests run on many different platforms. If you look at treeherder you can see a little (hopefully green) “dt” among the tests. I think that the zooming in WebIDE test that I just described is in the dt batch of tests (but I am not sure yet).

I hope describing the process of learning about this small part of Firefox’s test framework is useful to someone! I have always felt that I missed out by not having a college level background in CS or deep expertise in any particular language. And yet I have still been a developer on and off for the last 20 years and can jump back into the pool and figure stuff out. No genius badge necessary. I hope you can see that actually writing code is only one part of working in this kind of huge, collaborative environment. The main skill you need (like I keep saying) is the ability not to freak out about what you don’t know, and keep on playing around, while reading and learning and talking with people.

Thoughts on working at Mozilla and the Firefox release process

December 1, 2014 Liz

Some thoughts on my life at Mozilla as I head into our company-wide work week here in Portland! My first year at Mozilla I spent managing the huge volume of bugs, updating docs on how to triage incoming bugs and helping out with Bugzilla itself. For my second year I’ve been more closely tied into the Firefox release process, and I switched from being on the A-Team (Automation and Tools) to desktop Firefox QA, following Firefox 31 and then Firefox 34 through their beginning to release.

picard saying if it's not in bugzilla it doesn't exist

I also spent countless hours fooling around with Socorro, filing bugs on the highest volume crash signatures for Firefox, and then updating the bugs and verifying fixes, especially for startup crashes or crashes associated with “my” releases.

Over this process I’ve really enjoyed working with everyone I’ve met online and in person. The constant change of Mozilla environments and the somewhat anarchic processes are completely fascinating. Though sometimes unnerving. I have spent these two years becoming more of a generalist. I have to talk with end users, developers on every Mozilla engineering team, project or product managers if they exist, other QA teams, my own Firefox/Desktop QA team, the people maintaining all the tools that all of us use, release management, and release engineering. Much of the actual QA has been done by our Romanian team so I have coordinated a lot with them and hope to meet them all some day. When I need to dig into a bug or feature and figure out how it works, it means poking around in documentation or in the code or talking with people, then documenting whatever it is.

This is a job (and a workplace) suited for people who can cope with rapid change and shifting ground. Without a very specific focus, expertise is difficult. People complain a lot about this. But I kind of like it! And I’m constantly impressed by how well our processes work and what we produce. There are specific people who I think of as total rock stars of knowing things. Long time experts like bz or dbaron. Or dmajor who does amazing crash investigations. I am in their secret fan club. If there were bugzilla “likes” I would be liking up a storm on there! It’s really amazing how well the various engineering teams work together. As we scramble around to improve things and as we nitpick I want to keep that firmly in mind. And, as we are all a bit burned out from dealing with the issues from the 33 Firefox release and many point releases, then the 10th anniversary release, then last minute scrambling to incorporate surprise/secret changes to Firefox 34 because of our new deal with Yahoo.

I was in a position to see some of the hard work of the execs, product and design people, and engineers as well as going through some rounds of iterating very quickly with them these last 2 weeks (and seeing gavin and lmandel keep track of that rapid pace) Then seeing just part of what the release team needed to do, and knowing that from their perspective they also could see what IT and infrastructure teams had to do in response. So many ripples of teamwork and people thinking things through, discussing, building an ever-changing consensus reality.

I find that for every moment I feel low for not knowing something, or making a mistake (in public no less) there are more moments when I know something no one else knows in a particular context or am able to add something productive because I’m bringing a generalist perspective that includes my years of background as a developer.

As an author and editor I am reminded of what it takes to edit an anthology. My usual role is as a general editor with a vision: putting out a call, inviting people to contribute, working back and forth with authors, tracking all the things necessary (quite a lot to track, more than you’d imagine!) and shifting back and forth from the details of different versions of a story or author bios, to how the different pieces of the book fit together. And in that equation you also, if you’re lucky, have a brilliant and meticulous copy editor. At Aqueduct Press I worked twice with Kath Wilham. I would have gone 10 rounds of nitpick and Kath would still find things wrong. My personal feeling for my past year’s work is that my job as test plan lead on a release has been 75% “editor” and 25% copy editor.

If I had a choice I would always go with the editor and “glue” work rather than the final gatekeeper work. That last moment of signing off on any of the stages of a release freak me out. I’m not enough of a nitpicker and in fact, have zero background in QA — instead, 20 years as a lone wolf (or nearly so) developer, a noisy, somewhat half-assed one who has never *had* QA to work with much less *doing* QA. It’s been extremely interesting, and it has also been part of my goal for working in big teams. The other thing I note from this stressful last 6 weeks is that, consistent with other situations for me, in a temporary intense “crisis” mode my skills shine out. Like with the shifting-every-30-minutes disaster relief landscape, where I had great tactical and logistical skills and ended up as a good leader. My problem is that I can’t sustain that level of awareness, productivity, or activeness, both physically and mentally, for all that long. I go till I drop, crash & burn. The trick is knowing that’s going to happen, communicating it beforehand, and having other people to back you up. The real trick which I hope to improve at is knowing my limits and not crashing and burning at all. On the other hand, I like knowing that in a hard situation I have this ability to tap into. It just isn’t something I can expect to do all the time, and isn’t sustainable. One thing I miss about my “old” life is building actual tools. These last 2 years have been times of building human and institutional infrastructure for me much more than making something more obvious and concrete. I would like to write another book and to build a useful open source tool of some sort. Either for Mozilla, for an anti-harassment tool suite, or for Ingress…. 🙂 Alternatively I have a long term idea about open source hardware project for mobility scooters and powerchairs. That may never happen, so at the least, I’m resolving to write up my outline of what should happen, in case someone else has the energy and time to do it.

In 2013 and 2014 I mentored three interns for the GNOME-OPW project at Mozilla. Thanks so much to Tiziana Selitto, Maja Frydrychowicz, and Francesca Ciceri for being awesome to work with! As well as the entire GNOME-OPW team at Mozilla and beyond. I spoke about the OPW projects this summer at Open Source Bridge and look forward to more work as a mentor and guide in the future.

Meanwhile! I started a nonprofit in my “spare time”! It’s Double Union, a feminist, women-only hacker and maker space in San Francisco and it has around 150 members. I’m so proud of everything that DU has become. And I continue my work on the advisory boards of two other feminist organizations, Ada Initiative and GimpGirl as well as work in the backchannels of Geekfeminism.org. I wrote several articles for Model View Culture this year and advised WisCon on security issues and threat modeling. I read hundreds of books, mostly science fiction, fantasy, and history. I followed the awesome work my partner Danny does with EFF, as well. These things are not just an important part of my life, they also make my work at Mozilla more valuable because I am bringing perspectives from these communities to the table in all my work.

The other day I thought of another analogy for my last year’s work at Mozilla that made me laugh pretty hard. I feel personally like the messy “glue” Perl scripts it used to be my job to write to connect tools and data. Part of that coding landscape was because we didn’t have very good practices or design patterns but part of it I see as inevitable in our field. We need human judgement and routing and communication to make complicated systems work, as well as good processes.

I think for 2015 I will be working more closely with the e10s team as well as keeping on with crash analysis and keeping an eye on the release train.

Mad respect and appreciation to everyone at Mozilla!!

Firefox launch party liz larissa in fox ears

How to test new features in Firefox 34 Aurora

September 11, 2014 Liz

If you’re a fan of free and open source software and would like to contribute to Firefox, join me for some Firefox feature testing!

There are some nifty features under development right now for Firefox 34 including translation in the browser, making voice or video calls (a feature called “Hello” or “Loop”), debugging information for web developers in the Dev Tools Inspector, and recent improvements to HTML5 gaming.

I’ve written step by step instructions on these
ways to test Firefox 34. If you would like to see what it’s like to improve a popular open source project, trying out these tasks is a good introduction.

Aurora

First, Install the Aurora version of Firefox. It is best to set it up to use multiple profiles. That ensures you don’t use your everyday version of Firefox for testing, so you won’t risk losing your usual profile information. It also makes it easy to restart Firefox with a new, clean profile with all the default settings, very useful for testing. Sometimes I realize I’m running 5 different versions of Firefox at once!

To test “Hello”, try making some voice or video calls from Firefox Aurora. You will need a friend to test with. Or, use two computers that you control. This is a good task to try while joining our chat channels, #qa or #testday on irc.mozilla.org; ask if anyone there wants to test Hello with you. The goal here is mostly to find and report new bugs.

If you test the translation infobar in Aurora you may find some new bugs. This is a fun feature to test. I like trying it on Wikipedia in many different languages, and also looking at newspapers!

If you’re a web developer, you may use Developer Tools in Firefox. I’m asking Aurora users to go through some unconfirmed bug reports, to help improve the Developer Tools Inspector.

If you like games you can test HTML5 web-based games in Firefox Aurora. This helps us improve Firefox and also helps the independent game developers. We have a list of demo games so you can play them, report glitches, and feel like a virtuous open source citizen all at once. Along the way you have opportunities to learn some interesting stuff about how graphics on the web can work (or not work).

Monster madness

These testing tasks are all set up in One and Done, Mozilla QA’s site to start people along the path to joining our open source community. This site was developed with a lot of community contribution including the design and concept by long-time community member Parul and a lot of code by two interns this summer, Pankaj and Maja.

Testing gives a great view into the development process for people who may not (yet) be programmers. I especially love how transparent Mozilla’s process can be. Anyone can report a bug, visible to the entire world in bugzilla.mozilla.org. There are many people watching that incoming stream of bug reports, confirming them and routing them to developer teams, sometimes tagging them as good first bugs for new contributors. Developers who may or may not be Mozilla employees show up in the bugs, like magic . . . if you think of bugmail notifications as magic . . .

It is amazing to see this very public and somewhat anarchic collaboration process at work. Of course, it can also be extremely satisfying to see a bug you discovered and reported, your pet bug, finally get fixed.

Mozilla's bug reporting, QA, and release processes

June 27, 2014 Liz

AdaCamp Portland was an amazing conference for feminist women in open source tech and culture. Not all, but many of the conference attendees are developers, system administrators, or do other technical work in open source software. I gave an informal talk meant to be an overview of some things I currently do at Mozilla. Lots of people came to the session! We all introduced ourselves going around the room.

To start off with, I showed a sample bug to talk about the process of reporting a bug, using Bugzilla, and practicing the skill of reading and understanding a bug report.

We looked first at Bug 926292.

Bugzilla couple

Let’s look at the life of this bug!

This bug was reported in October 2013 for Firefox 24 by someone new to bugzilla.mozilla.org. New users have basic permissions to file and comment on bugs. For around their first 25 bugs filed or commented on, they are marked “New to Bugzilla” to anyone with more permissions on the system. This helps more experienced users to know when they’re in conversation with people who are relatively new to the system. And, bugs reported by new users are automatically entered into Bugzilla with a status of “UNCONFIRMED”.

Our bug reporter was answered the same day by a community bug triager who used the “needinfo” checkbox to ask the bug reporter more questions. A bit later, in Comment 2, I was able to confirm the bug; I marked it NEW. Community members often jump in to do this from Bug Triage bug days, from our One and Done community taskboard, or because they watch the “Firefox::Untriaged” component. (Yes . . . you too can sign up to get email from Bugzilla every time a new bug is filed!)

Francesca Ciceri is currently working on bug triage and verification with our team as part of the GNOME-OPW internship program, doing similiar work to Tiziana Selitto who was an OPW intern last year! Both their blogs have good insights into what it’s like to approach QA in a huge and somewhat chaotic system like Mozilla’s.

In our example bug, I took a guess as to which product and component to add to the bug. This is like putting the bug into the right place where developers who work in a particular area will be likely to see it, and pay attention to it. I moved it from “Firefox” to “Core” and thought it may be something to do with the CSS Object Model. Picking the right product and component is tricky. Sometimes I look for similar bugs, to see what component they’re in. Sometimes I use Bugzilla’s Browse pages to skim or search through the descriptions of components for Firefox, Core, and Toolkit. Even after doing this for a year and a half, I get it wrong. Here, a developer moved the bug to what he thought was a better component for it, Core::Layout. (Developers also sometimes guess wrong, and keep passing a bug around to each others’ components like a hot potato.)

At this point a few developers explored the bug, and went back and forth with each other and the bug reporter about whether it had been fixed or not, exactly what the bug was, whether it is a Mac issue or a Firefox issue, and how to fix it. It was resolved as a duplicate of another bug in October, but the bug reporter came back to reopen it in February 2014. The bug reporter was polite but persistent in explaining their view, giving more details of the browser behavior, trying to find the bug in the very latest developer build (Nightly), giving a test case and comparing the behavior in different browsers. A developer submitted a patch, asked for code review. Related bugs were mentioned and linked. At least two new bugs were filed.

One important thing to note is that people working on QA and development tend to move very fluidly between using various Firefox versions. One of the best things you can do to get involved with helping out is to set up all four “channels” of Firefox with the capability to run them all at once with different profiles, and to start with new, clean profiles. In fact, we need better and more up to date documentation of how to do that on different operating systems, with screenshots! Here are some links that may help you set that up:
* http://www.callum-macdonald.com/about/faq/multiple-firefox-instances/
* https://developer.mozilla.org/en-US/docs/Mozilla/Multiple_Firefox_Profiles
* https://support.mozilla.org/en-US/kb/profile-manager-create-and-remove-firefox-profiles

OK, back to bug 926292!

Since I had worked on the bug and added myself to the cc field, I got bugmail about all these changes, and more or less followed a long. I often think that the collaboration that happens in bug fixing is very beautiful, and even fairly efficient!

In comment 29 you can see that code got committed to a mercurial repository, to “inbound”. From there, it goes through automated tests and is merged by one of the “sheriffs” into another hg repository, mozilla-central, where it will go into the next build of Nightly, which at that point in April, was Firefox 31.

Comment 30 suggests uplifting the patch to versions that will soon be released, to Aurora and Beta. Release managers started to get involved, commenting and asking the developers to formally nominate the bug for uplift.

At this point in my talk I explained a little bit about the “trains”.

Trains

The versions of Firefox under development advance on a 6 week cycle, from Nightly to Aurora to Beta to the main release of Firefox. In this rapid release schedule, Firefox 31 was Nightly, so Aurora was 30, 29 was Beta, and the release version most folks use was 28. The uplift request was refused so the patch “rode the train”. That means, if you were using Firefox 31 any time after the patch was merged into mozilla-central, you will see its effect. (It would also be fixed for Firefox 32 and 33 which are currently in use as Aurora and Nightly, since 31 is currently Beta.)

Our bug was marked “FIXED” when the patch was merged into mozilla-central. You can see near the end of its comments that I tagged the bug “verifyme” to put it into the queue of bugs that need verifying for Firefox 31. Many people see that list and work on verifying bugs including community members in our Bug Verification test days. I hope the story of this particular bug is over. I don’t have the number immediately to hand but I believe that over 1000 bugs are fixed for each version of Firefox over its release cycle. We can’t verify them all, but it’s amazing what we do get done as a team!

Other tools we looked at in my talk and the ensuing discussion:

Datazilla, which tests and measures Firefox performace: https://datazilla.mozilla.org/

Mozmill, a UI automation framework for Mozilla apps including Firefox and Thunderbird: https://github.com/mozilla/mozmill

Socorro, or crash-stats, where QA and other teams keep track of crashes in Firefox and other Mozilla products: https://crash-stats.mozilla.com

The ftp directories where Firefox builds and build candidates are stored: ftp://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/

The mercurial repositories or “the tree”: http://hg.mozilla.org/

DXR, a nifty tool to search Mozilla’s code: http://dxr.mozilla.org/mozilla-central/source/

TBPL which shows the test results for every commit that’s merged into different branches https://tbpl.mozilla.org/

And a quick view into Mozilla’s Jenkins continuous integration dashboard which you can only see from our VPN, just to give an idea of the work we do when Firefox is in Beta. As a particular version of Firefox advances through rapid release, QA pays more attention to particular areas and uses different tools. We have to know a little bit about everything, be able to reproduce a user’s bug on many different possible platforms, figure out which developers may be able to fix a bug (or whose commit may have caused a regression or crash).

It was a lot to cover in an hour long talk! I wanted to pilot this informally as a test for doing a more formal talk with slides.

It represented fairly well that QA covers quite a lot of territory; it’s complicated and interesting work.

Mozilla Summit 2013: bugs, crafts, and fun

October 10, 2013 Liz

The Mozilla Summit in Toronto was a lot of fun. I met so many intense, idealistic, motivated open source contributors, it made me feel renewed energy for my own contribution.

On the first day I heard some of the keynotes but missed Mitchell Baker’s keynote on the Nature of Mozilla, which I’m going to watch today. People were talking about it through the rest of the conference, so I’m curious.

The talk I did was called Awesome Bugzilla Tricks and was a fairly short presentation of some tips for using Bugzilla.

Awesome bugzilla-tricks-slides2 from Liz Henry

As we went through these tips, the talk turned often into a good discussion of how people use these features of Bugzilla. We thought of a new way to implement Bugzilla tags to look more like browser bookmarks.

People talked about their own workflow as we went around the room describing what we do with bugzilla.mozilla.org! That is usually my favorite part of a workshop level technical discussion. It is like a series of sharing mini-demonstrations about actual working process, what we use and what we do and why. That is very illuminating. I realized in our discussion that I was talking with Kohei who made bzdeck, a tool that makes reading Bugzilla.mozilla.org bug reports more like an email inbox. I had particularly wanted to meet him so was very happy he came to the talk and the discussion! 😀

Durng the talk I said something a little weird and abstract that was not in my slides, inspired by the discussion. Here it is…

As a literary critic I find it fascinating that it is a huge collection of textual information which we engage with as authors and readers. It is like the ultimate “difficult work”, like reading James Joyce except 900,000 times better and with a more interesting result. There is no way to make it easy to understand, even if you can make little pieces of it easier to use and learn, the underlying information and tasks are beautifully complex and demanding for anyone who engages with it, which means you can’t fail to learn something if you try.

Then, luckily, we went back to talking about practical things, features, dashboards, and workflow.

The other main activity I did at the Summit was to set up a big table with craft supplies. Based on an idea of Lukas Blakk’s, I set up beads, string, and charms shaped like ladybugs, dragonflies, bees, and beetles so that people could make necklaces, bracelets, and other wearable souvenirs. The beads had numbers and letters so that you could spell out the number of your favorite bug.

People did just that!

I loved seeing people think about what bug was their favorite or most important to them personally. Sometimes a first bug, or one that people were proud of fixing, or one with an important, complicated discussion. Several people told me that during the Summit, they looked up other people’s favorite bugs from the bug bracelets, and learned something interesting.

Bug 356038 was represented by number and by its Bugzilla alias, BCP47:

Rust and Github bug #5677, ‘Rustpkg “ready for use” metabug’ got some love:

Vu gave a shout out to bug 780076,

And here is another bracelet featuring bug 808964,

For my own favorites I made three objects, one a wire necklace for bug 923590, “Pledge never to implement HTML5 DRM”, and a bracelet and a barette with my first patch because I was proud of submitting a patch. The EME or DRM issue was discussed very intensely on Sunday by many engineers. Feelings definitely ran high and people were determined to continue discussion. I was glad it got an official slot in the schedule for discussion and that there was widespread interest.

Bug 298619 was put onto a cell phone charm along with a blue and orange glass bead shaped like a beetle!

Many other people made Mozilla or MozReps necklaces, spelled out their names or their loved one’s names, with bug charms, or with wire, like this beautiful and creative copper wire creation:

And this Maker Party necklace!

This resulted in “conference swag” was personal and made on the spot, worn and also given as presents! Many people commented that they felt soothed and comforted by hands-on activity in the middle of intense social interaction. I observed that people discussed what bugs or words to spell, how to design their objects, and what techniques to use, very collaboratively, so it was a nice physical representation of work and process.

After the first day, I left the craft station open 24 hours in the lounge, replenishing it with beads from West Queen Street on day 2 becasue we ran out of the letters Z and A, popular in spelling out “Bugzilla” and “Mozilla”. Of course this was because we were in Canada, where they use a lot of “eh”!

I would do this again at a conference, and now I know from experience what supplies are needed and which things are likely to be popular.

The other silly and fun part of the conference for me was wearing (and lending to people) the brainwave controlled robot fox ears! The looks on people’s faces, so priceless, as they realized the ears were really moving. Everyone who tried it laughed very hard!

Thanks for a great and inspiring Summit, everybody!