firefox – Composite

Nifty addition to treeherder

February 5, 2018 Liz

At work I just found out we have a nice change in Treeherder, the tool that monitors current Firefox (and other) builds. Without writing my usual 10 pages of explanation and backstory, I’ll just say that Treeherder shows whether the release engineering infrastructure has started the release promotion process to take the builds from the changeset specified and do some super mystery magic on them to make them suitable for actual release to Firefox users. We get an email from the notification system that this happened, but now we can follow along in the same tool we use to watch tests pass or fail.

gecko decision task for firefox and fennec

In this case, I used a different tool (Ship-It) to set up the parameters to take the test builds for the 1a24837f9ed232d8d2dc4535d11ee53c9847b109 changeset (say it 6 times, fast) for Firefox 59 beta 7 (plus the android and developer edition versions) and declare us ready to kick off this process. Instead of checking email notifications that only go to a particular mailing list, or diving straight into TaskCluster, I can see that near the end of the enormous list of automated tests running on various platforms, “Gecko Decision Task”. This tells us that the builds are now being “promoted”. I haven’t tried drilling deeper yet, but if you click on any of the links there (such as “promote_firefox”) then you get even more detail, including links to TaskCluster and its myriad, confusing joys.

As usual, I think it’s damn cool that all this happens “in the open” at Mozilla, so you don’t even have to be logged into anything to see quite a lot of detail. Anyone with an interest can learn a lot here, or even get involved and contribute, because of this level of commitment to transparency. It’s a good way to find out if you like the work, or to get work experience that you can easily show off in the future.

Captain's log, stardate 2016-10-26

October 26, 2016 Liz

Once again I resolve to write about my work at Mozilla as a Firefox release manager. It’s hard to do, because even the smallest thing could fill LONG paragraphs with many links! Since I keep daily notes on what I work on, let me try translating that in brief. When moved, maybe I’ll go into depth.

This week we are coming into the home stretch of a 7 week release cycle. “My” release for right now is Firefox 49, which was released on I’m still juggling problems and responses and triaging for that every day. In a week and a half, we were scheduled to release Firefox 50. Today after some discussion we pushed back that schedule by a week.

Meanwhile, I am also helping a new release manager (Gerry) to go through tracked bugs, new regressions, top crash reports, and uplift requests for Aurora/Developer Edition (Firefox 51). I’m going through uplift requests for Firefox 45.5.0esr, the extended support release. There’s still more – I paid some attention to our “update orphaning” project to bring users stuck on older versions of Firefox forward to the current, better and safer versions.

As usual, this means talking to developers and managers across pretty much all the teams at Mozilla, so it is never boring. Our goal is to get fixes and improvements as fast as possible while making sure, as best we can, that those fixes aren’t causing worse problems. We also have the interesting challenges of working across many time zones around the world.

Firefox stuffed animal installed

Today I had a brief 1:1 meeting with my manager and went to the Firefox Product cross-functional meeting. I always find useful as it brings together many teams. There was a long Firefox team all hands discussion and then I skipped going to another hour long triage meeting with the platform/firefox engineering managers. Whew! We had a lively discussion over the last couple of days about a performance regression (Bug 1304434). The issues are complicated to sort out. Everyone involved is super smart and the discussions have a collegiate quality. No one is “yelling at each other”, while we regularly challenge each other’s assumptions and are free to disagree – usually in public view on a mailing list or in our bug tracker. This is part of why I really love Mozilla. While we can get a bit heated and stressed, overall, the culture is good. YMMV of course.

By that time (11am) I had been working since 7:30am, setting many queries in bugs, and on IRC, and in emails into motion and made a lot of small but oddly difficult decisions. Often this meant exercising my wontfix powers on bugs — deferring uplift (aka “backport” ) to 50, 51, or leaving a fix in 52 to ride the trains to release some time next year. As I’m feeling pretty good I headed out to have lunch and work from a cafe downtown (Mazarine – the turkey salad sandwich was very good!)

This afternoon I’m focusing on ESR 45, and Aurora 51, doing a bit more bug triage. There are a couple of ESR uplifts stressing me out — seriously, I was having kittens over these patches — but now that we have an extra week until we release, it feels like a better position for asking for a 2nd code review, a bit more time for QA, and so on.

Heading out soon for drinks with friends across the street from this cafe, and then to the Internet Archive’s 20th anniversary party. Yay, Internet Archive!

That Zarro Boogs feeling

April 20, 2016 Liz

This is my third Firefox release as release manager, and the fifth that I’ve followed closely from the beginning to the end of the release cycle. (31 and 36 as QA lead; 39, 43, and 46 as release manager.) This time I felt more than usually okay with things, even while there was a lot of change in our infrastructure and while we started triaging and following even more bugs than usual. No matter how on top of things I get, there is still chaos and things still come up at the last minute. Stuff breaks, and we never stop finding new issues!

I’m not going into all the details because that would take forever and would mostly be me complaining or blaming myself for things. Save it for the post-mortem meeting. This post is to record my feeling of accomplishment from today.

During the approximately 6 week beta cycle of Firefox development we release around 2 beta versions per week. I read through many bugs nominated as possibly important regressions, and many that need review and assessment to decide if the benefit of backporting warrants the risk of breaking something else.

During this 7 week beta cycle I have made some sort of decision about at least 480 bugs. That usually means that I’ve read many more bugs, since figuring out what’s going on in one may mean reading through its dependencies, duplicates, and see-alsos, or whatever someone randomly mentions in comment 45 of 96.

And today I got to a point I’ve never been at near the end of a beta cycle: Zarro Boogs found!

This is what Bugzilla says when you do a query and it returns 0. I think everyone likes saying (and seeing) “Zarro Boogs”. Its silliness expresses the happy feeling you get when you have burned down a giant list of bugs.

This particular query is for bugs that anyone at all has nominated for the release management team to pay attention to.

Here is the list of requests for uplift (or backporting, same thing) to the mozilla-beta repo:

Yes!! Also zarro boogs.

Since we build our release candidate a week (or a few days) from the mozilla-release repo, I check up on requests to uplift there too:

PEAK ZARRO BOOGS.

For the bugs that are unresolved and that I’m still tracking into the 46 release next week, it’s down to 4: Two fairly high volume crashes that may not be actionable yet, one minor issue in a system addon that will be resolved in a planned out-of-band upgrade, and one web compatibility issue that should be resolved soon by an external site. Really not bad!

Our overall regression tracking has a release health dashboard on displays in many Mozilla offices. Blockers, 0. Known new regressions that we are still working on and haven’t explicitly decided to wontfix: 1. (But this will be fixed by the system addon update once 46 ships.) Carryover regressions: 41; about 15 of them are actually fixed but not marked up correctly yet. The rest are known regressions we shipped with already that still aren’t fixed. Some of those are missed uplift opportunities. We will do better in the next release!

In context, I approved 196 bugs for uplift during beta, and 329 bugs for aurora. And, we fix several thousands of issues in every release during the approx. 12 week development cycle. Which ones of those should we pay the most attention to, and which of those can be backported? Release managers act as a sort of Maxwell’s Demon to let in only particular patches …

Will this grim activity level for the past 7 weeks and my current smug feeling of being on top of regression burndown translate to noticeably better “quality”… for Firefox users? That is hard to tell, but I feel hopeful that it will over time. I like the feeling of being caught up, even temporarily.

Here I am with drink in hand on a sunny afternoon, toasting all the hard working developers, QA testers, beta users, release engineers, PMs, managers and product folks who did most of the actual work to fix this stuff and get it firmly into place in this excellent, free, open source browser. Cheers!

Automated test harnesses for Firefox, zooming out a bit

February 7, 2015 Liz

In December I was going through some of the steps to be able to change, fix, and interpret the results of a small subset of the gazillion automated tests that Mozilla runs on Firefox builds. My last post arrived at the point of being able to run mochitests locally on my laptop on the latest Firefox code. Over the holidays I dug further into the underlying situation and broaded my perspective. I wanted to make sure that I didn’t end up grubbing away at something that no one cared about or wasn’t missing the big picture.

My question was, how can I, or QA in general, understand where Firefox is with e10s (Electrolysis, the code name for the multiprocess Firefox project). Can I answer the question, are we ready to have e10s turned on by default in Firefox — for the Developer Edition (formerly known as Aurora), for Beta, and for a new release? What criteria are we judging by? Complicated. And given those things, how can we help move the project along and ensure good quality; Firefox that works as well as or, we hope, better than, Firefox with e10s not enabled? My coworker Juan and I boiled it down to basically, stability (lowering the crash rate) and automated test coverage. To answer my bigger questions about how to improve automated test coverage I had to kind of zoom out, and look from another angle. For Firefox developers a lot of what I am about to describe is basic knowledge. It seems worth explaining, since it took me significant time to figure out.

First of all let’s look at treeherder. Treeherder is kind of the new TBPL, which is the old new Tinderbox. It lets us monitor the current state of the code repositories and the tests that run against them. It is a window into Mozilla’s continuous integration setup. A battery of tests are poised to run against Firefox builds on many different platforms. Have a look!

Current view of mozilla-central on treeherder

Treeherder2

Edward Tufte would have a cow. Luckily, this is not for Tufte to enjoy. And I love it. You can just keep digging around in there, and it will keep telling you things. What a weird, complicated gold mine.

Digression! When I first started working for Mozilla I went to the Automation and Tools team work week where they all came up with Treeherder. We wanted to name it something about Ents, because it is about the Tree(s) of the code repos. I explained the whole thing to my son, I think from the work week, which awesomely was in London. He was 11 or 12 at the time and he suggested the name “Yggdrazilla” keeping the -zilla theme and in reference to Yggdrasil, the World-Tree from Norse mythology. My son is pretty awesome. We had to reject that name because we can’t have more -zilla names and also no one would be able to spell Yggdrasil. Alas! So, anyway, treeherder.

The left side of the screen describes the latest batch of commits that were merged into mozilla-central. (The “tree” of code that is used to build Nightly.) On the right, there are a lot of operating systems/platforms listed. Linux opt (optimized version of Firefox for release) is at the top, along with a string of letters and numbers which we hope are green. Those letter and numbers represent batches of tests. You can hover over them to see a description. The tests marked M (1 2 3 etc) are mochitests, bc1 is mochitest-browser-chrome, dt are developer tools tests, and so on. For linux-opt you can see that there are some batches of tests with e10s in the name. We need the mochitest-plain tests to run on Firefox if it has e10s not enabled, or enabled. So the tests are duplicated, possibly changed to work under e10s, and renamed. We have M(1 2 3 . . .) tests, and also M-e10s(1 2 3 . . . ). Tests that are green are all passing. Orange means they aren’t passing. I am not quite sure what red (busted) means (bustage in the tree! red alert!) but let’s just worry about orange. (If you want to read more about the war on orange and what all this means, read Let’s have more green trees from Vaibhav’s blog. )

I kept asking, in order to figure out what needed doing that I could usefully do within the scope of As Soon As Possible, “So, what controls what tests are in which buckets? How do I know how many there are and what they are? Where are they in the codebase? How can I turn them on and off in a way that doesn’t break everything, or breaks it productively?” Good questions. Therefore the answers are long.

There are many other branches of the code other than mozilla-central. Holly is a branch where the builds for all the platforms have e10s enabled. (Many of these repos or branches or twigs or whatever, are named after different kinds of tree.) The tests are the standard set of tests, not particularly tweaked to allow for e10s. We can see what is succeeded and failing on treeherder’s view of holly. A lot of tests are orange on holly! Have a look at holly by clicking through on the link above. Here is a picture of the current state of holly.

Treeherder holly1

If you click a batch of tests where there are some failures — an orange one — then a new panel will open up in treeherder! I will pick a juicy looking one. Right now, for MacOS 10.6 opt, M(2) is orange. Clicking it gives me a ton of info. Scrolling down a bit in the bottom left panel tells me this:

mochitest-plain-chunked 164546/12/14136

The first number is how many tests ran. Scary. Really? 164546 tests ran? Kind of. This is counting assertions, in other words, “is” statements from SimpleTest. The first number lists how many assertions passed. The second number is for assertion failures and the third is for “todo” statements.

The batches of tests running on holly are all running against an e10s build of Firefox. Anything that’s consistently green on holly, we can move over to mozilla-central by making some changes in mozharness. I asked a few people how to do this, and Jim Matthies helpfully pointed me at a past example in Bug 1061014. I figured I could make a stab at adding some of the newly passing tests in Bug 1122901.

As I looked at how to do this, I realized I needed commit access level 2 so I filed a bug to ask for that. And, I also tried merging mozilla-central to holly. That was ridiculously exciting though I hadn’t fixed anything yet. I was just bringing the branch up to date. On my first try doing this, I immediately got a ping on IRC from one of the sheriffs (who do merges and watch the state of the “tree” or code repository) asking me why I had done something irritating and wrong. It took us a bit to figure out what had happened. When I set up mercurial, the setup process and docs told me to install a bunch of Mozilla specific hg extensions. So, I had an extension set up to post to bugzilla every time I updated something. Since I was merging several weeks of one branch to another this touched hundreds of bugs, sending bugmail to untold numbers of people. Mercifully, Bugzilla cut this off after some limit was reached. It was so embarrassing I could feel myself turning beet red as I thought of how many people just saw my mistake and wondered what the heck I was doing. And yet just had to forge onwards, fix my config file, and try it again. Super nicely, Clint Talbert told me that the first time he tried pushing some change he broke all branches of every product and had no idea what had happened. Little did he know I would blog about his sad story to make myself feel less silly….. That was years ago and I think Mozilla was still using cvs at that point! I merged mozilla-central to holly again, did not break anything this time, and watched the tests run and gradually appear across the screen. Very cool.

I also ended up realizing that the changes to mozharness to turn these batches of tests on again were not super obvious and to figure it out I needed to read a 2000-line configuration file which has somewhat byzantine logic. I’m not judging it, it is clearly something that has grown organically over time and someone else probably in release engineering is an expert on it and can tweak it casually to do whatever is needed.

Back to our story. For a batch of tests on holly that have failures and thus are showing up on treeherder as orange, it should be possible to go through the logs for the failing tests, figure out how to turn them off with some skip-if statements, filing bugs for each skipped failing test. Then, keep doing that till a batch of tests is green and it is ready to be moved over.

That seems like a reasonable plan for improving the automated test landscape, which should help developers to know that their code works in Firefox whether e10s is enabled or not. In effect, having the tests should mean that many problems are prevented from ever becoming bugs. The effect of this test coverage is hard to measure. How do you prove something didn’t happen? Perhaps by looking at which e10s tests fail on pushes to the try server. Another issue here is that there are quite a lot of tests that have been around for many years. It is hard too know how many of them are useful, whether there are a lot of redundant tests, in short whether there is a lot of cruft and there probably is. With a bit more experience in the code and fixing and writing tests it would be easier to judge the usefulness of these tests.

I can also see that, despite this taking a while to figure out (and to even begin to explain) it is a good entry point to contributing to Firefox. It has more or less finite boundaries. If you can follow what I just described in this post and my last post, and you can read and follow a little python and javascript, then you can do this. And, if you were to go through many of the tests, over time you would end up understanding more about how the codebase is structured.

As usual when I dive into anything technical at Mozilla, I think it’s pretty cool that most of this work happens in the open. It is a great body of data for academics to study, it’s an example of how this work actually happens for anyone interested in the field, and it’s something that anyone can contribute to if they have the time and interest to put in some effort.

This post seems very plain with only some screenshots of Treeherder for illustration. Here, have a photo of me making friends with a chicken.

Liz with chicken

Running mochitests for Firefox

January 2, 2015 Liz

I’m experimenting with automated testing for Firefox and figured it may be useful to record what I learned. I had a look at the Mochitest page on MDN as well as the main page on Automated testing at Mozilla. It is hard to know how to even begin to explain this. Mochitests are a huge ball of tests for Firefox. They run every time a change is pushed to mozilla-central, which is the sort of tip of the current state of our code and is used every day to build the Nightly version of Firefox. They’re run automatically for changes on other code repositories too. And, you can run them locally on your own version of Firefox.

This is going to have ridiculous levels of detail and jargon. Warning!

The first thing to do is to download the current code from mozilla-central and build it on my laptop. Here are the Firefox build instructions!

As usual I need to do several other things before I can do those things. This means hours of twiddling around on the command line, installing things, trying different configurations, fixing directory permissions and so on. Here are a few of the sometimes non-trivial things I ended up doing:

* updated Xcode and command line tools
* ran brew doctor and brew update, fixed all errors with much help from Stack Overflow, ended up doing a hard reset of brew
* Also, if you need to install a specific version of a utility, for example, autoconf: brew tap homebrew/versions; brew install autoconf213
* re-installed mercurial and git since they were screwed up somehow from a move from one Mac to another
* tried two different sample .mozconfig files, read through other Mac build config files, several layers deep (very confusing)
* updating my Firefox mozilla-central directory (hg pull -u)
* filed a bug for a build error and fixed some minor points on MDN

The build takes around an hour the first time. After that, pulling the changes from mozilla-central and reticulating the splines takes much less time.

Firefox build

Now I’m to the point where I can have a little routine every morning:
* brew doctor
* brew upgrade
* cd mozilla-central, hg pull -u
* ./mach build

Then I’m set up to run tests. Running all the mochitest-plain tests takes a long time. Running a single test may fail because it has dependencies on other tests it expects to have run first. You can also run all the tests in a particular directory, which may work out better than single tests.

Here is an example of running a single test.

./mach mochitest-browser browser/base/content/test/general/browser_aboutHome.js

Your Nightly or Nightly-debug browser will open and run through some tests. There will be a ton of output. Nifty.

Here is that same test, run with e10s enabled.

./mach mochitest-browser —e10s browser/base/content/test/general/browser_aboutHome.js

BTW if you add “2>&1 | tee -a test.log” to those commands they will pipe the output into a log file.

Back to testing. I poked around to see if I could find a super easy to understand test. The first few, I read through the test code, the associated bugs, and some other stuff. A bit overwhelming. My coworker Juan and I then talked to Joel Maher who walked us through some of the details of how mochitests work and are organized. The tests are scattered throughout the “tree” of directories in the code repository. It is useful generally to use DXR to search but I also ended up just bouncing around and getting familiar with some of the structure of where things are. For example, scarily, I now know my way to testing/mochitest/tests/SimpleTest/. Just by trying different things and looking around you start to get familiar.

Meanwhile, my goal was still to find something easy enough to grasp in an afternoon and run through as much of the process to fix a simple bug as I could manage. I looked around for tests that are known not to work under e10s, and are marked in manifest files that they should be skipped if you’re testing with e10s on. I tried turning some of these tests on and off and reading through their bugs.

Also meanwhile I asked for commit access level 1 (for the try server) so when I start changing and fixing things I can at least throw them at a remote test server as well as my own local environment’s tests.

Then, hurray, Joel lobbed me a very easy test bug.

From reading his description and looking at the html file it links to, I got that I could try this test, but it might not fail. The failure was caused by god knows what other test. Here is how to try it on its own:

./mach mochitest-chrome browser/devtools/webide/test/test_zoom.html

I could see the test open up Nightly-debug and then try to zoom in and out. Joel had also described how the test opens a window, zooms, closes the window, then opens another window, zooms, but doesn’t close. I have not really looked at any JavaScript for several years and it was never my bag. But I can fake it. Hurray, this really is the simplest possible example. Not like the 600-line things I was ending up in at random. Danny looked at the test with me and walked through the JavaScript a bit. If you look at the test file, it is first loading up the SimpleTest harness and some other stuff. I did not really read through the other stuff. *handwave* Then in the main script, it says that when the window is loaded, first off wait till we really know SimpleTest is done because the script tells us so (Maybe in opposition to something like timing out.) Then a function opens the WebIDE interface, and (this is the main bit Danny explained) the viewer= winQuerInterface etc. bit is an object that shows you the state of the viewer. *more handwaving, I do not need to know* More handwaving about “yield” but I get the idea it is giving control over partly to the test window and partly keeping it. Then it zooms in a few times, then this bit is actually the meat of the test:

 is(roundZoom, 0.6, "Reach min zoom");

Which is calling the “is” function in SimpleTest.js, Which I had already been reading, and so going back to it to think about what “is” was doing was useful. Way back several paragraphs ago I mentioned looking in testing/mochitest/tests/SimpleTest/. That is where this function lives. I also felt I did not entirely need to know the details of what the min and max zoom should be. Then, we close the window. Then open a new one. Now we see the other point of this test. We are checking to see that when you open a WebIDE window, then zoom to some zoomy state, then close the window, then re-open it, it should stay zoomed in or out to the state you left it in.

OK, now at this point I need to generate a patch with my tiny one line change. I went back to MDN to check how to do this in whatever way is Mozilla style. Ended up at How to Submit a Patch, then at How can I generate a patch for somebody else to check in for me?, and then messing about with mq which is I guess like Quilt. (Quilt is a nice name, but, welcome to hell.) I ended up feeling somewhat unnerved by mq and unsure of what it was doing. mq, or qnew, did not offer me a way to put a commit message onto my patch. After a lot of googling… not sure where I even found the answer to this, but after popping and then re-pushing and flailing some more, and my boyfriend watching over my shoulder and screeching “You’re going to RUIN IT ALL” (and desperately quoting Kent Beck at me) as I threatened to hand-edit the patch file, here is how I added a commit message:

hg qrefresh -m "Bug 1116802: closes the WebIDE a second time"

Should I keep using mq? Why add another layer into mercurial? Worth it?

To make sure my patch didn’t cause something shocking to happen (It was just one line and very simple, but, famous last words….) I ran the chrome tests that were in the same directory as my buggy test. (Piping the output into a log file and then looked for anything that mentioned test_zoom. ) The test output is a study in itself but not my focus right now as long as nothing says FAIL.

./mach mochitest-chrome browser/devtools/webide/test/

Then I exported the patch still using mq commands which I cannot feel entirely sure of.

 hg export qtip > ~/bug-1116802-fix.patch

That looked fine and, yay, had my commit message. I attached it to the bug and asked Joel to have a look.

I still don’t have a way to make the test fail but it seems logical you would want to close the window.

That is a lot of setup to get to the point where I could make a useful one line change in a file! I feel very satisfied that I got to that point.

Only 30,000 more tests to go, many of which are probably out of date. As I contemplate the giant mass of tests I wonder how many of them are useful and what the maintenance cost is and how to ever keep up or straighten them out. It’s very interesting!

You can have a look at the complicated nature of the automated tests that run constantly to test Firefox at treeherder. For any batch of commits merged into mozilla-central, huge numbers of tests run on many different platforms. If you look at treeherder you can see a little (hopefully green) “dt” among the tests. I think that the zooming in WebIDE test that I just described is in the dt batch of tests (but I am not sure yet).

I hope describing the process of learning about this small part of Firefox’s test framework is useful to someone! I have always felt that I missed out by not having a college level background in CS or deep expertise in any particular language. And yet I have still been a developer on and off for the last 20 years and can jump back into the pool and figure stuff out. No genius badge necessary. I hope you can see that actually writing code is only one part of working in this kind of huge, collaborative environment. The main skill you need (like I keep saying) is the ability not to freak out about what you don’t know, and keep on playing around, while reading and learning and talking with people.

Thoughts on working at Mozilla and the Firefox release process

December 1, 2014 Liz

Some thoughts on my life at Mozilla as I head into our company-wide work week here in Portland! My first year at Mozilla I spent managing the huge volume of bugs, updating docs on how to triage incoming bugs and helping out with Bugzilla itself. For my second year I’ve been more closely tied into the Firefox release process, and I switched from being on the A-Team (Automation and Tools) to desktop Firefox QA, following Firefox 31 and then Firefox 34 through their beginning to release.

picard saying if it's not in bugzilla it doesn't exist

I also spent countless hours fooling around with Socorro, filing bugs on the highest volume crash signatures for Firefox, and then updating the bugs and verifying fixes, especially for startup crashes or crashes associated with “my” releases.

Over this process I’ve really enjoyed working with everyone I’ve met online and in person. The constant change of Mozilla environments and the somewhat anarchic processes are completely fascinating. Though sometimes unnerving. I have spent these two years becoming more of a generalist. I have to talk with end users, developers on every Mozilla engineering team, project or product managers if they exist, other QA teams, my own Firefox/Desktop QA team, the people maintaining all the tools that all of us use, release management, and release engineering. Much of the actual QA has been done by our Romanian team so I have coordinated a lot with them and hope to meet them all some day. When I need to dig into a bug or feature and figure out how it works, it means poking around in documentation or in the code or talking with people, then documenting whatever it is.

This is a job (and a workplace) suited for people who can cope with rapid change and shifting ground. Without a very specific focus, expertise is difficult. People complain a lot about this. But I kind of like it! And I’m constantly impressed by how well our processes work and what we produce. There are specific people who I think of as total rock stars of knowing things. Long time experts like bz or dbaron. Or dmajor who does amazing crash investigations. I am in their secret fan club. If there were bugzilla “likes” I would be liking up a storm on there! It’s really amazing how well the various engineering teams work together. As we scramble around to improve things and as we nitpick I want to keep that firmly in mind. And, as we are all a bit burned out from dealing with the issues from the 33 Firefox release and many point releases, then the 10th anniversary release, then last minute scrambling to incorporate surprise/secret changes to Firefox 34 because of our new deal with Yahoo.

I was in a position to see some of the hard work of the execs, product and design people, and engineers as well as going through some rounds of iterating very quickly with them these last 2 weeks (and seeing gavin and lmandel keep track of that rapid pace) Then seeing just part of what the release team needed to do, and knowing that from their perspective they also could see what IT and infrastructure teams had to do in response. So many ripples of teamwork and people thinking things through, discussing, building an ever-changing consensus reality.

I find that for every moment I feel low for not knowing something, or making a mistake (in public no less) there are more moments when I know something no one else knows in a particular context or am able to add something productive because I’m bringing a generalist perspective that includes my years of background as a developer.

As an author and editor I am reminded of what it takes to edit an anthology. My usual role is as a general editor with a vision: putting out a call, inviting people to contribute, working back and forth with authors, tracking all the things necessary (quite a lot to track, more than you’d imagine!) and shifting back and forth from the details of different versions of a story or author bios, to how the different pieces of the book fit together. And in that equation you also, if you’re lucky, have a brilliant and meticulous copy editor. At Aqueduct Press I worked twice with Kath Wilham. I would have gone 10 rounds of nitpick and Kath would still find things wrong. My personal feeling for my past year’s work is that my job as test plan lead on a release has been 75% “editor” and 25% copy editor.

If I had a choice I would always go with the editor and “glue” work rather than the final gatekeeper work. That last moment of signing off on any of the stages of a release freak me out. I’m not enough of a nitpicker and in fact, have zero background in QA — instead, 20 years as a lone wolf (or nearly so) developer, a noisy, somewhat half-assed one who has never *had* QA to work with much less *doing* QA. It’s been extremely interesting, and it has also been part of my goal for working in big teams. The other thing I note from this stressful last 6 weeks is that, consistent with other situations for me, in a temporary intense “crisis” mode my skills shine out. Like with the shifting-every-30-minutes disaster relief landscape, where I had great tactical and logistical skills and ended up as a good leader. My problem is that I can’t sustain that level of awareness, productivity, or activeness, both physically and mentally, for all that long. I go till I drop, crash & burn. The trick is knowing that’s going to happen, communicating it beforehand, and having other people to back you up. The real trick which I hope to improve at is knowing my limits and not crashing and burning at all. On the other hand, I like knowing that in a hard situation I have this ability to tap into. It just isn’t something I can expect to do all the time, and isn’t sustainable. One thing I miss about my “old” life is building actual tools. These last 2 years have been times of building human and institutional infrastructure for me much more than making something more obvious and concrete. I would like to write another book and to build a useful open source tool of some sort. Either for Mozilla, for an anti-harassment tool suite, or for Ingress…. 🙂 Alternatively I have a long term idea about open source hardware project for mobility scooters and powerchairs. That may never happen, so at the least, I’m resolving to write up my outline of what should happen, in case someone else has the energy and time to do it.

In 2013 and 2014 I mentored three interns for the GNOME-OPW project at Mozilla. Thanks so much to Tiziana Selitto, Maja Frydrychowicz, and Francesca Ciceri for being awesome to work with! As well as the entire GNOME-OPW team at Mozilla and beyond. I spoke about the OPW projects this summer at Open Source Bridge and look forward to more work as a mentor and guide in the future.

Meanwhile! I started a nonprofit in my “spare time”! It’s Double Union, a feminist, women-only hacker and maker space in San Francisco and it has around 150 members. I’m so proud of everything that DU has become. And I continue my work on the advisory boards of two other feminist organizations, Ada Initiative and GimpGirl as well as work in the backchannels of Geekfeminism.org. I wrote several articles for Model View Culture this year and advised WisCon on security issues and threat modeling. I read hundreds of books, mostly science fiction, fantasy, and history. I followed the awesome work my partner Danny does with EFF, as well. These things are not just an important part of my life, they also make my work at Mozilla more valuable because I am bringing perspectives from these communities to the table in all my work.

The other day I thought of another analogy for my last year’s work at Mozilla that made me laugh pretty hard. I feel personally like the messy “glue” Perl scripts it used to be my job to write to connect tools and data. Part of that coding landscape was because we didn’t have very good practices or design patterns but part of it I see as inevitable in our field. We need human judgement and routing and communication to make complicated systems work, as well as good processes.

I think for 2015 I will be working more closely with the e10s team as well as keeping on with crash analysis and keeping an eye on the release train.

Mad respect and appreciation to everyone at Mozilla!!

Firefox launch party liz larissa in fox ears

How to test new features in Firefox 34 Aurora

September 11, 2014 Liz

If you’re a fan of free and open source software and would like to contribute to Firefox, join me for some Firefox feature testing!

There are some nifty features under development right now for Firefox 34 including translation in the browser, making voice or video calls (a feature called “Hello” or “Loop”), debugging information for web developers in the Dev Tools Inspector, and recent improvements to HTML5 gaming.

I’ve written step by step instructions on these
ways to test Firefox 34. If you would like to see what it’s like to improve a popular open source project, trying out these tasks is a good introduction.

Aurora

First, Install the Aurora version of Firefox. It is best to set it up to use multiple profiles. That ensures you don’t use your everyday version of Firefox for testing, so you won’t risk losing your usual profile information. It also makes it easy to restart Firefox with a new, clean profile with all the default settings, very useful for testing. Sometimes I realize I’m running 5 different versions of Firefox at once!

To test “Hello”, try making some voice or video calls from Firefox Aurora. You will need a friend to test with. Or, use two computers that you control. This is a good task to try while joining our chat channels, #qa or #testday on irc.mozilla.org; ask if anyone there wants to test Hello with you. The goal here is mostly to find and report new bugs.

If you test the translation infobar in Aurora you may find some new bugs. This is a fun feature to test. I like trying it on Wikipedia in many different languages, and also looking at newspapers!

If you’re a web developer, you may use Developer Tools in Firefox. I’m asking Aurora users to go through some unconfirmed bug reports, to help improve the Developer Tools Inspector.

If you like games you can test HTML5 web-based games in Firefox Aurora. This helps us improve Firefox and also helps the independent game developers. We have a list of demo games so you can play them, report glitches, and feel like a virtuous open source citizen all at once. Along the way you have opportunities to learn some interesting stuff about how graphics on the web can work (or not work).

Monster madness

These testing tasks are all set up in One and Done, Mozilla QA’s site to start people along the path to joining our open source community. This site was developed with a lot of community contribution including the design and concept by long-time community member Parul and a lot of code by two interns this summer, Pankaj and Maja.

Testing gives a great view into the development process for people who may not (yet) be programmers. I especially love how transparent Mozilla’s process can be. Anyone can report a bug, visible to the entire world in bugzilla.mozilla.org. There are many people watching that incoming stream of bug reports, confirming them and routing them to developer teams, sometimes tagging them as good first bugs for new contributors. Developers who may or may not be Mozilla employees show up in the bugs, like magic . . . if you think of bugmail notifications as magic . . .

It is amazing to see this very public and somewhat anarchic collaboration process at work. Of course, it can also be extremely satisfying to see a bug you discovered and reported, your pet bug, finally get fixed.

Screen reader and accessibility bug day

March 11, 2013 Liz

Tomorrow Mozilla is hosting a screen reader and general accessibility bug day.

Len Burns and I have invited screen reader users of Firefox and other Mozilla products to join us in sorting through existing accessibility bugs. Some folks from the Mozilla Accessibility team will be on hand to talk with us.

I’m pretty sure we will also collect some new bugs along the way!

I hope that people will make new connections, and that we can attract a wider accessibility bugmaster team to do ongoing work with Mozilla’s existing developers and a11y experts.

The screen reader landscape for web access is fairly complicated. For example, here are the Firefox Gecko docs for Windows accessibility vendors which explain the relationship between the DOM tree and Microsoft’s accessibility API. Common screen reader software includes NVDA, Window-Eyes, JAWS, and Orca.

Orca2 sm

If you would like a quick overview of common web access issues, look through Aaron Leventhal’s presentation. I like it because he includes some political dimensions and context for accessibility.

So far, the “bugmaster” bug days have been combined with QA’s efforts. I’m hoping to also hold focused bug days like this one, in cooperation with various teams across Mozilla. As we gather more templates for bug managing and triage, I hope we’ll coalesce bugmaster teams with expertise in particular areas. And we can repeat topic-focused bug days periodically.

If you are interested in web accessibility or if you use a screen reader, please drop by on Tuesday and say hello in the #accessibility channel on irc.mozilla.org.

You can also add the name you go by, your irc handle, your contact infomation, or anything else you use to identify yourself, on the wiki page for the bug day under Participants: https://wiki.mozilla.org/Bugmasters/Bug_days/a11y#Participants .

The goals of the screen reader bug day are to improve everyone’s experience of Firefox and other Mozilla tools. We would like for everyone to be able to access the web smoothly. Through collecting more information on accessibility bugs, we hope to connect committed technical users with accessibility developers, and make our community better and more powerful. Our bugmaster work should help to make developers’ work easier. That way they can spend more time fixing stuff.

If you have any questions or feedback, please feel free to email me at lhenry@mozilla.org.

In setting up this event I tried to make sure that the tools we are using are as accessible as possible. Etherpad, which Mozilla teams often use for bug triage events, is not useable for screen readers. The wiki.mozilla.org pages seem readable though editing may be more of a challenge. IRC feels like our best bet for good communication. I also went through about 100 screen-reader-related bugs and emailed the bug reporters and commenters to send them invitations to participate.

Len is particularly interested in developing a plan for Thunderbird and screen reader vendors. If you share this interest I recommend joining the mailing list tb-planning.

Here is what Len has to say on Thunderbird accessibility:

It is complicated, because the issues are really between the two major screen reader vendors and Mozilla. Meaning, that the solution would need to be a cooperative one. Because the screen reader vendors perceive, right or wrong, that little more than security bugs are being updated in Thunderbird, they do not seem terribly motivated. I am not quite sure where to take it.

Unless I could convince the vendors that solving these issues are worthy of their time, I am a bit stuck.

I would definitely be willing to raise the specifics with both vendors if I could give them some reason to encourage a belief that there is a mutual interest in improving things. When I have raised several with GW Micro I hear things like: This has been filed for over a year with no response, and the like.

Those of us using screen readers are currently in quite a pickle regarding email. The choices are quite limited. A lot of us have been using Thunderbird for some time, but when things reach a point where you are spending too much time trying to send a simple email, something must give.

I can also tell you that what finally tipped me were problems between the composition screen, and other open apps on-screen. If I were going to compose an email in TBird right now, I would have to be sure that Skype was minimized, MirandaIM was minimized, etc. If I did not, I would be likely to encounter a range of strange behaviors in the edit window such as being unable to read back the text I am writing, inaability to use my backspace, format distortion, and more.

What has been slowing me down on these issues was a lack of knowing avenues of pursuit. The challenge will be convincing vendors that investing time is a benefit. My position is that I am not sure, but, we have a good chance of catalyzing and contributing to change and possible strengthened relationships.

Len has been a professional system administrator, coder and web accessibility consultant since the Internet was a kinder and gentler place. He makes his living these days free lancing as a web accessibility consultant for colleges and universities and coding the back-end glue of web sites for small to mid-sized businesses.

Thanks to Len for his insights on web dev, email, and access in the last couple of weeks as well as his outreach efforts to talk with vendors, software users, and developers!