mozilla – Composite

Bad tests sink ships, or something

November 3, 2016 Liz

As a connoisseur of the caution sign, I really enjoyed these gorgeous, strange, slightly disturbing safety posters from WWII-era Britain. There are stylized warnings about driving at night, about putting an eye out with that thing, about what happens if you are very imprudent with boxes, giant boards, and ladders, and then a jolly man who is ominously having way too much fun with a sort of compressed air power gun.

http://www.slate.com/blogs/the_vault/2016/09/16/modernist_british_safety_posters_from_the_middle_of_the_twentieth_century.html

Here is my new version of the “I didn’t protect my eyes” poster specially for Firefox developers,

I Didnt push to try

And another very silly new poster for developers – based on World War II’s finest modernist art –

Make sure it has tests

Really, I do love caution signs. They’re so earnest. They try very hard to be persuasive! Here are some from my Flickr collection of signs. This beautiful, detailed image of someone falling through rotten boards on a pier and then drowning helplessly is from the long pier at Aquatic Park in San Francisco.

Caution dont fall off the pier

This last one is from an old artillery testing range in Essex that you have to drive through before you can get to the incredibly dangerous and fascinating place, The Broomway. The Broomway is a submerged and mostly unmarked 6 mile long path through ocean quicksand, only usable at low tide. Did I mention the unexploded ordnance, the rapidly advancing tide, and the frequent heavy fog? Also, did I mention there is a tour that takes you over The Broomway on a giant tractor?! Sign me up!

Do not approach or touch any object or debris it may explode and kill you

That Zarro Boogs feeling

April 20, 2016 Liz

This is my third Firefox release as release manager, and the fifth that I’ve followed closely from the beginning to the end of the release cycle. (31 and 36 as QA lead; 39, 43, and 46 as release manager.) This time I felt more than usually okay with things, even while there was a lot of change in our infrastructure and while we started triaging and following even more bugs than usual. No matter how on top of things I get, there is still chaos and things still come up at the last minute. Stuff breaks, and we never stop finding new issues!

I’m not going into all the details because that would take forever and would mostly be me complaining or blaming myself for things. Save it for the post-mortem meeting. This post is to record my feeling of accomplishment from today.

During the approximately 6 week beta cycle of Firefox development we release around 2 beta versions per week. I read through many bugs nominated as possibly important regressions, and many that need review and assessment to decide if the benefit of backporting warrants the risk of breaking something else.

During this 7 week beta cycle I have made some sort of decision about at least 480 bugs. That usually means that I’ve read many more bugs, since figuring out what’s going on in one may mean reading through its dependencies, duplicates, and see-alsos, or whatever someone randomly mentions in comment 45 of 96.

And today I got to a point I’ve never been at near the end of a beta cycle: Zarro Boogs found!

This is what Bugzilla says when you do a query and it returns 0. I think everyone likes saying (and seeing) “Zarro Boogs”. Its silliness expresses the happy feeling you get when you have burned down a giant list of bugs.

This particular query is for bugs that anyone at all has nominated for the release management team to pay attention to.

Here is the list of requests for uplift (or backporting, same thing) to the mozilla-beta repo:

Yes!! Also zarro boogs.

Since we build our release candidate a week (or a few days) from the mozilla-release repo, I check up on requests to uplift there too:

PEAK ZARRO BOOGS.

For the bugs that are unresolved and that I’m still tracking into the 46 release next week, it’s down to 4: Two fairly high volume crashes that may not be actionable yet, one minor issue in a system addon that will be resolved in a planned out-of-band upgrade, and one web compatibility issue that should be resolved soon by an external site. Really not bad!

Our overall regression tracking has a release health dashboard on displays in many Mozilla offices. Blockers, 0. Known new regressions that we are still working on and haven’t explicitly decided to wontfix: 1. (But this will be fixed by the system addon update once 46 ships.) Carryover regressions: 41; about 15 of them are actually fixed but not marked up correctly yet. The rest are known regressions we shipped with already that still aren’t fixed. Some of those are missed uplift opportunities. We will do better in the next release!

In context, I approved 196 bugs for uplift during beta, and 329 bugs for aurora. And, we fix several thousands of issues in every release during the approx. 12 week development cycle. Which ones of those should we pay the most attention to, and which of those can be backported? Release managers act as a sort of Maxwell’s Demon to let in only particular patches …

Will this grim activity level for the past 7 weeks and my current smug feeling of being on top of regression burndown translate to noticeably better “quality”… for Firefox users? That is hard to tell, but I feel hopeful that it will over time. I like the feeling of being caught up, even temporarily.

Here I am with drink in hand on a sunny afternoon, toasting all the hard working developers, QA testers, beta users, release engineers, PMs, managers and product folks who did most of the actual work to fix this stuff and get it firmly into place in this excellent, free, open source browser. Cheers!

Trip to Whistler for Mozilla's work week

June 22, 2015 Liz

Our work week hasn’t started yet, but since I got to Whistler early I have had lots of adventures.

First the obligatory nostril-flaring over what it is like to travel with a wheelchair. As we started the trip to Vancouver I had an interesting experience with United Airlines as I tried to persuade them that it was OK for me to fold up my mobility scooter and put it into the overhead bin on the plane. Several gate agents and other people got involved telling me many reasons why this could not, should not, and never has or would happen:

* It would not fit
* It is illegal
* The United Airlines handbook says no
* The battery has to go into the cargo hold
* Electric wheelchairs must go in the cargo hold
* The scooter might fall out and people might be injured
* People need room for their luggage in the overhead bins
* Panic!!

The Air Carrier Access Act of 1986 says,

Assistive devices do not count against any limit on the number of pieces of carry-on baggage. Wheelchairs and other assistive devices have priority for in-cabin storage space over other passengers’ items brought on board at the same airport, if the disabled passenger chooses to preboard.

In short I boarded the airplane, and my partner Danny folded up the scooter and put it in the overhead bin. Then, the pilot came out and told me that he could not allow my battery on board. One of the gate agents had told him that I have a wet cell battery (like a car battery). It is not… it is a lithium ion battery. In fact, airlines do not allow lithium batteries in the cargo hold! The pilot, nicely, did not demand proof it is a lithium battery. He believed me, and everyone backed down.

The reason I am stubborn about this is that I specially have a very portable, foldable electric wheelchair so that I can fold it up and take it with me. Two times in the past few years, I have had my mobility scooters break in the cargo hold of a plane. That made my traveling very difficult! The airlines never reimbursed me for the damage. Another reason is that the baggage handlers may lose the scooter, or bring it to the baggage pickup area rather than to the gate of the plane.

Onward to Whistler! We took a shuttle and I was pleasantly (and in a way, sadly) surprised that the shuttle liason, and the driver, both just treated me like any other human being. What a relief! It is not so hard! This experience is so rare for me that I am going to email the shuttle company to compliment them and their employees.

The driver, Ivan, took us through Vancouver, across a bridge that is a beautiful turquoise color with stone lions at its entrance, and through Stanley Park. I particularly noticed the tiny beautiful harbor or lagoon full of boats as we got off the bridge. Then, we went up Highway 99, or the Sea to Sky Highway, to Squamish and then Whistler.

Sea to sky highway

When I travel to new places I get very excited about the geology and history and all the geography! I love to read about it beforehand or during a trip.

The Sea to Sky Highway was improved in preparation for the Winter Olympics and Paralympics in 2010. Before it was rebuilt it was much twistier with more steeply graded hills and had many bottlenecks where the road was only 2 lanes. I believe it must also have been vulnerable to landslides or flooding or falling rocks in places. As part of this deal the road signs are bilingual in English and Squamish. I read a bit on the way about the ongoing work to revitalize the Squamish language.

The highway goes past Howe Sound, on your left driving up to Squamish. It is a fjord, created by retreated glaciers around 11,000 years ago. Take my geological knowledge with a grain of salt (or a cube of ice) but here is a basic narrative of the history. AT some point it was a shallow sea here but a quite muddy one, not one with much of a coral reef system, and the mountains were an archipelago of island volcanoes. So there are ocean floor sediments around, somewhat metamorphosed; a lot of shale.

There is a little cove near the beginning of the highway with some boats and tumble-down buildings, called Porteau Cove. Interesting history there. Then you will notice a giant building up the side of a hill, the Britannia Mining Museum. That was once the Britannia Mines, producing billions of dollars’ worth of copper, gold, and other metals. The entire hill behind the building is honeycombed with tunnels! While a lot of polluted groundwater has come out of this mine damaging the coast and the bay waters, it was recently plugged with concrete: the Millenium Plug, and that improved water quality a lot, so that shellfish, fish, and marine mammals are returning to the area. The creek also has trout and salmon returning. That’s encouraging!

Then you will see huge granite cliffs and Shannon Falls. The giant monolith made me think of El Capitan in Yosemite. And also of Enchanted Rock, a huge pink granite dome in central Texas. Granite weathers and erodes in very distinctive ways. Once you know them you can recognize a granite landform from far away! I haven’t had a chance to look close up at any rocks on this trip…. Anyway, there is a lot of granite and also basalt or some other igneous extrusive rock. Our shuttle driver told me that there is columnar basalt near by at a place called French Fry Hill.

The mountain is called Stawamus Chief Mountain. Squamish history tells us it was a longhouse turned to stone by the Transformer Brothers. I want to read more about that! Sounds like a good story! Rock climbers love this mountain.

There are some other good stories, I think one about two sisters turned to stone lions. Maybe that is why there are stone lions on the Vancouver bridge.

The rest of the drive brought us up into the snowy mountains! Whistler is only 2000 feet above sea level but the mountains around it are gorgeous!

The “village” where tourists stay is sort of a giant, upscale, outdoor shopping mall with fake streets in a dystopian labyrinth. It is very nice and pretty but it can also feel, well, weird and artificial! I have spent some time wandering around with maps, backtracking a lot when I come to dead ends and stairways. I am also playing Ingress (in the Resistance) so I have another geographical overlay on the map.

Whistler bridge lost lake

On Sunday I got some groceries and went down paved and then gravel trails to Lost Lake. It was about an hour long trip to get there. The lake was beautiful, cold, and full of people sunbathing, having picnics, and swimming. Lots of bikes and hikers. I ran out of battery (nearly), then realized that the lake is next to a parking lot. I got a taxi back to the Whistler Village hotel! Better for me anyway since the hour long scooter trip over gravel just about killed me (I took painkiller halfway there and then was just laid flat with pain anyway.) Too ambitious of an expedition, sadly. I had many thoughts about the things I enjoyed when I was younger (going down every trail, and the hardest trails, and swimming a lot) Now I can think of those memories, and I can look at beautiful things and also read all the information about an area which is enjoyable in a different way. This is just how life is and you will all come to it when you are old. I have this sneak preview…. at 46…. When I am actually old, I will have a lot of practice and will be really good at it. Have you thought about what kind of old person you would like to be, and how you will become that person?

Today I stayed closer to home just going out to Rebagliati Park. This was fabulous since it wasn’t far away, seriously 5 minutes away! It was very peaceful. I sat in a giant Adirondack chair in a flower garden overlooking the river and a covered bridge. Watching the clouds, butterflies, bees, birds, and a bear! And of course hacking the portals (Ingress again). How idyllic! I wish I had remembered to bring my binoculars. I have not found a shop in the Whistler Mall-Village that stocks binoculars. If I find some, I will buy them.

I also went through about 30 bugs tracked for Firefox 39, approved some for uplift, wontfixed others, emailed a lot of people for work, and started the RC build going. Releng was heroic in fixing some issues with the build infrastructure! But, we planned for coverage for all of us. Good planning! I was working Sunday and Monday while everyone else travelled to get here…. Because of our release schedule for Firefox it made good sense for me to get here early. It also helps that I am somewhat rested from the trip!

I went to the conference center, found the room that is the home base for the release management and other platform teams, and got help from a conference center setup guy to lay down blue tape on the floor of the room from the doorway to the back of the room. The tape marks off a corridor to be kept clear, not full of backpacks or people standing and talking in groups, so that everyone can freely get in and out of the room. I hope this works to make the space easy for me to get around in, in my wheelchair, and it will surely benefit other people as well.

Travel lane

At this work week I hope to learn more about what other teams are doing, any cool projects etc, especially in release engineering and in testing and automated tools and to catch up with the Bugzilla team too. And will be talking a bunch about the release process, how we plan and develop new Firefox features, and so on! Looking forward now to the reception and seeing everyone who I see so much online!

A useful Bugzilla trick

February 27, 2015 Liz

At the beginning of February I changed teams within Mozilla and am now working as a release manager. It follows naturally from a lot of the work I’ve already been doing at Mozilla and I’m excited to join the team working with Lukas, Lawrence, and Sylvestre!

I just learned a cool trick for dealing with several bugzilla.mozilla.org bugs at once, on MacOS X.

1) Install Bugzilla Services.

2) Add a keyboard shortcut as Alex Keybl describes in the blog post above. (I am using Control-Command-B)

3) Install the Tree Style Tab addon.

Now, from any text, whether in email, a desktop text file, or anywhere in the browser, I can highlight a bunch of text and bug number will be parsed out of the text. For example, from an email this morning:

Bug 1137050 - Startup up Crash - patch should land soon, potentially risky
David Major seems to think it is risky for the release.

Besides that, we are going to take:
Bug 1137469 - Loop exception - patch waiting for review
Bug 1136855 - print preferences - patch approved
Bug 1137141 - Fx account + hello - patch waiting for review
Bug 1136300 - Hello + share buttons - Mike  De Boer will work on a patch today

And maybe a fix for the ANY query (bug 1093983) if we have one...

I highlighted the entire email and hit the “open in bugzilla” keystroke. This resulted in a Bugzilla list view for the 6 bugs mentioned in the email.

Bugzilla list view example

With BugzillaJS installed, I have an extra option at the bottom of the page, “Open All in Tabs”, so if I wanted to triage these bugs, I can open them all at once. The tabs show up in my sidebar, indented from their parent tab. This is handy if I want to collapse this group of tabs, or close the parent tab and all its children at once (The original list view of these 6 bugs, and each of its individual tabs.) Tree Style Tab is my new favorite thing!

Tree style tabs bugzilla

In this case, after I had read each bug from this morning and closed the tabs, my coworker Sylvestre asked me to make sure I cc-ed myself into all of them to keep an eye on them later today and over the weekend so that when fixes are checked in, I can approve them for release.

Here I did not want to open up every bug in its own tab but instead went for “Change Several Bugs at Once” which is also at the bottom of the page.

Bugzilla batch edit

This batch edit view of bugs is a bit scarily powerful since it will result in bugmail to many people for each bug’s changes. When you need it, it’s a great feature. I added myself to the cc: field all in one swoop instead of having to click each tab open, click around several times in each bug to add myself and save and close the tab again.

It was a busy day yesterday at work but I had a nice time working from the office rather than at home. Here is the view from the SF Mozilla office 7th floor deck where I was working and eating cake in the sun. Cannot complain about life, really.
Mozilla bridge view

Automated test harnesses for Firefox, zooming out a bit

February 7, 2015 Liz

In December I was going through some of the steps to be able to change, fix, and interpret the results of a small subset of the gazillion automated tests that Mozilla runs on Firefox builds. My last post arrived at the point of being able to run mochitests locally on my laptop on the latest Firefox code. Over the holidays I dug further into the underlying situation and broaded my perspective. I wanted to make sure that I didn’t end up grubbing away at something that no one cared about or wasn’t missing the big picture.

My question was, how can I, or QA in general, understand where Firefox is with e10s (Electrolysis, the code name for the multiprocess Firefox project). Can I answer the question, are we ready to have e10s turned on by default in Firefox — for the Developer Edition (formerly known as Aurora), for Beta, and for a new release? What criteria are we judging by? Complicated. And given those things, how can we help move the project along and ensure good quality; Firefox that works as well as or, we hope, better than, Firefox with e10s not enabled? My coworker Juan and I boiled it down to basically, stability (lowering the crash rate) and automated test coverage. To answer my bigger questions about how to improve automated test coverage I had to kind of zoom out, and look from another angle. For Firefox developers a lot of what I am about to describe is basic knowledge. It seems worth explaining, since it took me significant time to figure out.

First of all let’s look at treeherder. Treeherder is kind of the new TBPL, which is the old new Tinderbox. It lets us monitor the current state of the code repositories and the tests that run against them. It is a window into Mozilla’s continuous integration setup. A battery of tests are poised to run against Firefox builds on many different platforms. Have a look!

Current view of mozilla-central on treeherder

Treeherder2

Edward Tufte would have a cow. Luckily, this is not for Tufte to enjoy. And I love it. You can just keep digging around in there, and it will keep telling you things. What a weird, complicated gold mine.

Digression! When I first started working for Mozilla I went to the Automation and Tools team work week where they all came up with Treeherder. We wanted to name it something about Ents, because it is about the Tree(s) of the code repos. I explained the whole thing to my son, I think from the work week, which awesomely was in London. He was 11 or 12 at the time and he suggested the name “Yggdrazilla” keeping the -zilla theme and in reference to Yggdrasil, the World-Tree from Norse mythology. My son is pretty awesome. We had to reject that name because we can’t have more -zilla names and also no one would be able to spell Yggdrasil. Alas! So, anyway, treeherder.

The left side of the screen describes the latest batch of commits that were merged into mozilla-central. (The “tree” of code that is used to build Nightly.) On the right, there are a lot of operating systems/platforms listed. Linux opt (optimized version of Firefox for release) is at the top, along with a string of letters and numbers which we hope are green. Those letter and numbers represent batches of tests. You can hover over them to see a description. The tests marked M (1 2 3 etc) are mochitests, bc1 is mochitest-browser-chrome, dt are developer tools tests, and so on. For linux-opt you can see that there are some batches of tests with e10s in the name. We need the mochitest-plain tests to run on Firefox if it has e10s not enabled, or enabled. So the tests are duplicated, possibly changed to work under e10s, and renamed. We have M(1 2 3 . . .) tests, and also M-e10s(1 2 3 . . . ). Tests that are green are all passing. Orange means they aren’t passing. I am not quite sure what red (busted) means (bustage in the tree! red alert!) but let’s just worry about orange. (If you want to read more about the war on orange and what all this means, read Let’s have more green trees from Vaibhav’s blog. )

I kept asking, in order to figure out what needed doing that I could usefully do within the scope of As Soon As Possible, “So, what controls what tests are in which buckets? How do I know how many there are and what they are? Where are they in the codebase? How can I turn them on and off in a way that doesn’t break everything, or breaks it productively?” Good questions. Therefore the answers are long.

There are many other branches of the code other than mozilla-central. Holly is a branch where the builds for all the platforms have e10s enabled. (Many of these repos or branches or twigs or whatever, are named after different kinds of tree.) The tests are the standard set of tests, not particularly tweaked to allow for e10s. We can see what is succeeded and failing on treeherder’s view of holly. A lot of tests are orange on holly! Have a look at holly by clicking through on the link above. Here is a picture of the current state of holly.

Treeherder holly1

If you click a batch of tests where there are some failures — an orange one — then a new panel will open up in treeherder! I will pick a juicy looking one. Right now, for MacOS 10.6 opt, M(2) is orange. Clicking it gives me a ton of info. Scrolling down a bit in the bottom left panel tells me this:

mochitest-plain-chunked 164546/12/14136

The first number is how many tests ran. Scary. Really? 164546 tests ran? Kind of. This is counting assertions, in other words, “is” statements from SimpleTest. The first number lists how many assertions passed. The second number is for assertion failures and the third is for “todo” statements.

The batches of tests running on holly are all running against an e10s build of Firefox. Anything that’s consistently green on holly, we can move over to mozilla-central by making some changes in mozharness. I asked a few people how to do this, and Jim Matthies helpfully pointed me at a past example in Bug 1061014. I figured I could make a stab at adding some of the newly passing tests in Bug 1122901.

As I looked at how to do this, I realized I needed commit access level 2 so I filed a bug to ask for that. And, I also tried merging mozilla-central to holly. That was ridiculously exciting though I hadn’t fixed anything yet. I was just bringing the branch up to date. On my first try doing this, I immediately got a ping on IRC from one of the sheriffs (who do merges and watch the state of the “tree” or code repository) asking me why I had done something irritating and wrong. It took us a bit to figure out what had happened. When I set up mercurial, the setup process and docs told me to install a bunch of Mozilla specific hg extensions. So, I had an extension set up to post to bugzilla every time I updated something. Since I was merging several weeks of one branch to another this touched hundreds of bugs, sending bugmail to untold numbers of people. Mercifully, Bugzilla cut this off after some limit was reached. It was so embarrassing I could feel myself turning beet red as I thought of how many people just saw my mistake and wondered what the heck I was doing. And yet just had to forge onwards, fix my config file, and try it again. Super nicely, Clint Talbert told me that the first time he tried pushing some change he broke all branches of every product and had no idea what had happened. Little did he know I would blog about his sad story to make myself feel less silly….. That was years ago and I think Mozilla was still using cvs at that point! I merged mozilla-central to holly again, did not break anything this time, and watched the tests run and gradually appear across the screen. Very cool.

I also ended up realizing that the changes to mozharness to turn these batches of tests on again were not super obvious and to figure it out I needed to read a 2000-line configuration file which has somewhat byzantine logic. I’m not judging it, it is clearly something that has grown organically over time and someone else probably in release engineering is an expert on it and can tweak it casually to do whatever is needed.

Back to our story. For a batch of tests on holly that have failures and thus are showing up on treeherder as orange, it should be possible to go through the logs for the failing tests, figure out how to turn them off with some skip-if statements, filing bugs for each skipped failing test. Then, keep doing that till a batch of tests is green and it is ready to be moved over.

That seems like a reasonable plan for improving the automated test landscape, which should help developers to know that their code works in Firefox whether e10s is enabled or not. In effect, having the tests should mean that many problems are prevented from ever becoming bugs. The effect of this test coverage is hard to measure. How do you prove something didn’t happen? Perhaps by looking at which e10s tests fail on pushes to the try server. Another issue here is that there are quite a lot of tests that have been around for many years. It is hard too know how many of them are useful, whether there are a lot of redundant tests, in short whether there is a lot of cruft and there probably is. With a bit more experience in the code and fixing and writing tests it would be easier to judge the usefulness of these tests.

I can also see that, despite this taking a while to figure out (and to even begin to explain) it is a good entry point to contributing to Firefox. It has more or less finite boundaries. If you can follow what I just described in this post and my last post, and you can read and follow a little python and javascript, then you can do this. And, if you were to go through many of the tests, over time you would end up understanding more about how the codebase is structured.

As usual when I dive into anything technical at Mozilla, I think it’s pretty cool that most of this work happens in the open. It is a great body of data for academics to study, it’s an example of how this work actually happens for anyone interested in the field, and it’s something that anyone can contribute to if they have the time and interest to put in some effort.

This post seems very plain with only some screenshots of Treeherder for illustration. Here, have a photo of me making friends with a chicken.

Liz with chicken

Thoughts on working at Mozilla and the Firefox release process

December 1, 2014 Liz

Some thoughts on my life at Mozilla as I head into our company-wide work week here in Portland! My first year at Mozilla I spent managing the huge volume of bugs, updating docs on how to triage incoming bugs and helping out with Bugzilla itself. For my second year I’ve been more closely tied into the Firefox release process, and I switched from being on the A-Team (Automation and Tools) to desktop Firefox QA, following Firefox 31 and then Firefox 34 through their beginning to release.

picard saying if it's not in bugzilla it doesn't exist

I also spent countless hours fooling around with Socorro, filing bugs on the highest volume crash signatures for Firefox, and then updating the bugs and verifying fixes, especially for startup crashes or crashes associated with “my” releases.

Over this process I’ve really enjoyed working with everyone I’ve met online and in person. The constant change of Mozilla environments and the somewhat anarchic processes are completely fascinating. Though sometimes unnerving. I have spent these two years becoming more of a generalist. I have to talk with end users, developers on every Mozilla engineering team, project or product managers if they exist, other QA teams, my own Firefox/Desktop QA team, the people maintaining all the tools that all of us use, release management, and release engineering. Much of the actual QA has been done by our Romanian team so I have coordinated a lot with them and hope to meet them all some day. When I need to dig into a bug or feature and figure out how it works, it means poking around in documentation or in the code or talking with people, then documenting whatever it is.

This is a job (and a workplace) suited for people who can cope with rapid change and shifting ground. Without a very specific focus, expertise is difficult. People complain a lot about this. But I kind of like it! And I’m constantly impressed by how well our processes work and what we produce. There are specific people who I think of as total rock stars of knowing things. Long time experts like bz or dbaron. Or dmajor who does amazing crash investigations. I am in their secret fan club. If there were bugzilla “likes” I would be liking up a storm on there! It’s really amazing how well the various engineering teams work together. As we scramble around to improve things and as we nitpick I want to keep that firmly in mind. And, as we are all a bit burned out from dealing with the issues from the 33 Firefox release and many point releases, then the 10th anniversary release, then last minute scrambling to incorporate surprise/secret changes to Firefox 34 because of our new deal with Yahoo.

I was in a position to see some of the hard work of the execs, product and design people, and engineers as well as going through some rounds of iterating very quickly with them these last 2 weeks (and seeing gavin and lmandel keep track of that rapid pace) Then seeing just part of what the release team needed to do, and knowing that from their perspective they also could see what IT and infrastructure teams had to do in response. So many ripples of teamwork and people thinking things through, discussing, building an ever-changing consensus reality.

I find that for every moment I feel low for not knowing something, or making a mistake (in public no less) there are more moments when I know something no one else knows in a particular context or am able to add something productive because I’m bringing a generalist perspective that includes my years of background as a developer.

As an author and editor I am reminded of what it takes to edit an anthology. My usual role is as a general editor with a vision: putting out a call, inviting people to contribute, working back and forth with authors, tracking all the things necessary (quite a lot to track, more than you’d imagine!) and shifting back and forth from the details of different versions of a story or author bios, to how the different pieces of the book fit together. And in that equation you also, if you’re lucky, have a brilliant and meticulous copy editor. At Aqueduct Press I worked twice with Kath Wilham. I would have gone 10 rounds of nitpick and Kath would still find things wrong. My personal feeling for my past year’s work is that my job as test plan lead on a release has been 75% “editor” and 25% copy editor.

If I had a choice I would always go with the editor and “glue” work rather than the final gatekeeper work. That last moment of signing off on any of the stages of a release freak me out. I’m not enough of a nitpicker and in fact, have zero background in QA — instead, 20 years as a lone wolf (or nearly so) developer, a noisy, somewhat half-assed one who has never *had* QA to work with much less *doing* QA. It’s been extremely interesting, and it has also been part of my goal for working in big teams. The other thing I note from this stressful last 6 weeks is that, consistent with other situations for me, in a temporary intense “crisis” mode my skills shine out. Like with the shifting-every-30-minutes disaster relief landscape, where I had great tactical and logistical skills and ended up as a good leader. My problem is that I can’t sustain that level of awareness, productivity, or activeness, both physically and mentally, for all that long. I go till I drop, crash & burn. The trick is knowing that’s going to happen, communicating it beforehand, and having other people to back you up. The real trick which I hope to improve at is knowing my limits and not crashing and burning at all. On the other hand, I like knowing that in a hard situation I have this ability to tap into. It just isn’t something I can expect to do all the time, and isn’t sustainable. One thing I miss about my “old” life is building actual tools. These last 2 years have been times of building human and institutional infrastructure for me much more than making something more obvious and concrete. I would like to write another book and to build a useful open source tool of some sort. Either for Mozilla, for an anti-harassment tool suite, or for Ingress…. 🙂 Alternatively I have a long term idea about open source hardware project for mobility scooters and powerchairs. That may never happen, so at the least, I’m resolving to write up my outline of what should happen, in case someone else has the energy and time to do it.

In 2013 and 2014 I mentored three interns for the GNOME-OPW project at Mozilla. Thanks so much to Tiziana Selitto, Maja Frydrychowicz, and Francesca Ciceri for being awesome to work with! As well as the entire GNOME-OPW team at Mozilla and beyond. I spoke about the OPW projects this summer at Open Source Bridge and look forward to more work as a mentor and guide in the future.

Meanwhile! I started a nonprofit in my “spare time”! It’s Double Union, a feminist, women-only hacker and maker space in San Francisco and it has around 150 members. I’m so proud of everything that DU has become. And I continue my work on the advisory boards of two other feminist organizations, Ada Initiative and GimpGirl as well as work in the backchannels of Geekfeminism.org. I wrote several articles for Model View Culture this year and advised WisCon on security issues and threat modeling. I read hundreds of books, mostly science fiction, fantasy, and history. I followed the awesome work my partner Danny does with EFF, as well. These things are not just an important part of my life, they also make my work at Mozilla more valuable because I am bringing perspectives from these communities to the table in all my work.

The other day I thought of another analogy for my last year’s work at Mozilla that made me laugh pretty hard. I feel personally like the messy “glue” Perl scripts it used to be my job to write to connect tools and data. Part of that coding landscape was because we didn’t have very good practices or design patterns but part of it I see as inevitable in our field. We need human judgement and routing and communication to make complicated systems work, as well as good processes.

I think for 2015 I will be working more closely with the e10s team as well as keeping on with crash analysis and keeping an eye on the release train.

Mad respect and appreciation to everyone at Mozilla!!

Firefox launch party liz larissa in fox ears

How to test new features in Firefox 34 Aurora

September 11, 2014 Liz

If you’re a fan of free and open source software and would like to contribute to Firefox, join me for some Firefox feature testing!

There are some nifty features under development right now for Firefox 34 including translation in the browser, making voice or video calls (a feature called “Hello” or “Loop”), debugging information for web developers in the Dev Tools Inspector, and recent improvements to HTML5 gaming.

I’ve written step by step instructions on these
ways to test Firefox 34. If you would like to see what it’s like to improve a popular open source project, trying out these tasks is a good introduction.

Aurora

First, Install the Aurora version of Firefox. It is best to set it up to use multiple profiles. That ensures you don’t use your everyday version of Firefox for testing, so you won’t risk losing your usual profile information. It also makes it easy to restart Firefox with a new, clean profile with all the default settings, very useful for testing. Sometimes I realize I’m running 5 different versions of Firefox at once!

To test “Hello”, try making some voice or video calls from Firefox Aurora. You will need a friend to test with. Or, use two computers that you control. This is a good task to try while joining our chat channels, #qa or #testday on irc.mozilla.org; ask if anyone there wants to test Hello with you. The goal here is mostly to find and report new bugs.

If you test the translation infobar in Aurora you may find some new bugs. This is a fun feature to test. I like trying it on Wikipedia in many different languages, and also looking at newspapers!

If you’re a web developer, you may use Developer Tools in Firefox. I’m asking Aurora users to go through some unconfirmed bug reports, to help improve the Developer Tools Inspector.

If you like games you can test HTML5 web-based games in Firefox Aurora. This helps us improve Firefox and also helps the independent game developers. We have a list of demo games so you can play them, report glitches, and feel like a virtuous open source citizen all at once. Along the way you have opportunities to learn some interesting stuff about how graphics on the web can work (or not work).

Monster madness

These testing tasks are all set up in One and Done, Mozilla QA’s site to start people along the path to joining our open source community. This site was developed with a lot of community contribution including the design and concept by long-time community member Parul and a lot of code by two interns this summer, Pankaj and Maja.

Testing gives a great view into the development process for people who may not (yet) be programmers. I especially love how transparent Mozilla’s process can be. Anyone can report a bug, visible to the entire world in bugzilla.mozilla.org. There are many people watching that incoming stream of bug reports, confirming them and routing them to developer teams, sometimes tagging them as good first bugs for new contributors. Developers who may or may not be Mozilla employees show up in the bugs, like magic . . . if you think of bugmail notifications as magic . . .

It is amazing to see this very public and somewhat anarchic collaboration process at work. Of course, it can also be extremely satisfying to see a bug you discovered and reported, your pet bug, finally get fixed.

Mozilla's bug reporting, QA, and release processes

June 27, 2014 Liz

AdaCamp Portland was an amazing conference for feminist women in open source tech and culture. Not all, but many of the conference attendees are developers, system administrators, or do other technical work in open source software. I gave an informal talk meant to be an overview of some things I currently do at Mozilla. Lots of people came to the session! We all introduced ourselves going around the room.

To start off with, I showed a sample bug to talk about the process of reporting a bug, using Bugzilla, and practicing the skill of reading and understanding a bug report.

We looked first at Bug 926292.

Bugzilla couple

Let’s look at the life of this bug!

This bug was reported in October 2013 for Firefox 24 by someone new to bugzilla.mozilla.org. New users have basic permissions to file and comment on bugs. For around their first 25 bugs filed or commented on, they are marked “New to Bugzilla” to anyone with more permissions on the system. This helps more experienced users to know when they’re in conversation with people who are relatively new to the system. And, bugs reported by new users are automatically entered into Bugzilla with a status of “UNCONFIRMED”.

Our bug reporter was answered the same day by a community bug triager who used the “needinfo” checkbox to ask the bug reporter more questions. A bit later, in Comment 2, I was able to confirm the bug; I marked it NEW. Community members often jump in to do this from Bug Triage bug days, from our One and Done community taskboard, or because they watch the “Firefox::Untriaged” component. (Yes . . . you too can sign up to get email from Bugzilla every time a new bug is filed!)

Francesca Ciceri is currently working on bug triage and verification with our team as part of the GNOME-OPW internship program, doing similiar work to Tiziana Selitto who was an OPW intern last year! Both their blogs have good insights into what it’s like to approach QA in a huge and somewhat chaotic system like Mozilla’s.

In our example bug, I took a guess as to which product and component to add to the bug. This is like putting the bug into the right place where developers who work in a particular area will be likely to see it, and pay attention to it. I moved it from “Firefox” to “Core” and thought it may be something to do with the CSS Object Model. Picking the right product and component is tricky. Sometimes I look for similar bugs, to see what component they’re in. Sometimes I use Bugzilla’s Browse pages to skim or search through the descriptions of components for Firefox, Core, and Toolkit. Even after doing this for a year and a half, I get it wrong. Here, a developer moved the bug to what he thought was a better component for it, Core::Layout. (Developers also sometimes guess wrong, and keep passing a bug around to each others’ components like a hot potato.)

At this point a few developers explored the bug, and went back and forth with each other and the bug reporter about whether it had been fixed or not, exactly what the bug was, whether it is a Mac issue or a Firefox issue, and how to fix it. It was resolved as a duplicate of another bug in October, but the bug reporter came back to reopen it in February 2014. The bug reporter was polite but persistent in explaining their view, giving more details of the browser behavior, trying to find the bug in the very latest developer build (Nightly), giving a test case and comparing the behavior in different browsers. A developer submitted a patch, asked for code review. Related bugs were mentioned and linked. At least two new bugs were filed.

One important thing to note is that people working on QA and development tend to move very fluidly between using various Firefox versions. One of the best things you can do to get involved with helping out is to set up all four “channels” of Firefox with the capability to run them all at once with different profiles, and to start with new, clean profiles. In fact, we need better and more up to date documentation of how to do that on different operating systems, with screenshots! Here are some links that may help you set that up:
* http://www.callum-macdonald.com/about/faq/multiple-firefox-instances/
* https://developer.mozilla.org/en-US/docs/Mozilla/Multiple_Firefox_Profiles
* https://support.mozilla.org/en-US/kb/profile-manager-create-and-remove-firefox-profiles

OK, back to bug 926292!

Since I had worked on the bug and added myself to the cc field, I got bugmail about all these changes, and more or less followed a long. I often think that the collaboration that happens in bug fixing is very beautiful, and even fairly efficient!

In comment 29 you can see that code got committed to a mercurial repository, to “inbound”. From there, it goes through automated tests and is merged by one of the “sheriffs” into another hg repository, mozilla-central, where it will go into the next build of Nightly, which at that point in April, was Firefox 31.

Comment 30 suggests uplifting the patch to versions that will soon be released, to Aurora and Beta. Release managers started to get involved, commenting and asking the developers to formally nominate the bug for uplift.

At this point in my talk I explained a little bit about the “trains”.

Trains

The versions of Firefox under development advance on a 6 week cycle, from Nightly to Aurora to Beta to the main release of Firefox. In this rapid release schedule, Firefox 31 was Nightly, so Aurora was 30, 29 was Beta, and the release version most folks use was 28. The uplift request was refused so the patch “rode the train”. That means, if you were using Firefox 31 any time after the patch was merged into mozilla-central, you will see its effect. (It would also be fixed for Firefox 32 and 33 which are currently in use as Aurora and Nightly, since 31 is currently Beta.)

Our bug was marked “FIXED” when the patch was merged into mozilla-central. You can see near the end of its comments that I tagged the bug “verifyme” to put it into the queue of bugs that need verifying for Firefox 31. Many people see that list and work on verifying bugs including community members in our Bug Verification test days. I hope the story of this particular bug is over. I don’t have the number immediately to hand but I believe that over 1000 bugs are fixed for each version of Firefox over its release cycle. We can’t verify them all, but it’s amazing what we do get done as a team!

Other tools we looked at in my talk and the ensuing discussion:

Datazilla, which tests and measures Firefox performace: https://datazilla.mozilla.org/

Mozmill, a UI automation framework for Mozilla apps including Firefox and Thunderbird: https://github.com/mozilla/mozmill

Socorro, or crash-stats, where QA and other teams keep track of crashes in Firefox and other Mozilla products: https://crash-stats.mozilla.com

The ftp directories where Firefox builds and build candidates are stored: ftp://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/

The mercurial repositories or “the tree”: http://hg.mozilla.org/

DXR, a nifty tool to search Mozilla’s code: http://dxr.mozilla.org/mozilla-central/source/

TBPL which shows the test results for every commit that’s merged into different branches https://tbpl.mozilla.org/

And a quick view into Mozilla’s Jenkins continuous integration dashboard which you can only see from our VPN, just to give an idea of the work we do when Firefox is in Beta. As a particular version of Firefox advances through rapid release, QA pays more attention to particular areas and uses different tools. We have to know a little bit about everything, be able to reproduce a user’s bug on many different possible platforms, figure out which developers may be able to fix a bug (or whose commit may have caused a regression or crash).

It was a lot to cover in an hour long talk! I wanted to pilot this informally as a test for doing a more formal talk with slides.

It represented fairly well that QA covers quite a lot of territory; it’s complicated and interesting work.

Mozilla Summit 2013: bugs, crafts, and fun

October 10, 2013 Liz

The Mozilla Summit in Toronto was a lot of fun. I met so many intense, idealistic, motivated open source contributors, it made me feel renewed energy for my own contribution.

On the first day I heard some of the keynotes but missed Mitchell Baker’s keynote on the Nature of Mozilla, which I’m going to watch today. People were talking about it through the rest of the conference, so I’m curious.

The talk I did was called Awesome Bugzilla Tricks and was a fairly short presentation of some tips for using Bugzilla.

Awesome bugzilla-tricks-slides2 from Liz Henry

As we went through these tips, the talk turned often into a good discussion of how people use these features of Bugzilla. We thought of a new way to implement Bugzilla tags to look more like browser bookmarks.

People talked about their own workflow as we went around the room describing what we do with bugzilla.mozilla.org! That is usually my favorite part of a workshop level technical discussion. It is like a series of sharing mini-demonstrations about actual working process, what we use and what we do and why. That is very illuminating. I realized in our discussion that I was talking with Kohei who made bzdeck, a tool that makes reading Bugzilla.mozilla.org bug reports more like an email inbox. I had particularly wanted to meet him so was very happy he came to the talk and the discussion! 😀

Durng the talk I said something a little weird and abstract that was not in my slides, inspired by the discussion. Here it is…

As a literary critic I find it fascinating that it is a huge collection of textual information which we engage with as authors and readers. It is like the ultimate “difficult work”, like reading James Joyce except 900,000 times better and with a more interesting result. There is no way to make it easy to understand, even if you can make little pieces of it easier to use and learn, the underlying information and tasks are beautifully complex and demanding for anyone who engages with it, which means you can’t fail to learn something if you try.

Then, luckily, we went back to talking about practical things, features, dashboards, and workflow.

The other main activity I did at the Summit was to set up a big table with craft supplies. Based on an idea of Lukas Blakk’s, I set up beads, string, and charms shaped like ladybugs, dragonflies, bees, and beetles so that people could make necklaces, bracelets, and other wearable souvenirs. The beads had numbers and letters so that you could spell out the number of your favorite bug.

People did just that!

I loved seeing people think about what bug was their favorite or most important to them personally. Sometimes a first bug, or one that people were proud of fixing, or one with an important, complicated discussion. Several people told me that during the Summit, they looked up other people’s favorite bugs from the bug bracelets, and learned something interesting.

Bug 356038 was represented by number and by its Bugzilla alias, BCP47:

Rust and Github bug #5677, ‘Rustpkg “ready for use” metabug’ got some love:

Vu gave a shout out to bug 780076,

And here is another bracelet featuring bug 808964,

For my own favorites I made three objects, one a wire necklace for bug 923590, “Pledge never to implement HTML5 DRM”, and a bracelet and a barette with my first patch because I was proud of submitting a patch. The EME or DRM issue was discussed very intensely on Sunday by many engineers. Feelings definitely ran high and people were determined to continue discussion. I was glad it got an official slot in the schedule for discussion and that there was widespread interest.

Bug 298619 was put onto a cell phone charm along with a blue and orange glass bead shaped like a beetle!

Many other people made Mozilla or MozReps necklaces, spelled out their names or their loved one’s names, with bug charms, or with wire, like this beautiful and creative copper wire creation:

And this Maker Party necklace!

This resulted in “conference swag” was personal and made on the spot, worn and also given as presents! Many people commented that they felt soothed and comforted by hands-on activity in the middle of intense social interaction. I observed that people discussed what bugs or words to spell, how to design their objects, and what techniques to use, very collaboratively, so it was a nice physical representation of work and process.

After the first day, I left the craft station open 24 hours in the lounge, replenishing it with beads from West Queen Street on day 2 becasue we ran out of the letters Z and A, popular in spelling out “Bugzilla” and “Mozilla”. Of course this was because we were in Canada, where they use a lot of “eh”!

I would do this again at a conference, and now I know from experience what supplies are needed and which things are likely to be popular.

The other silly and fun part of the conference for me was wearing (and lending to people) the brainwave controlled robot fox ears! The looks on people’s faces, so priceless, as they realized the ears were really moving. Everyone who tried it laughed very hard!

Thanks for a great and inspiring Summit, everybody!

File a bug: the missing manual, now with unicorns

April 9, 2013 Liz

At countless conference talks I have heard standard advice on “how to get involved in an open source project”. It goes something like this:

Step 1. File a bug!
Step 2. Submit a patch! (repeat steps 1 and 2 for a while)
Step 3. Now you are ready to write some new features and stuff! Fly and be free!

I always thought that was interesting because it is an attempt to reassure people that you don’t have to leap into immediate coding. Just file a bug, that is the first step. This results in people coming into projects and wondering vaguely how to find bugs.

Part of what I want to do as bugmaster for Mozilla is to put in another step — look at the bugs already reported, since there are a squizillion of them, and see what you can do to improve the meta information of those bugs.

Today on Planet Mozilla I noticed some really good advice from Jason Smith on how to find bugs: WebRTC testing: Try out conversat.io and file bugs. It is really sensible and practical. Jason’s blog has a few posts like this that advise focus in a particular area, like WebRTC or Desktop web apps, by incorporating use of those tools in your daily life. Our “get involved in open source” sequence now looks more like this,

Step 1. Find some feature that could use testing.
Step 2. Figure out how to use it regularly.
Step 3. Use it.
Step 4. Encounter behavior you think is a bug.
Step 5. FILE A BUG (BUT HOW?)

There is a lot of background knowledge necessary to actually file a bug in the complicated system that is bugzilla.mozilla.org (or BMO).

So let’s take the WebRTC example. Say you’ve followed Jason’s advice, used conversat.io for a while, and found A BUG. Jason helpfully provides a link directly into Bugzilla to the enter_bug form, with the Product and Component pre-filled out to be for a bug with WebRTC (the component) and Core (the product.) Bugs in BMO are categorized according to Product, like Firefox Desktop, Firefox for Android, FirefoxOS, Thunderbird, etc. “Core” is the product for the code that is shared between many other products. If you were looking to file a bug with WebRTC from scratch you would probably not know which product to file it under, though you have to choose one. So it’s great that Jason gives a link to the right product and component!

But wait. There is so much more background, or context, to understand. You don’t have to, but it is very good to understand it!

First of all you have to have a bugzilla account for the link to work properly. If you do that, you will be a new bugzilla user, and some of the bug entry forms will look different to you — you’ll be automatically directed to a “guided bug entry form” which is broken into several steps, rather than the form that shows you the whole “advanced view” with several dozen fields and dropdown menus.

Second, how about looking at the list of all the components in the Core product. This is a good part of the background knowledge – a little piece of the map or geography of Bugzilla. As you can see, there are a lot of components that are part of Core. Scrolling down to the WebRTC bit, you can see that there are several sub categories: WebRTC, WebRTC:Audio/Video, WebRTC:Networking, and WebRTC:Signaling. Click on the general WebRTC component to see a list of open WebRTC bugs. This is where your geography lesson gets useful.

Right now there are 113 open bugs for WebRTC. You might look over them simply to get an idea of what kinds of bugs others have found. More about this later!

The important thing at this moment is: Is your bug already reported? Depending on how many bugs there are in this list, and your levels of interest and patience, you might want to either quickly read through the summaries (the title of the bug) or do a search down the page for words that might be in the bug you’re about to file.

If you find something in that list that you think is your bug, take a closer look at it. Read it and the comments and try to understand what they’re talking about. If it is the same as your bug, you may want to leave a comment describing what you saw happen.

But let’s say you don’t find your bug on that list. Aha! Here we use the File a bug link from the original blog post, link to file a bug with WebRTC. If you are me, or a user of Bugzilla who has made more than 25 edits or comments to bugs, you will see the advanced bug entry form, which is huge (you can see it from space) and looks like this:

Enter bug webrtc advanced

If you are a relatively new user of Bugzilla, you’ll come first to a guided entry form, broken into several screens. (At any point, you can switch to the advanced entry form with a link at the bottom of the page, if clicking through multiple screens annoys you.) The first screen for guided bug entry would normally be for selecting the product and component. Since you have these already chosen in Jason’s convenient link, you start on Screen 2, where you can enter the summary for your bug. In this case I am reporting a sort of unicorn invasion:

Enterbug webrtc screen2

After you enter a summary you will see a list pop up underneath the summary field, of bugs that may be similar. It is worth reading through those to see if anyone else has reported unicorns invading their conversat.io screens in Firefox. In this case, definitely not.

Enterbug webrtc screen2 list

Since no one else has reported this bug, I click the “My issue is not listed” button, and advance to screen 3, which suggests how I can describe my actions or steps to reproduce the issue, exactly what happened that I think is a bug, and what I think should have happened instead.

Enter bug screen3 unicorn jpg

Great, we have filed a bug! Back to that list of “how to contribute to an open source project”:

Step 1. Find some feature that could use testing.
Step 2. Figure out how to use it regularly.
Step 3. Use it.
Step 4. Encounter behavior you think is a bug.
Step 4.5 Make a bugzilla.mozilla.org account.
Step 4.6 Confirm it with the email confirmation.
Step 4.7 Log in to bugzilla.mozilla.org.
Step 5. FILE A BUG (which we now know how to do!!!)

But wait, there’s more — or there can be if you want to get your bearings on that map of Bugzilla. Take a look back at the list of all the general open WebRTC bugs. What can we understand from this list?

As I look over the current list it is pretty mysterious. From the language in the summaries, I would guess some of the bugs are notes by the development team for their own to-do list, and some of them look like bugs discovered by general users of the software. My first impulse is to sort the page a few different ways to see what that reveals. Sorting by Status shows the UNCONFIRMED bugs at the top and the NEW bugs listed just underneath. There is one titled getUserMedia freeze all system that isn’t confirmed yet and may be a good example.

Here is an interesting one, No event when remote peer disappeared. My view of this bug is going to be different from yours, if you are new to Bugzilla, because I have more magic powers, ie, canconfirm and editbugs permissions, as well as some extra admin stuff. There is a lot of stuff on the page. It’s extremely “busy” with text! You have to learn to skim it and parse it mentally so that you can see and notice the stuff that is important to you at the moment. Here is what this bug looks like for a new Bugzilla user.

Example webrtc bug2

What I can see from reading this bug is that there is at least one person actively looking at newly filed bugs, triaging them, and working on the project. And in fact as I click around and read more of the bugs for WebRTC I can see Jason is actively engaged with most of them, which is not a big surprise since he is blogging about the subject.

Jason’s blog looks like a quite useful place to find out areas that welcome testers and bug-finders. You can also look at the QMO quality assurance and testing pages which explain how to run nightly builds and participate in QA’s bug test and triage days.

My bigger point here, though, is that to start contributing to an open source project, aside from reporting one-off bugs you accidentally discover, it is super helpful to learn the landscape of the project. Adopt the project and poke around to learn about it. If you are reporting a bug, look at the other bugs. Look at who is commenting and working on those bugs. Join their IRC channel and read their wiki pages and (usually more formal than the wiki pages…) developer documentation. Or simply google the project to learn more about what’s going on with it. In this case I found that conversat.io is quite new and was developed partly to show off what WebRTC is and what it can do.

It was really apparent, from my morning of poking around, how much transparency there is at Mozilla, and the amazing technical depths you can get to from half a day’s reading and bug surfing. As a society we really have yet to realize the implications for education and educational institutions. It is a major cultural shift I am happy to be part of.