Festival of the battling bugs

My dad has been uploading photos from his mother’s albums and there is an interesting page of a religious festival in San Francisco de Yare in Venezuela. I vaguely remember him telling me stories about that or something similar and we made terrifying paper maché masks for some occasion (Maybe just for something fun to do).

A photo from my dad’s slides that I digitized some years ago:
devil dancer

And here are some of the pics with captions from my grandma’s album.
dancing devils festival

I believe we should have not only fabulous monuments to the Internet and other technical achievements but we should have amazing festivals. As I read out the description of the devils approaching the church, singing décimas and then kneeling in submission, Danny suggested a ceremonial Battle of the Bugs. Noisebridge could host a giant parade where we enact open source bugs (the demons) and the developers defeating them. I can picture different contingents acting out their particular dramas. I bet it would be easy to get companies as well as open source projects to participate.

I just love secular festivals and while I would not advocate stealing anything specific from this Venezuelan folkloric tradition it would be very cool to create some new festivals that are more like a giant participatory play, with dramas enacted, than a parade where we just walk around.

Suddenly imagining the Internet Drama play of the Content Moderators. Wow! It would be amazing!!!!

Jouissance and a sense of agency

Morning reading: Introduction to Hacking Diversity: The Politics of Inclusion in Open Technology Cultures by Christina Dunbar-Hester. This is going to be fun since everyone I know is quoted in it (often pseudonymously) But no quotes from me (I think) as during the interview phase I was having some sort of major health flare-up. And if there’s ever a book where I should be obscurely in the footnotes somewhere it’s this one!

Though “diversity in tech” discourse is emanating from many quarters in our current historical moment, it is important that the mandate of open-technology cultures is not identical to that of industry and higher education. Here, the reasons for engagement with technology nominally include experiencing jouissance and a sense of agency. This is experienced through, yet not reducible to, community members’ engagement with technology. If we tease apart the emancipatory politics from the technical engagement, we find that the calls for inclusion and for reframing power relations are not only about technical domains; rather, they are about agency, equity, and self-determination at individual and collective levels.

At that “jouissance” sentence I felt my heart sing and I felt so seen. Yes! This bodes well for the entire book’s understanding of our feelings and our context. So many histories leave out crucial things like love and fun and joy. Why have I fucked around with computers my whole life? Because love and happiness is why. They’re exciting, the Internet is still like a dream to me, the access to information and the possibilities of unfiltered/unmediated publishing or production, and consumption, still holds so much hope. Because I (we) like it that’s why. Like Mole seeing the Water Rat’s boat for the first time,

The Rat said nothing, but stooped and unfastened a rope and hauled on it; then lightly stepped into a little boat which the Mole had not observed. It was painted blue outside and white within, and was just the size for two animals; and the Mole’s whole heart went out to it at once, even though he did not yet fully understand its uses.

We still don’t, of course.

Also good, everything in this chapter about collectivity. *heart eyes emoji*

Silly songs

Day 3 of DWeb Camp. Judy and I sang our translation of the Free Software Song into Toki Pona (only got through the first verse though…)

o kama poka mi mute o pana
e sitelen pali pi ilo sona
sina mute li ken pali e ale,
jan pi pakala pona o

Then I showed off me and Danny’s Hackernationale and a bunch of us sang it in the geodesic dome! (I need to re-do the syllables which aren’t always correctly under their notes, there)

Then, somehow, we ended up on the floor of the dome collectively writing a filk version of Sweet Caroline, which we then performed in the EFF Lounge at midnight! Here are the lyrics!

Sweet Copyleft
(lyrics by Liz, Leez, and Judy)

Where it began, I can’t begin to knowing
But then I know it’s growing strong
First GPL,
Then came Creative Commons,
Who’d have believed you’d come along
Files, touching files
Reusing them, remixing me, remixing you

Sweet copyleft,
License never felt so good
I’d be inclined
To share everything I could

but now I
look at the net and it don’t seem so lonely
we filled it up with all of us
when i create,
the code flows out my fingers
We make so much when I remix you
nodes touching nodes….
touching nodes, touching nodes, touching nodes

Sweet Copyleft,
license never felt so good!

Earlier in the day I got to play with a Waffletone, which is kinda like a Monome but with mechanical keyboard keys! The github repo is a beautiful and elegant marvel of great documentation!

Actual work work

I should mention my actual work while I’m blogging about stuff. “My” Firefox release is now in beta, Firefox 66. We have a team of release managers so can rotate ownership of upcoming releases. Usually this means you’re in the hot seat every 3 releases, about every 4 months. We have a bigger team now so there is more breathing room and we can support each other more; basically that means we’re less burned out. Yay! (A couple of years ago there were like, 3 of us and one was the team manager.)

The feeling of ownership is somewhat silly since it is an enormous collaborative effort, but it helps in the job to follow one version closely, you care about it and are able to filter out & not pay so much attention to the other versions in development. (In practice, that is a lot of email I can skim and archive when it isn’t for “my” release.) The release engineers also rotate through version ownership. It’s a good system.

If you’d like to see all the bugs fixed for Firefox 66 so far, between its birth as Firefox Nightly last December and now, here’s a Bugzilla query which you shouldn’t click if you’re on your phone or a slow connection because it’s going to take a while to load – it will show a list of the 3000+ bugs already fixed. Probably don’t click that unless you really want to wait a minute and a half for the page to load.

While Firefox 66 is in beta, I’m watching for fixes in the version right behind it, Firefox 67, that we might be able to bring to users more quickly. We backport, or uplift, between 200 and 400 issues while a version is in beta. Here’s what I’ve approved to uplift so far, patches from about 40 bugs. It’s helpful when looking at these lists to sort the information different ways (by clicking on the column headers.) These next few weeks till March 19 (the release date) will be my crunch time.

Today I did about one million different things. Hilariously, as so much of our work is right out in the open, you can see some of those one million things (in 68 bugs), at least the bits of it that were me reading a lot of bug reports and figuring out what to do with them (poke someone from another team, engineers, QA, etc, add relevant tags or other info, and while I’m looking, decide how important it is for it to be in Firefox 66.)

Nifty addition to treeherder

At work I just found out we have a nice change in Treeherder, the tool that monitors current Firefox (and other) builds. Without writing my usual 10 pages of explanation and backstory, I’ll just say that Treeherder shows whether the release engineering infrastructure has started the release promotion process to take the builds from the changeset specified and do some super mystery magic on them to make them suitable for actual release to Firefox users. We get an email from the notification system that this happened, but now we can follow along in the same tool we use to watch tests pass or fail.

gecko decision task for firefox and fennec

In this case, I used a different tool (Ship-It) to set up the parameters to take the test builds for the 1a24837f9ed232d8d2dc4535d11ee53c9847b109 changeset (say it 6 times, fast) for Firefox 59 beta 7 (plus the android and developer edition versions) and declare us ready to kick off this process. Instead of checking email notifications that only go to a particular mailing list, or diving straight into TaskCluster, I can see that near the end of the enormous list of automated tests running on various platforms, “Gecko Decision Task”. This tells us that the builds are now being “promoted”. I haven’t tried drilling deeper yet, but if you click on any of the links there (such as “promote_firefox”) then you get even more detail, including links to TaskCluster and its myriad, confusing joys.

As usual, I think it’s damn cool that all this happens “in the open” at Mozilla, so you don’t even have to be logged into anything to see quite a lot of detail. Anyone with an interest can learn a lot here, or even get involved and contribute, because of this level of commitment to transparency. It’s a good way to find out if you like the work, or to get work experience that you can easily show off in the future.

That Zarro Boogs feeling

This is my third Firefox release as release manager, and the fifth that I’ve followed closely from the beginning to the end of the release cycle. (31 and 36 as QA lead; 39, 43, and 46 as release manager.) This time I felt more than usually okay with things, even while there was a lot of change in our infrastructure and while we started triaging and following even more bugs than usual. No matter how on top of things I get, there is still chaos and things still come up at the last minute. Stuff breaks, and we never stop finding new issues!

I’m not going into all the details because that would take forever and would mostly be me complaining or blaming myself for things. Save it for the post-mortem meeting. This post is to record my feeling of accomplishment from today.

During the approximately 6 week beta cycle of Firefox development we release around 2 beta versions per week. I read through many bugs nominated as possibly important regressions, and many that need review and assessment to decide if the benefit of backporting warrants the risk of breaking something else.

During this 7 week beta cycle I have made some sort of decision about at least 480 bugs. That usually means that I’ve read many more bugs, since figuring out what’s going on in one may mean reading through its dependencies, duplicates, and see-alsos, or whatever someone randomly mentions in comment 45 of 96.

And today I got to a point I’ve never been at near the end of a beta cycle: Zarro Boogs found!

list of zero bugs

This is what Bugzilla says when you do a query and it returns 0. I think everyone likes saying (and seeing) “Zarro Boogs”. Its silliness expresses the happy feeling you get when you have burned down a giant list of bugs.

This particular query is for bugs that anyone at all has nominated for the release management team to pay attention to.

Here is the list of requests for uplift (or backporting, same thing) to the mozilla-beta repo:

more zero pending requests

Yes!! Also zarro boogs.

Since we build our release candidate a week (or a few days) from the mozilla-release repo, I check up on requests to uplift there too:

list of zero pending requests


For the bugs that are unresolved and that I’m still tracking into the 46 release next week, it’s down to 4: Two fairly high volume crashes that may not be actionable yet, one minor issue in a system addon that will be resolved in a planned out-of-band upgrade, and one web compatibility issue that should be resolved soon by an external site. Really not bad!

Our overall regression tracking has a release health dashboard on displays in many Mozilla offices. Blockers, 0. Known new regressions that we are still working on and haven’t explicitly decided to wontfix: 1. (But this will be fixed by the system addon update once 46 ships.) Carryover regressions: 41; about 15 of them are actually fixed but not marked up correctly yet. The rest are known regressions we shipped with already that still aren’t fixed. Some of those are missed uplift opportunities. We will do better in the next release!

In context, I approved 196 bugs for uplift during beta, and 329 bugs for aurora. And, we fix several thousands of issues in every release during the approx. 12 week development cycle. Which ones of those should we pay the most attention to, and which of those can be backported? Release managers act as a sort of Maxwell’s Demon to let in only particular patches …

Will this grim activity level for the past 7 weeks and my current smug feeling of being on top of regression burndown translate to noticeably better “quality”… for Firefox users? That is hard to tell, but I feel hopeful that it will over time. I like the feeling of being caught up, even temporarily.

liz in sunglasses with a drink in hand

Here I am with drink in hand on a sunny afternoon, toasting all the hard working developers, QA testers, beta users, release engineers, PMs, managers and product folks who did most of the actual work to fix this stuff and get it firmly into place in this excellent, free, open source browser. Cheers!

Automated test harnesses for Firefox, zooming out a bit

In December I was going through some of the steps to be able to change, fix, and interpret the results of a small subset of the gazillion automated tests that Mozilla runs on Firefox builds. My last post arrived at the point of being able to run mochitests locally on my laptop on the latest Firefox code. Over the holidays I dug further into the underlying situation and broaded my perspective. I wanted to make sure that I didn’t end up grubbing away at something that no one cared about or wasn’t missing the big picture.

My question was, how can I, or QA in general, understand where Firefox is with e10s (Electrolysis, the code name for the multiprocess Firefox project). Can I answer the question, are we ready to have e10s turned on by default in Firefox — for the Developer Edition (formerly known as Aurora), for Beta, and for a new release? What criteria are we judging by? Complicated. And given those things, how can we help move the project along and ensure good quality; Firefox that works as well as or, we hope, better than, Firefox with e10s not enabled? My coworker Juan and I boiled it down to basically, stability (lowering the crash rate) and automated test coverage. To answer my bigger questions about how to improve automated test coverage I had to kind of zoom out, and look from another angle. For Firefox developers a lot of what I am about to describe is basic knowledge. It seems worth explaining, since it took me significant time to figure out.

First of all let’s look at treeherder. Treeherder is kind of the new TBPL, which is the old new Tinderbox. It lets us monitor the current state of the code repositories and the tests that run against them. It is a window into Mozilla’s continuous integration setup. A battery of tests are poised to run against Firefox builds on many different platforms. Have a look!

Current view of mozilla-central on treeherder


Edward Tufte would have a cow. Luckily, this is not for Tufte to enjoy. And I love it. You can just keep digging around in there, and it will keep telling you things. What a weird, complicated gold mine.

Digression! When I first started working for Mozilla I went to the Automation and Tools team work week where they all came up with Treeherder. We wanted to name it something about Ents, because it is about the Tree(s) of the code repos. I explained the whole thing to my son, I think from the work week, which awesomely was in London. He was 11 or 12 at the time and he suggested the name “Yggdrazilla” keeping the -zilla theme and in reference to Yggdrasil, the World-Tree from Norse mythology. My son is pretty awesome. We had to reject that name because we can’t have more -zilla names and also no one would be able to spell Yggdrasil. Alas! So, anyway, treeherder.

The left side of the screen describes the latest batch of commits that were merged into mozilla-central. (The “tree” of code that is used to build Nightly.) On the right, there are a lot of operating systems/platforms listed. Linux opt (optimized version of Firefox for release) is at the top, along with a string of letters and numbers which we hope are green. Those letter and numbers represent batches of tests. You can hover over them to see a description. The tests marked M (1 2 3 etc) are mochitests, bc1 is mochitest-browser-chrome, dt are developer tools tests, and so on. For linux-opt you can see that there are some batches of tests with e10s in the name. We need the mochitest-plain tests to run on Firefox if it has e10s not enabled, or enabled. So the tests are duplicated, possibly changed to work under e10s, and renamed. We have M(1 2 3 . . .) tests, and also M-e10s(1 2 3 . . . ). Tests that are green are all passing. Orange means they aren’t passing. I am not quite sure what red (busted) means (bustage in the tree! red alert!) but let’s just worry about orange. (If you want to read more about the war on orange and what all this means, read Let’s have more green trees from Vaibhav’s blog. )

I kept asking, in order to figure out what needed doing that I could usefully do within the scope of As Soon As Possible, “So, what controls what tests are in which buckets? How do I know how many there are and what they are? Where are they in the codebase? How can I turn them on and off in a way that doesn’t break everything, or breaks it productively?” Good questions. Therefore the answers are long.

There are many other branches of the code other than mozilla-central. Holly is a branch where the builds for all the platforms have e10s enabled. (Many of these repos or branches or twigs or whatever, are named after different kinds of tree.) The tests are the standard set of tests, not particularly tweaked to allow for e10s. We can see what is succeeded and failing on treeherder’s view of holly. A lot of tests are orange on holly! Have a look at holly by clicking through on the link above. Here is a picture of the current state of holly.

Treeherder holly1

If you click a batch of tests where there are some failures — an orange one — then a new panel will open up in treeherder! I will pick a juicy looking one. Right now, for MacOS 10.6 opt, M(2) is orange. Clicking it gives me a ton of info. Scrolling down a bit in the bottom left panel tells me this:

mochitest-plain-chunked 164546/12/14136

The first number is how many tests ran. Scary. Really? 164546 tests ran? Kind of. This is counting assertions, in other words, “is” statements from SimpleTest. The first number lists how many assertions passed. The second number is for assertion failures and the third is for “todo” statements.

The batches of tests running on holly are all running against an e10s build of Firefox. Anything that’s consistently green on holly, we can move over to mozilla-central by making some changes in mozharness. I asked a few people how to do this, and Jim Matthies helpfully pointed me at a past example in Bug 1061014. I figured I could make a stab at adding some of the newly passing tests in Bug 1122901.

As I looked at how to do this, I realized I needed commit access level 2 so I filed a bug to ask for that. And, I also tried merging mozilla-central to holly. That was ridiculously exciting though I hadn’t fixed anything yet. I was just bringing the branch up to date. On my first try doing this, I immediately got a ping on IRC from one of the sheriffs (who do merges and watch the state of the “tree” or code repository) asking me why I had done something irritating and wrong. It took us a bit to figure out what had happened. When I set up mercurial, the setup process and docs told me to install a bunch of Mozilla specific hg extensions. So, I had an extension set up to post to bugzilla every time I updated something. Since I was merging several weeks of one branch to another this touched hundreds of bugs, sending bugmail to untold numbers of people. Mercifully, Bugzilla cut this off after some limit was reached. It was so embarrassing I could feel myself turning beet red as I thought of how many people just saw my mistake and wondered what the heck I was doing. And yet just had to forge onwards, fix my config file, and try it again. Super nicely, Clint Talbert told me that the first time he tried pushing some change he broke all branches of every product and had no idea what had happened. Little did he know I would blog about his sad story to make myself feel less silly….. That was years ago and I think Mozilla was still using cvs at that point! I merged mozilla-central to holly again, did not break anything this time, and watched the tests run and gradually appear across the screen. Very cool.

I also ended up realizing that the changes to mozharness to turn these batches of tests on again were not super obvious and to figure it out I needed to read a 2000-line configuration file which has somewhat byzantine logic. I’m not judging it, it is clearly something that has grown organically over time and someone else probably in release engineering is an expert on it and can tweak it casually to do whatever is needed.

Back to our story. For a batch of tests on holly that have failures and thus are showing up on treeherder as orange, it should be possible to go through the logs for the failing tests, figure out how to turn them off with some skip-if statements, filing bugs for each skipped failing test. Then, keep doing that till a batch of tests is green and it is ready to be moved over.

That seems like a reasonable plan for improving the automated test landscape, which should help developers to know that their code works in Firefox whether e10s is enabled or not. In effect, having the tests should mean that many problems are prevented from ever becoming bugs. The effect of this test coverage is hard to measure. How do you prove something didn’t happen? Perhaps by looking at which e10s tests fail on pushes to the try server. Another issue here is that there are quite a lot of tests that have been around for many years. It is hard too know how many of them are useful, whether there are a lot of redundant tests, in short whether there is a lot of cruft and there probably is. With a bit more experience in the code and fixing and writing tests it would be easier to judge the usefulness of these tests.

I can also see that, despite this taking a while to figure out (and to even begin to explain) it is a good entry point to contributing to Firefox. It has more or less finite boundaries. If you can follow what I just described in this post and my last post, and you can read and follow a little python and javascript, then you can do this. And, if you were to go through many of the tests, over time you would end up understanding more about how the codebase is structured.

As usual when I dive into anything technical at Mozilla, I think it’s pretty cool that most of this work happens in the open. It is a great body of data for academics to study, it’s an example of how this work actually happens for anyone interested in the field, and it’s something that anyone can contribute to if they have the time and interest to put in some effort.

This post seems very plain with only some screenshots of Treeherder for illustration. Here, have a photo of me making friends with a chicken.

Liz with chicken

Make hackerspaces better – support Ada initiative

Hello! I love hackerspaces! And I’m asking hackerspaces around the world to donate to the Ada Initiative in support of making hackerspaces welcoming and safe places for women. My goal is to raise $4096 and if we do, I’ll match the first $1028 donated. **UPDATE** extending the deadline to next Friday, Oct. 3.

Adacamp liz and heidi in tiara

My home base is Double Union, a feminist hackerspace in San Francisco and it’s going strong. It has lively events every week and over 150 members.

A group of us at AdaCamp SF last year decided we could start a maker and hacker space for women. AdaCamp SF is an unconference run by the Ada Initiative for feminist women in open tech and culture. There were so many of us all together at once. So powerful feeling! With months of hard work, it happened — we opened Double Union.

We get to hack there without sexist bullshit or constantly fending off creepy dudes!

The thing is, I believe that any hackerspace can potentially be that way. You, too, could have a hackerspace where many women feel comfortable, welcome, valued, in their creative, coding, and entrepreneurial and activist endeavors, in a space full of allies and comrades of all genders. This can’t happen overnight. It will take work and education and above all, listening to women, not just the few women who have stuck around, but also the ones who left because they were uncomfortable.

I want to persuade hacker and maker spaces around the world that they are missing out on infinite potential. Hackerspaces.org has some good advice on adding anti-harassment policies to the design patterns for running a space. This is exactly the sort of work that Ada Initiative is good at; their Example anti-harassment policy has been used as a template by many events and organizations.

I’d like to challenge all hackerspace members to do two things in support of my campaign:

* Donate to Ada Initiative! I will match up to $1028 donated when we reach the $4096 goal!

* Add your anti-harassment policy to your organization’s page at hackerspace.org, and link to it from the list on the Geek Feminism wiki. (And if your space doesn’t have a code of conduct or policy, start the ball rolling to implement one!)

I love Double Union. We have set aside a permanent physical space, equipment, organization, and time that is focused on making and creating things together. We have the keys in our hands and the tools to do whatever we like in a safe, supportive environment free from harassment. We agreed to a basic code of conduct and some assumptions we share about behavior in the space, which helps establish trust for us to share knowledge, time, and tools. We try to follow Community anti-harassment standards. We have members who are also part of, or supporters of, Noisebridge, sudo room, LOLspace, Mothership Hackermoms, Ace Monster Toys, and other San Francsico Bay Area spaces.

Double union shopbot

We’re having writers’ groups, book groups, readings, zine workshops, open source software coding, cryptography meetups, circuit hacking, making stuff with our CNC routers, 3D printer, vinyl cutter, drawing and art supplies, and sewing machines — in short, doing whatever we like and learning a lot from each others’ expertise. We celebrate other women’s work and cultural diversity. Our hackerspace is against putting others down for what they do or don’t know. Once we don’t have to fight to prove we are ‘hacker enough’, great things happen.

Double Union’s founding group had the vision to make this space happen because of the pioneering work of the Ada Initiative. Ada Initiative’s demands for policy changes for events and companies, its fierce uncompromising voice, and especially its empowering and inspiring events, are having a good and useful effect to shift our culture.

More AdaCamps, like the ones this year in Berlin, Bangalore, and Portland, will help improve women’s participation in hackerspaces. With your donation, we could potentially host MORE of these fabulous unconferences for women in open tech and culture.

Please join me in donating to Ada Initiative so they can keep on being a positive force for change in the world!

Liz and Cristin smiling at DU

How to test new features in Firefox 34 Aurora

If you’re a fan of free and open source software and would like to contribute to Firefox, join me for some Firefox feature testing!

There are some nifty features under development right now for Firefox 34 including translation in the browser, making voice or video calls (a feature called “Hello” or “Loop”), debugging information for web developers in the Dev Tools Inspector, and recent improvements to HTML5 gaming.

I’ve written step by step instructions on these
ways to test Firefox 34. If you would like to see what it’s like to improve a popular open source project, trying out these tasks is a good introduction.


First, Install the Aurora version of Firefox. It is best to set it up to use multiple profiles. That ensures you don’t use your everyday version of Firefox for testing, so you won’t risk losing your usual profile information. It also makes it easy to restart Firefox with a new, clean profile with all the default settings, very useful for testing. Sometimes I realize I’m running 5 different versions of Firefox at once!

To test “Hello”, try making some voice or video calls from Firefox Aurora. You will need a friend to test with. Or, use two computers that you control. This is a good task to try while joining our chat channels, #qa or #testday on irc.mozilla.org; ask if anyone there wants to test Hello with you. The goal here is mostly to find and report new bugs.

If you test the translation infobar in Aurora you may find some new bugs. This is a fun feature to test. I like trying it on Wikipedia in many different languages, and also looking at newspapers!

If you’re a web developer, you may use Developer Tools in Firefox. I’m asking Aurora users to go through some unconfirmed bug reports, to help improve the Developer Tools Inspector.

If you like games you can test HTML5 web-based games in Firefox Aurora. This helps us improve Firefox and also helps the independent game developers. We have a list of demo games so you can play them, report glitches, and feel like a virtuous open source citizen all at once. Along the way you have opportunities to learn some interesting stuff about how graphics on the web can work (or not work).

Monster madness

These testing tasks are all set up in One and Done, Mozilla QA’s site to start people along the path to joining our open source community. This site was developed with a lot of community contribution including the design and concept by long-time community member Parul and a lot of code by two interns this summer, Pankaj and Maja.

Testing gives a great view into the development process for people who may not (yet) be programmers. I especially love how transparent Mozilla’s process can be. Anyone can report a bug, visible to the entire world in bugzilla.mozilla.org. There are many people watching that incoming stream of bug reports, confirming them and routing them to developer teams, sometimes tagging them as good first bugs for new contributors. Developers who may or may not be Mozilla employees show up in the bugs, like magic . . . if you think of bugmail notifications as magic . . .

It is amazing to see this very public and somewhat anarchic collaboration process at work. Of course, it can also be extremely satisfying to see a bug you discovered and reported, your pet bug, finally get fixed.