Firefox release, diner interlude, gingerbread

After some especially intense weeks at work (leading up to this week’s Firefox 66 release) we then had to respond to the Pwn2own exploits. It went pretty much like last year when we also released in under a day; everyone worked hard and some people worked long hours to make it happen. I’m so proud to be part of Mozilla and get to work with such smart and dedicated people.

Between the release on Tuesday and the security-driven release on Thurs/Fri I spent a bunch of time going over and summarizing the various problems & uncertainties. It is very interesting to try and judge importance, impact, and “risk” in a software release. On the one hand you are offering a lot of improvements and fixes but you will inevitably also break something else for some other population of users. Is it an overall improvement for most users? Is some particular population (that may not be a majority) impacted by a new problem? All users of a particular site will crash with the new version, but with the old version, everyone editing a document with a Korean keyboard can’t use their backspace key. The different issues don’t cancel each other out, but still have to be compared and weighed. You start out reading bug and crash reports, and end up pondering epistemology and trying to steer a course in decision making through rocky waters — the geography of operating systems, hardware, web sites, and standards where nothing ever stands still. In the end the pattern seems to remain the same: cautiously release, pause and fix the worst new problems as best we can, then go full throttle. It will never be rock solid, like how you’d have to be releasing software for NASA, though, amazingly, that seems to be what people expect.

By the way, my boots came out great! I stripped off the glaze and brown dye with acetone, dyed them with Angelus red and wine-red and a bit of black, mink-oiled them, put on a Resolene sealant coat, then a coat of neutral polish. Now, I can look down at my boots and think YAY! RED! rather than seeing a sort of yucky ochre.

red boots

Ended my work day early (after pulling long hours for a while) and headed out to my sister’s house for the afternoon. BART continues to entertain me. I had a nice look at Oakland 19th street station and its beautiful deep varigated blue tiling. At first I thought they were bricks, but they’re actually thick tiles laid over a wire mesh. I may make this station the “Water” artifact puzzle for the game in honor of the lovely tiles. (Originally that was going to be Lake Merritt, but it needs to be a station on the Red line.)

Realized on the way that I hadn’t had lunch and barely had breakfast – I stopped at Mama’s Royal Cafe & had a biscuit sandwich and endless cups of decaf. The waiter spotted me playing Ingress and turned out to be a long time player on my same team (the Resistance or blue team)! I tried to work on my game a bit, but didn’t get far, too fried to really think things through. This is a great cafe — relaxing, gorgeous, cozy, good food.

At Laura’s I helped my nephew make gingerbread for his school bake sale (5 cakes’ worth so we mixed it all up in a huge pot.) I then ate like half a pan of gingerbread (for dinner I guess). Laura and I worked on our wordpress sites – I needed to change the theme of mine and she was just setting up a new one. Also, a couple of years ago I migrated in some way that broke all my images from past posts. That is now fixed! Huzzah! I also ended up minorly hacking this new theme to force it not to do post excerpts, which I fucking hate. Just let me scroll forever and read other people’s long posts! But no… the fashion is so heavily for showing excerpts and making people click a zillion times and load new pages. Bah. I don’t CARE if you’re on your phone – you can scroll! It’s been a long time since I even looked at php but, reassuringly, it all came back to me.

gingerbread mixing

It was my son’s 19th birthday on Friday as well and he had a final and then was heading off to a weekend camping LARP with his dad. I’ll see him next week! But, I have to admit I went and looked at pics of him from babyhood on in a sentimental way. He’s so grown up and wonderful, and I’m so so proud of him.

milo-hashtag-sweater

Actual work work

I should mention my actual work while I’m blogging about stuff. “My” Firefox release is now in beta, Firefox 66. We have a team of release managers so can rotate ownership of upcoming releases. Usually this means you’re in the hot seat every 3 releases, about every 4 months. We have a bigger team now so there is more breathing room and we can support each other more; basically that means we’re less burned out. Yay! (A couple of years ago there were like, 3 of us and one was the team manager.)

The feeling of ownership is somewhat silly since it is an enormous collaborative effort, but it helps in the job to follow one version closely, you care about it and are able to filter out & not pay so much attention to the other versions in development. (In practice, that is a lot of email I can skim and archive when it isn’t for “my” release.) The release engineers also rotate through version ownership. It’s a good system.

If you’d like to see all the bugs fixed for Firefox 66 so far, between its birth as Firefox Nightly last December and now, here’s a Bugzilla query which you shouldn’t click if you’re on your phone or a slow connection because it’s going to take a while to load – it will show a list of the 3000+ bugs already fixed. Probably don’t click that unless you really want to wait a minute and a half for the page to load.

While Firefox 66 is in beta, I’m watching for fixes in the version right behind it, Firefox 67, that we might be able to bring to users more quickly. We backport, or uplift, between 200 and 400 issues while a version is in beta. Here’s what I’ve approved to uplift so far, patches from about 40 bugs. It’s helpful when looking at these lists to sort the information different ways (by clicking on the column headers.) These next few weeks till March 19 (the release date) will be my crunch time.

Today I did about one million different things. Hilariously, as so much of our work is right out in the open, you can see some of those one million things (in 68 bugs), at least the bits of it that were me reading a lot of bug reports and figuring out what to do with them (poke someone from another team, engineers, QA, etc, add relevant tags or other info, and while I’m looking, decide how important it is for it to be in Firefox 66.)

Work from home looks

I really enjoyed these cartoons today:

There are 4 pages of work from home outfits of uncanny accuracy to how I normally dress at least, until noon or so. Some details vary, but I totally put my hair (short as it is) into a top of the head ponytail and I tuck my tattered pajamas into my (too big, mens’) socks. The t-shirt boob tuck is also just TOO REAL. I did wear actual pants today. My t-shirt is one that I got in 1988 at the Guinness factory in Dublin which has miraculously survived all this time with no holes and amazing softness.

The artist has an adorable etsy shop!

Playing catch-up and tweaking small habits

Coming back to work after 6 weeks leave – Here is my plan.

* Bugzilla needinfo: A few people were still trying to talk to me while I was out so, doing that first.
* Deal with email: inbox ~2000, anything older than a week, I will archive it in large batches by filtering and then read whatever is leftover and seems important.
* Study the calendar and absorb where we are in the current release cycle
* Read meeting notes: my team meetings, product cross-functional, recent channel meetings
* Read the post-mortem notes for the 62 release, and the subsequent dot releases, though that may depress me (I got sick in the last stages of this (“my”) release)
* Figure out what I should be doing next, probably that means prep for Firefox 65 if I’ll be the release owner for that, and I believe for ESR 60.4.0 (that goes along with the 64 release)
* I can always peck away at some new regression triage

That’s plenty for the next day or so.

Meanwhile during my recovery from surgery I have been trying to change or add small daily habits. Another list!

* I have integrated doing 3-5 minutes of gentle tai chi a few times a day (first thing in the morning, mid-morning, and before lunch are the most important).
* Writing and drawing around 15 minutes each, even if not particularly inspired, I have to stop whatever else I’m doing and give it a try. Sometimes I end up going further or having good ideas!
* Making sure to go outside and lie in the sun, while we still have sunny patches on the front porch and on our back patio. Sun hour is currently around 9:45 to 11 in the front. By late October we won’t have any more direct sun, not until March. So I had better take advantage while it’s still here (And also, it structures my day nicely — important, as Coleridge says, to organize the hours and give them a soul.)
* Posting more here, in my more private journal, or writing in my notebook rather than just on Facebook, which has become an ingrained bad habit.
* I moved Twitter off the front page of my phone and replaced it with Duolingo and Feedly, so now when I have the impulse to zone out reading Twitter (and rage-tweeting about politics) I instead either read some actual blogs that I like, or do some poking away at Duolingo. (Spanish and French). I also still look over Hacker News which usually has something worth reading, if only so that I can laugh at n-gate afterwards
* Sending snail mail. When I realized I’d be in bed for weeks, I asked for postcards from people, and got around 100 cards! It was very cheering. I’m still answering that batch of cards.

Here’s a little sharpie marker illustration of a black cat from one of my drawing sessions!
drawing of a black cat

Nifty addition to treeherder

At work I just found out we have a nice change in Treeherder, the tool that monitors current Firefox (and other) builds. Without writing my usual 10 pages of explanation and backstory, I’ll just say that Treeherder shows whether the release engineering infrastructure has started the release promotion process to take the builds from the changeset specified and do some super mystery magic on them to make them suitable for actual release to Firefox users. We get an email from the notification system that this happened, but now we can follow along in the same tool we use to watch tests pass or fail.

gecko decision task for firefox and fennec

In this case, I used a different tool (Ship-It) to set up the parameters to take the test builds for the 1a24837f9ed232d8d2dc4535d11ee53c9847b109 changeset (say it 6 times, fast) for Firefox 59 beta 7 (plus the android and developer edition versions) and declare us ready to kick off this process. Instead of checking email notifications that only go to a particular mailing list, or diving straight into TaskCluster, I can see that near the end of the enormous list of automated tests running on various platforms, “Gecko Decision Task”. This tells us that the builds are now being “promoted”. I haven’t tried drilling deeper yet, but if you click on any of the links there (such as “promote_firefox”) then you get even more detail, including links to TaskCluster and its myriad, confusing joys.

As usual, I think it’s damn cool that all this happens “in the open” at Mozilla, so you don’t even have to be logged into anything to see quite a lot of detail. Anyone with an interest can learn a lot here, or even get involved and contribute, because of this level of commitment to transparency. It’s a good way to find out if you like the work, or to get work experience that you can easily show off in the future.

Captain's log, stardate 2016-10-26

Once again I resolve to write about my work at Mozilla as a Firefox release manager. It’s hard to do, because even the smallest thing could fill LONG paragraphs with many links! Since I keep daily notes on what I work on, let me try translating that in brief. When moved, maybe I’ll go into depth.

This week we are coming into the home stretch of a 7 week release cycle. “My” release for right now is Firefox 49, which was released on I’m still juggling problems and responses and triaging for that every day. In a week and a half, we were scheduled to release Firefox 50. Today after some discussion we pushed back that schedule by a week.

Meanwhile, I am also helping a new release manager (Gerry) to go through tracked bugs, new regressions, top crash reports, and uplift requests for Aurora/Developer Edition (Firefox 51). I’m going through uplift requests for Firefox 45.5.0esr, the extended support release. There’s still more – I paid some attention to our “update orphaning” project to bring users stuck on older versions of Firefox forward to the current, better and safer versions.

As usual, this means talking to developers and managers across pretty much all the teams at Mozilla, so it is never boring. Our goal is to get fixes and improvements as fast as possible while making sure, as best we can, that those fixes aren’t causing worse problems. We also have the interesting challenges of working across many time zones around the world.

Firefox stuffed animal  installed

Today I had a brief 1:1 meeting with my manager and went to the Firefox Product cross-functional meeting. I always find useful as it brings together many teams. There was a long Firefox team all hands discussion and then I skipped going to another hour long triage meeting with the platform/firefox engineering managers. Whew! We had a lively discussion over the last couple of days about a performance regression (Bug 1304434). The issues are complicated to sort out. Everyone involved is super smart and the discussions have a collegiate quality. No one is “yelling at each other”, while we regularly challenge each other’s assumptions and are free to disagree – usually in public view on a mailing list or in our bug tracker. This is part of why I really love Mozilla. While we can get a bit heated and stressed, overall, the culture is good. YMMV of course.

By that time (11am) I had been working since 7:30am, setting many queries in bugs, and on IRC, and in emails into motion and made a lot of small but oddly difficult decisions. Often this meant exercising my wontfix powers on bugs — deferring uplift (aka “backport” ) to 50, 51, or leaving a fix in 52 to ride the trains to release some time next year. As I’m feeling pretty good I headed out to have lunch and work from a cafe downtown (Mazarine – the turkey salad sandwich was very good!)

This afternoon I’m focusing on ESR 45, and Aurora 51, doing a bit more bug triage. There are a couple of ESR uplifts stressing me out — seriously, I was having kittens over these patches — but now that we have an extra week until we release, it feels like a better position for asking for a 2nd code review, a bit more time for QA, and so on.

Heading out soon for drinks with friends across the street from this cafe, and then to the Internet Archive’s 20th anniversary party. Yay, Internet Archive!

That Zarro Boogs feeling

This is my third Firefox release as release manager, and the fifth that I’ve followed closely from the beginning to the end of the release cycle. (31 and 36 as QA lead; 39, 43, and 46 as release manager.) This time I felt more than usually okay with things, even while there was a lot of change in our infrastructure and while we started triaging and following even more bugs than usual. No matter how on top of things I get, there is still chaos and things still come up at the last minute. Stuff breaks, and we never stop finding new issues!

I’m not going into all the details because that would take forever and would mostly be me complaining or blaming myself for things. Save it for the post-mortem meeting. This post is to record my feeling of accomplishment from today.

During the approximately 6 week beta cycle of Firefox development we release around 2 beta versions per week. I read through many bugs nominated as possibly important regressions, and many that need review and assessment to decide if the benefit of backporting warrants the risk of breaking something else.

During this 7 week beta cycle I have made some sort of decision about at least 480 bugs. That usually means that I’ve read many more bugs, since figuring out what’s going on in one may mean reading through its dependencies, duplicates, and see-alsos, or whatever someone randomly mentions in comment 45 of 96.

And today I got to a point I’ve never been at near the end of a beta cycle: Zarro Boogs found!

list of zero bugs

This is what Bugzilla says when you do a query and it returns 0. I think everyone likes saying (and seeing) “Zarro Boogs”. Its silliness expresses the happy feeling you get when you have burned down a giant list of bugs.

This particular query is for bugs that anyone at all has nominated for the release management team to pay attention to.

Here is the list of requests for uplift (or backporting, same thing) to the mozilla-beta repo:

more zero pending requests

Yes!! Also zarro boogs.

Since we build our release candidate a week (or a few days) from the mozilla-release repo, I check up on requests to uplift there too:

list of zero pending requests

PEAK ZARRO BOOGS.

For the bugs that are unresolved and that I’m still tracking into the 46 release next week, it’s down to 4: Two fairly high volume crashes that may not be actionable yet, one minor issue in a system addon that will be resolved in a planned out-of-band upgrade, and one web compatibility issue that should be resolved soon by an external site. Really not bad!

Our overall regression tracking has a release health dashboard on displays in many Mozilla offices. Blockers, 0. Known new regressions that we are still working on and haven’t explicitly decided to wontfix: 1. (But this will be fixed by the system addon update once 46 ships.) Carryover regressions: 41; about 15 of them are actually fixed but not marked up correctly yet. The rest are known regressions we shipped with already that still aren’t fixed. Some of those are missed uplift opportunities. We will do better in the next release!

In context, I approved 196 bugs for uplift during beta, and 329 bugs for aurora. And, we fix several thousands of issues in every release during the approx. 12 week development cycle. Which ones of those should we pay the most attention to, and which of those can be backported? Release managers act as a sort of Maxwell’s Demon to let in only particular patches …

Will this grim activity level for the past 7 weeks and my current smug feeling of being on top of regression burndown translate to noticeably better “quality”… for Firefox users? That is hard to tell, but I feel hopeful that it will over time. I like the feeling of being caught up, even temporarily.

liz in sunglasses with a drink in hand

Here I am with drink in hand on a sunny afternoon, toasting all the hard working developers, QA testers, beta users, release engineers, PMs, managers and product folks who did most of the actual work to fix this stuff and get it firmly into place in this excellent, free, open source browser. Cheers!

A useful Bugzilla trick

At the beginning of February I changed teams within Mozilla and am now working as a release manager. It follows naturally from a lot of the work I’ve already been doing at Mozilla and I’m excited to join the team working with Lukas, Lawrence, and Sylvestre!

I just learned a cool trick for dealing with several bugzilla.mozilla.org bugs at once, on MacOS X.

1) Install Bugzilla Services.

2) Add a keyboard shortcut as Alex Keybl describes in the blog post above. (I am using Control-Command-B)

3) Install the Tree Style Tab addon.

Now, from any text, whether in email, a desktop text file, or anywhere in the browser, I can highlight a bunch of text and bug number will be parsed out of the text. For example, from an email this morning:

Bug 1137050 - Startup up Crash - patch should land soon, potentially risky
David Major seems to think it is risky for the release.

Besides that, we are going to take:
Bug 1137469 - Loop exception - patch waiting for review
Bug 1136855 - print preferences - patch approved
Bug 1137141 - Fx account + hello - patch waiting for review
Bug 1136300 - Hello + share buttons - Mike  De Boer will work on a patch today

And maybe a fix for the ANY query (bug 1093983) if we have one...

I highlighted the entire email and hit the “open in bugzilla” keystroke. This resulted in a Bugzilla list view for the 6 bugs mentioned in the email.

Bugzilla list view example

With BugzillaJS installed, I have an extra option at the bottom of the page, “Open All in Tabs”, so if I wanted to triage these bugs, I can open them all at once. The tabs show up in my sidebar, indented from their parent tab. This is handy if I want to collapse this group of tabs, or close the parent tab and all its children at once (The original list view of these 6 bugs, and each of its individual tabs.) Tree Style Tab is my new favorite thing!

Tree style tabs bugzilla

In this case, after I had read each bug from this morning and closed the tabs, my coworker Sylvestre asked me to make sure I cc-ed myself into all of them to keep an eye on them later today and over the weekend so that when fixes are checked in, I can approve them for release.

Here I did not want to open up every bug in its own tab but instead went for “Change Several Bugs at Once” which is also at the bottom of the page.

Bugzilla batch edit

This batch edit view of bugs is a bit scarily powerful since it will result in bugmail to many people for each bug’s changes. When you need it, it’s a great feature. I added myself to the cc: field all in one swoop instead of having to click each tab open, click around several times in each bug to add myself and save and close the tab again.

It was a busy day yesterday at work but I had a nice time working from the office rather than at home. Here is the view from the SF Mozilla office 7th floor deck where I was working and eating cake in the sun. Cannot complain about life, really.
Mozilla bridge view

Automated test harnesses for Firefox, zooming out a bit

In December I was going through some of the steps to be able to change, fix, and interpret the results of a small subset of the gazillion automated tests that Mozilla runs on Firefox builds. My last post arrived at the point of being able to run mochitests locally on my laptop on the latest Firefox code. Over the holidays I dug further into the underlying situation and broaded my perspective. I wanted to make sure that I didn’t end up grubbing away at something that no one cared about or wasn’t missing the big picture.

My question was, how can I, or QA in general, understand where Firefox is with e10s (Electrolysis, the code name for the multiprocess Firefox project). Can I answer the question, are we ready to have e10s turned on by default in Firefox — for the Developer Edition (formerly known as Aurora), for Beta, and for a new release? What criteria are we judging by? Complicated. And given those things, how can we help move the project along and ensure good quality; Firefox that works as well as or, we hope, better than, Firefox with e10s not enabled? My coworker Juan and I boiled it down to basically, stability (lowering the crash rate) and automated test coverage. To answer my bigger questions about how to improve automated test coverage I had to kind of zoom out, and look from another angle. For Firefox developers a lot of what I am about to describe is basic knowledge. It seems worth explaining, since it took me significant time to figure out.

First of all let’s look at treeherder. Treeherder is kind of the new TBPL, which is the old new Tinderbox. It lets us monitor the current state of the code repositories and the tests that run against them. It is a window into Mozilla’s continuous integration setup. A battery of tests are poised to run against Firefox builds on many different platforms. Have a look!

Current view of mozilla-central on treeherder

Treeherder2

Edward Tufte would have a cow. Luckily, this is not for Tufte to enjoy. And I love it. You can just keep digging around in there, and it will keep telling you things. What a weird, complicated gold mine.

Digression! When I first started working for Mozilla I went to the Automation and Tools team work week where they all came up with Treeherder. We wanted to name it something about Ents, because it is about the Tree(s) of the code repos. I explained the whole thing to my son, I think from the work week, which awesomely was in London. He was 11 or 12 at the time and he suggested the name “Yggdrazilla” keeping the -zilla theme and in reference to Yggdrasil, the World-Tree from Norse mythology. My son is pretty awesome. We had to reject that name because we can’t have more -zilla names and also no one would be able to spell Yggdrasil. Alas! So, anyway, treeherder.

The left side of the screen describes the latest batch of commits that were merged into mozilla-central. (The “tree” of code that is used to build Nightly.) On the right, there are a lot of operating systems/platforms listed. Linux opt (optimized version of Firefox for release) is at the top, along with a string of letters and numbers which we hope are green. Those letter and numbers represent batches of tests. You can hover over them to see a description. The tests marked M (1 2 3 etc) are mochitests, bc1 is mochitest-browser-chrome, dt are developer tools tests, and so on. For linux-opt you can see that there are some batches of tests with e10s in the name. We need the mochitest-plain tests to run on Firefox if it has e10s not enabled, or enabled. So the tests are duplicated, possibly changed to work under e10s, and renamed. We have M(1 2 3 . . .) tests, and also M-e10s(1 2 3 . . . ). Tests that are green are all passing. Orange means they aren’t passing. I am not quite sure what red (busted) means (bustage in the tree! red alert!) but let’s just worry about orange. (If you want to read more about the war on orange and what all this means, read Let’s have more green trees from Vaibhav’s blog. )

I kept asking, in order to figure out what needed doing that I could usefully do within the scope of As Soon As Possible, “So, what controls what tests are in which buckets? How do I know how many there are and what they are? Where are they in the codebase? How can I turn them on and off in a way that doesn’t break everything, or breaks it productively?” Good questions. Therefore the answers are long.

There are many other branches of the code other than mozilla-central. Holly is a branch where the builds for all the platforms have e10s enabled. (Many of these repos or branches or twigs or whatever, are named after different kinds of tree.) The tests are the standard set of tests, not particularly tweaked to allow for e10s. We can see what is succeeded and failing on treeherder’s view of holly. A lot of tests are orange on holly! Have a look at holly by clicking through on the link above. Here is a picture of the current state of holly.

Treeherder holly1

If you click a batch of tests where there are some failures — an orange one — then a new panel will open up in treeherder! I will pick a juicy looking one. Right now, for MacOS 10.6 opt, M(2) is orange. Clicking it gives me a ton of info. Scrolling down a bit in the bottom left panel tells me this:

mochitest-plain-chunked 164546/12/14136

The first number is how many tests ran. Scary. Really? 164546 tests ran? Kind of. This is counting assertions, in other words, “is” statements from SimpleTest. The first number lists how many assertions passed. The second number is for assertion failures and the third is for “todo” statements.

The batches of tests running on holly are all running against an e10s build of Firefox. Anything that’s consistently green on holly, we can move over to mozilla-central by making some changes in mozharness. I asked a few people how to do this, and Jim Matthies helpfully pointed me at a past example in Bug 1061014. I figured I could make a stab at adding some of the newly passing tests in Bug 1122901.

As I looked at how to do this, I realized I needed commit access level 2 so I filed a bug to ask for that. And, I also tried merging mozilla-central to holly. That was ridiculously exciting though I hadn’t fixed anything yet. I was just bringing the branch up to date. On my first try doing this, I immediately got a ping on IRC from one of the sheriffs (who do merges and watch the state of the “tree” or code repository) asking me why I had done something irritating and wrong. It took us a bit to figure out what had happened. When I set up mercurial, the setup process and docs told me to install a bunch of Mozilla specific hg extensions. So, I had an extension set up to post to bugzilla every time I updated something. Since I was merging several weeks of one branch to another this touched hundreds of bugs, sending bugmail to untold numbers of people. Mercifully, Bugzilla cut this off after some limit was reached. It was so embarrassing I could feel myself turning beet red as I thought of how many people just saw my mistake and wondered what the heck I was doing. And yet just had to forge onwards, fix my config file, and try it again. Super nicely, Clint Talbert told me that the first time he tried pushing some change he broke all branches of every product and had no idea what had happened. Little did he know I would blog about his sad story to make myself feel less silly….. That was years ago and I think Mozilla was still using cvs at that point! I merged mozilla-central to holly again, did not break anything this time, and watched the tests run and gradually appear across the screen. Very cool.

I also ended up realizing that the changes to mozharness to turn these batches of tests on again were not super obvious and to figure it out I needed to read a 2000-line configuration file which has somewhat byzantine logic. I’m not judging it, it is clearly something that has grown organically over time and someone else probably in release engineering is an expert on it and can tweak it casually to do whatever is needed.

Back to our story. For a batch of tests on holly that have failures and thus are showing up on treeherder as orange, it should be possible to go through the logs for the failing tests, figure out how to turn them off with some skip-if statements, filing bugs for each skipped failing test. Then, keep doing that till a batch of tests is green and it is ready to be moved over.

That seems like a reasonable plan for improving the automated test landscape, which should help developers to know that their code works in Firefox whether e10s is enabled or not. In effect, having the tests should mean that many problems are prevented from ever becoming bugs. The effect of this test coverage is hard to measure. How do you prove something didn’t happen? Perhaps by looking at which e10s tests fail on pushes to the try server. Another issue here is that there are quite a lot of tests that have been around for many years. It is hard too know how many of them are useful, whether there are a lot of redundant tests, in short whether there is a lot of cruft and there probably is. With a bit more experience in the code and fixing and writing tests it would be easier to judge the usefulness of these tests.

I can also see that, despite this taking a while to figure out (and to even begin to explain) it is a good entry point to contributing to Firefox. It has more or less finite boundaries. If you can follow what I just described in this post and my last post, and you can read and follow a little python and javascript, then you can do this. And, if you were to go through many of the tests, over time you would end up understanding more about how the codebase is structured.

As usual when I dive into anything technical at Mozilla, I think it’s pretty cool that most of this work happens in the open. It is a great body of data for academics to study, it’s an example of how this work actually happens for anyone interested in the field, and it’s something that anyone can contribute to if they have the time and interest to put in some effort.

This post seems very plain with only some screenshots of Treeherder for illustration. Here, have a photo of me making friends with a chicken.

Liz with chicken