A useful Bugzilla trick

At the beginning of February I changed teams within Mozilla and am now working as a release manager. It follows naturally from a lot of the work I’ve already been doing at Mozilla and I’m excited to join the team working with Lukas, Lawrence, and Sylvestre!

I just learned a cool trick for dealing with several bugzilla.mozilla.org bugs at once, on MacOS X.

1) Install Bugzilla Services.

2) Add a keyboard shortcut as Alex Keybl describes in the blog post above. (I am using Control-Command-B)

3) Install the Tree Style Tab addon.

Now, from any text, whether in email, a desktop text file, or anywhere in the browser, I can highlight a bunch of text and bug number will be parsed out of the text. For example, from an email this morning:

Bug 1137050 - Startup up Crash - patch should land soon, potentially risky
David Major seems to think it is risky for the release.

Besides that, we are going to take:
Bug 1137469 - Loop exception - patch waiting for review
Bug 1136855 - print preferences - patch approved
Bug 1137141 - Fx account + hello - patch waiting for review
Bug 1136300 - Hello + share buttons - Mike  De Boer will work on a patch today

And maybe a fix for the ANY query (bug 1093983) if we have one...

I highlighted the entire email and hit the “open in bugzilla” keystroke. This resulted in a Bugzilla list view for the 6 bugs mentioned in the email.

Bugzilla list view example

With BugzillaJS installed, I have an extra option at the bottom of the page, “Open All in Tabs”, so if I wanted to triage these bugs, I can open them all at once. The tabs show up in my sidebar, indented from their parent tab. This is handy if I want to collapse this group of tabs, or close the parent tab and all its children at once (The original list view of these 6 bugs, and each of its individual tabs.) Tree Style Tab is my new favorite thing!

Tree style tabs bugzilla

In this case, after I had read each bug from this morning and closed the tabs, my coworker Sylvestre asked me to make sure I cc-ed myself into all of them to keep an eye on them later today and over the weekend so that when fixes are checked in, I can approve them for release.

Here I did not want to open up every bug in its own tab but instead went for “Change Several Bugs at Once” which is also at the bottom of the page.

Bugzilla batch edit

This batch edit view of bugs is a bit scarily powerful since it will result in bugmail to many people for each bug’s changes. When you need it, it’s a great feature. I added myself to the cc: field all in one swoop instead of having to click each tab open, click around several times in each bug to add myself and save and close the tab again.

It was a busy day yesterday at work but I had a nice time working from the office rather than at home. Here is the view from the SF Mozilla office 7th floor deck where I was working and eating cake in the sun. Cannot complain about life, really.
Mozilla bridge view

Like a warm pool

My new stomping grounds are at a physical rehab swimming pool in San Francisco. It’s called Herbst Pool and it’s at the Pomeroy Recreation and Rehabilitation Center. I used to go to the Betty Wright pool in Palo Alto (that then became C.A.R. and then Abilities United). This seems quite similar. The Center was started in 1952 as a rec center for people with disabilities. It seems to have a lot of day programs, gardens, playgrounds, a gym and art classrooms, and this awesome pool where the water is kept over 90 degrees. This is the important part to me. Cold is painful on my joints and I get stiff quickly. But it also is directly painful on my “bad” leg. Regular room temperature air on my leg feels like burning ice which is why I’m a huge fan of long underwear or at least secret leg warmers (under my jeans). So, warm water RULES.

Pomeroy herbstpool

Like the Betty Wright pool, Herbst Pool or Pomeroy Pool (whatever you want to call it) has a very wide sloping ramp to enter the water, and a variety of PVC wheelchairs useful for going down the ramp. There are also very shallow and broad steps with handrails. It is easy entry into the water. They have all sorts of adaptive equipment for the water. I don’t need that stuff but I appreciate the easy entry and not having to use one of those slow, free-show lift chairs into the water. The pool is a beautiful and relaxing space as there is a skylight, and one wall is entirely windows looking out at the trees bordering Lake Merced. The walls are covered with cheerful murals of cartoon whales.

The locker room and especially the main pool room are kept pretty warm. There are many heat lamps in the pool area. Really fantastic if you can’t tolerate the cold. Of course I always wish it were warmer and had a heated floor… and a sauna…. but it’s the best I’ve ever come across.

Once I am in things are pretty good at the moment. I can walk around slowly in collarbone-deep water, and do squats and other excercises hanging onto a kickboard. When I am not in a flare up, or, in some mythical past before my ankles went awry, I am a strong and good swimmer with good endurance and a long history of loving the surf. I feel super happy in the water, light and gravity-free (as long as it is warm). I’m going every week for physical therapy in a small group that’s run through Potrero Physical Therapy (note: they are awesome.) And I am trying to fit in at least one other pool visit per week, hoping to get to 3 hours total a week.

It is hard for me to get to Pomeroy. When I was gearing up to do this I didn’t find directions on their site for how to get there by bus, and it was really unclear on Google Maps from their address (on Skyline) where the entrance was on a rather large and confusing block of land between three streets. This matters to me because it can be non-trivial for me to get from point A to point B even in my scooter (because of pain, or nasty weather) and if I do’t know exactly where entrances are and what a building or “campus” is like, I’m not going to take my manual wheelchair for sure as it can easily be too difficult for me to go it alone. So here are the details of how to get there for anyone else who is thinking of rehab at this pool.

And, while taking the hour+ long journey from my house to the pool and during the somewhat boring hour of walking back and forth I often think how I should write up a post about the pool. Here it finally is.

Getting to the pool

There is parking and drop off that is level with the pool entrance, which is on Herbst Road and up a little hill. There are at least 5 blue parking spaces and some extra. Other parking is along Herbst and you will meet that “hill” along with a possibly significant walk. From the drop off point in the closest parking lot (where paratransit will drop you) It is a pretty short walk in to the pool. I haven’t measured it or tried it yet. It is too much walk for me to do with out a chair.

There are 2 buses that go nearby, the 23 and the 18. The 18 stops on Herbst Road just outside the Rec Center campus. To get to the pool you will have to cross a non-busy street and then go up a steep slope (a full story, not something I want to do in a manual chair but might in a pinch). There is a ramp for this but again it is at least one full building story high. At the top there is a little garden with benches, and a small (not accessible) playground and grassy area for toddlers. I like eating lunch there. (I think on the other side of the campus there may be an accessible playground.)

Pomeroy courtyard

To take the 23 (the bus I ride to get there) you have to get off at Sloat and Skyline just north east of the zoo. This street crossing is a complete nightmare. It is a 6 or 8 way stop, with cars coming very fast. There are 3 medians you will need to stop on to cross another section of the road. At least it has medians! This is a crossing I would not recommend to anyone blind. Instead get off the 23 at the zoo entrance stop and cross there where there is a light and only 1 median and traffic isn’t coming from 6 unpredictable directions. This will add another 15 minutes to your journey. The Hellish Intersection scares the crap out of me every time. But I still use it because I need to get back home so I can work. The 23rd itself is a pleasant bus, not crowded, bus drivers nice and not super stressed; they are the old style buses with an unfolding ramp instead of a lift. It comes about every 20-30 minutes depending on the time and day. Note, the bus stop going inbound is not marked well and is on one of the medians in the center of the Hellish Intersection.

Hellish intersection sloat skyline

It is half a mile from the Sloat and Skyline bus stop to the entrance to the pool, which takes me about 10 minutes to traverse. One minor problem I have is that the sidewalk along Herbst is often littered with eucalyptus nuts and branches, so I opt for going in the street. Not a big problem as the street isn’t too busy and it’s wide enough to have room.

cost of the pool; who goes there
The pool membership costs $50 a year and you have to have a doctor fill out the application. Then, the swim sessions or exercise classes are something like $8-10 each. There is not really a “drop in” mentality but instead you are expected to sign up for a 10-pack card at the least.

This high cost, and the difficult access, may explain why I have never seen any other wheelchair users at the pool. It is weird to feel like a damn unicorn at a place specifically meant for disabled people. The physical therapy class are mostly people with injuries or recovering from surgery who have PT for a short time covered by their insurance or Medicare and the arthritis exercise class seem to be retired people trying to keep fit. The other main users of the facility seem to be disabled people who have personal care attendants, or who have developmental disabilities and are there as a sort of day camp experience doing garden work, art classes, and basketball. Lots of wheelchair users around the grounds and buildings but none at the pool at the times that I go there. I can see the community that they serve. And I am an outlier in that community and yet this place is also *exactly* what I need (integrated into my life all the time). I do wish that the pool had some sort of option for low(er) cost access, not for me, but for people in the community who are living on disability benefits who would never be able to afford this and yet who are not “in the system” enough to get bused from a group home on paratransit (as I think many of the people hanging out at the Center are).

The other swimmers I have seen are all infant and toddlers with parents bringing them for swim lessons (with no relation to anyone being disabled). It is just a nice warm pool where they have baby swim lessons, like at Petit Baleen. It is lovely and cheerful to see all the kids coming out of the pool. They always seem happy and calm and sort of stimulated. And they make me happy as I think of the nice memories of when Milo and Ada were small.

Probably the kids’ swim lessons are basically a way for middle class non disabled people to financially support the rest of the facility. (Which also takes donations.) And, I think it may also get funding from places like Target that look for a place from which to hire disabled people for low pay and some sort of tax break; whatever happens there, I hope people are paid fairly.

Extra note on Janet Pomeroy who the Center is named after. Thank you wheelchair sports camp lady!

The facilities of the pool
The building itself has automatic doors. There is a station where you can check in and pay for your session. Also two vending machines with drinks and snacks.

The women’s locker room has a big heavy door quite difficult to open. They need a push button for the doors. Inside, there is a (wet) floored room lined with benches. There are a few hooks to hang coats or bags but no lockers at all and no curtained areas for changing. It is just one big room to change in. Probably that is to fit a maximum number of people (and wheelchairs?) into the room. The outcome of this is that many people lock themselves into the three bathroom stalls so I make sure to use the bathroom outside the locker room by the gym. Aside from this minor annoyance I do feel critical of the situation as the upshot is old ladies who have had hip replacements changing into bathing suits in a slippery tiled floor bathroom stall seems like a recipe for badness. (Add in an incoming and outgoing class of non-disabled toddlers to imagine this completely.) I think they are dealing with this by building a new (unisex) changing area for wheelchair users or people with attendants, in the main area of the pool. (The Abilities United pool also has this.) One last bitch about the locker room, it has no tampon machine. Come on folks. It’s a pool bathroom. What more important place are you going to be where you will need emergency tampons?! But I digress.

There are nice showers in the locker room, one with a curtain but all the rest open. The water is hot. Yay. The locker room has a heavy difficult door to get into the pool area.

From there it is cake; there is another shower for rinsing off near the pool and there are some open cubbyholes to put your stuff in if you don’t want to leave it in the (non-lockered) locker room. I notice that getting into the pool many people have both flip flops and walking canes. Some thought to this common situation would be good and it coudl be solved with another set of cubbies and some sort of cane holder that would take twenty-eleven canes. There is always a pile of flip flops and canes right at the point where the handrail begins! Plus, my scooter unless I am confident about the walk from the shower area to the pool and back.

Herbstpool window

emotions and memories of physical therapy
I can’t imagine how many of the ladies in PT with me manage the trip there or even the walk from the parking lot. It is always hard to remember how that works until I get back into that territory of ablilty. Driving + hobbling. I was there for years.

I remember so many times of “rehabbing” in pools over the last 20 years. At Valley Medical Center in the 90s it was depressing and squalid and yet the warm water was so freeing. I could move freely and learned some good exercises. It was hard to talk with the other people there who were all older women while I was in my 20s with a mohawk and two septum rings. These days at least I am middle aged looking and not unacceptable as a possible conversation partner to ladies who have hip replacements.

It was always an ordeal to get to a hospital and navigate it and also inherently depressing. I have swum at YMCAs, JCCs, many city pools, and so on which I far prefer to going to hospital therapy pools. The public pools are way too cold for me to tolerate. I worked up from 15 minutes to 45 minutes in a “normal” temperature pool where athletic people swim laps, but it’s never a good experience. The SF JCC has the second warmest pool in town, plus a hot tub and a sauna, by the way, but it is extremely pricey as well as being ultra clean, pretty, and posh. The CPMC hospital has a warm pool, and you can get PT from it, so that is an option, but last I checked it was not only out of service but you couldn’t “drop in” extra at all; you have to get on a waiting list for a weekly arthritis class and if you miss two classes they boot you to the end of the waiting list. That will not work well for me. So I am very glad that places like the Pomeroy Pool and Abilities United exist.

I thought over my times at the Betty Wright Center which had really good cameraderie and where I got a lot stronger. I remember starting out at Betty Wright crying uncontrollably and feeling that I could not take it but grimly slogging through. That always happens both physically and emotionally. In fact at Pomeroy I know we all cry in there sometimes. Even if not from pain, it is because it is easy to go about our lives somewhat disconnected from our bodies. Being in the water and having nothing else to do but gently move around, it can be an emotional experience. I am thinking directly about what I can and can’t do, and comparing it with other times in the water, often ones where I was hitting rock bottom physically. I notice consistently my problem is not, “not trying hard enough” it is pushing myself way too hard and re-injuring myself or making everything worse. This is the new era of caution for me. Some of this holding back is possible because of working with a behavioral therapist who is very knowledgeable about pain, disability, and chronic illness. Thanks Dr. C.

(There is also the aspect that doing this 3.5 hour journey means I have to put on a lot of psychic armor to deal with strangers, bus drivers, buses with broken lifts, the Hellish Intersection, people cursing at me for being in the way, people staring, people (nicely or otherwise) inquiring about my scooter or wheelchair, people angry with me for not accepting their “help” properly, and all the other things that make leaving the house an Epic Journey. I armor up and am prepared for it all but it is not without emotional effort and cost. It is never simple. For me to get anywhere, I have gone through this process already and have likely had many challenges to basic equanimity.)

Last fall I could not go into the water without soft ankle braces. It hurt too much to have the tiny currents of the water, wiggling my ankles around unpredictably. Right now I can do a lot of squats and toe-raises and walk back and forth for the whole hour. It’s great. I am not swimming yet as my knees and ankles (and i think back) aren’t strengthened enough. I cannot do things I used to be able to do in pool rehab, like write the alphabet with my foot in cursive. Nope nope nope nope nope nope! But, I have improved my gait (again) to be more even and to weight shift instead of limping or shuffling.

My goal right now is to get my ankles strong enough that I can drive to the pool, park there, and use my manual chair. That would be less physically grueling for me than the 2 hours of bus ride and scootering. It usually takes 3 and a half hours out of my day to get there by bus. By driving, I could be gone from work 2 hours instead. That means I could swim more often. My real goal though is simply to maintain this level of water-walking and aerobic exercise for health, flexibility, and strength without injuring myself. This is already the longest stretch of time I have consistently made it to PT or exercise without messing myself up, and also doing daily PT at home. Huzzah!

A last note, huge props to Pomeroy staff for letting me use a conference room in their office one day last fall as I wanted to be at a crucial team meeting but didn’t want to skip my PT appointment. They let me walk in out of the blue and use their office and their wifi and close the door for privacy. Super nice of them. I was able to give one of their staff all the details about TravelScoots and other lightweight scooters in return. I still feel I owe them some free computer advice or help! But, I thanked them for supporting a person “in the community” (me) in their job and also for helping make Firefox better (ha!) by getting me to my meeting on time!

Automated test harnesses for Firefox, zooming out a bit

In December I was going through some of the steps to be able to change, fix, and interpret the results of a small subset of the gazillion automated tests that Mozilla runs on Firefox builds. My last post arrived at the point of being able to run mochitests locally on my laptop on the latest Firefox code. Over the holidays I dug further into the underlying situation and broaded my perspective. I wanted to make sure that I didn’t end up grubbing away at something that no one cared about or wasn’t missing the big picture.

My question was, how can I, or QA in general, understand where Firefox is with e10s (Electrolysis, the code name for the multiprocess Firefox project). Can I answer the question, are we ready to have e10s turned on by default in Firefox — for the Developer Edition (formerly known as Aurora), for Beta, and for a new release? What criteria are we judging by? Complicated. And given those things, how can we help move the project along and ensure good quality; Firefox that works as well as or, we hope, better than, Firefox with e10s not enabled? My coworker Juan and I boiled it down to basically, stability (lowering the crash rate) and automated test coverage. To answer my bigger questions about how to improve automated test coverage I had to kind of zoom out, and look from another angle. For Firefox developers a lot of what I am about to describe is basic knowledge. It seems worth explaining, since it took me significant time to figure out.

First of all let’s look at treeherder. Treeherder is kind of the new TBPL, which is the old new Tinderbox. It lets us monitor the current state of the code repositories and the tests that run against them. It is a window into Mozilla’s continuous integration setup. A battery of tests are poised to run against Firefox builds on many different platforms. Have a look!

Current view of mozilla-central on treeherder

Treeherder2

Edward Tufte would have a cow. Luckily, this is not for Tufte to enjoy. And I love it. You can just keep digging around in there, and it will keep telling you things. What a weird, complicated gold mine.

Digression! When I first started working for Mozilla I went to the Automation and Tools team work week where they all came up with Treeherder. We wanted to name it something about Ents, because it is about the Tree(s) of the code repos. I explained the whole thing to my son, I think from the work week, which awesomely was in London. He was 11 or 12 at the time and he suggested the name “Yggdrazilla” keeping the -zilla theme and in reference to Yggdrasil, the World-Tree from Norse mythology. My son is pretty awesome. We had to reject that name because we can’t have more -zilla names and also no one would be able to spell Yggdrasil. Alas! So, anyway, treeherder.

The left side of the screen describes the latest batch of commits that were merged into mozilla-central. (The “tree” of code that is used to build Nightly.) On the right, there are a lot of operating systems/platforms listed. Linux opt (optimized version of Firefox for release) is at the top, along with a string of letters and numbers which we hope are green. Those letter and numbers represent batches of tests. You can hover over them to see a description. The tests marked M (1 2 3 etc) are mochitests, bc1 is mochitest-browser-chrome, dt are developer tools tests, and so on. For linux-opt you can see that there are some batches of tests with e10s in the name. We need the mochitest-plain tests to run on Firefox if it has e10s not enabled, or enabled. So the tests are duplicated, possibly changed to work under e10s, and renamed. We have M(1 2 3 . . .) tests, and also M-e10s(1 2 3 . . . ). Tests that are green are all passing. Orange means they aren’t passing. I am not quite sure what red (busted) means (bustage in the tree! red alert!) but let’s just worry about orange. (If you want to read more about the war on orange and what all this means, read Let’s have more green trees from Vaibhav’s blog. )

I kept asking, in order to figure out what needed doing that I could usefully do within the scope of As Soon As Possible, “So, what controls what tests are in which buckets? How do I know how many there are and what they are? Where are they in the codebase? How can I turn them on and off in a way that doesn’t break everything, or breaks it productively?” Good questions. Therefore the answers are long.

There are many other branches of the code other than mozilla-central. Holly is a branch where the builds for all the platforms have e10s enabled. (Many of these repos or branches or twigs or whatever, are named after different kinds of tree.) The tests are the standard set of tests, not particularly tweaked to allow for e10s. We can see what is succeeded and failing on treeherder’s view of holly. A lot of tests are orange on holly! Have a look at holly by clicking through on the link above. Here is a picture of the current state of holly.

Treeherder holly1

If you click a batch of tests where there are some failures — an orange one — then a new panel will open up in treeherder! I will pick a juicy looking one. Right now, for MacOS 10.6 opt, M(2) is orange. Clicking it gives me a ton of info. Scrolling down a bit in the bottom left panel tells me this:

mochitest-plain-chunked 164546/12/14136

The first number is how many tests ran. Scary. Really? 164546 tests ran? Kind of. This is counting assertions, in other words, “is” statements from SimpleTest. The first number lists how many assertions passed. The second number is for assertion failures and the third is for “todo” statements.

The batches of tests running on holly are all running against an e10s build of Firefox. Anything that’s consistently green on holly, we can move over to mozilla-central by making some changes in mozharness. I asked a few people how to do this, and Jim Matthies helpfully pointed me at a past example in Bug 1061014. I figured I could make a stab at adding some of the newly passing tests in Bug 1122901.

As I looked at how to do this, I realized I needed commit access level 2 so I filed a bug to ask for that. And, I also tried merging mozilla-central to holly. That was ridiculously exciting though I hadn’t fixed anything yet. I was just bringing the branch up to date. On my first try doing this, I immediately got a ping on IRC from one of the sheriffs (who do merges and watch the state of the “tree” or code repository) asking me why I had done something irritating and wrong. It took us a bit to figure out what had happened. When I set up mercurial, the setup process and docs told me to install a bunch of Mozilla specific hg extensions. So, I had an extension set up to post to bugzilla every time I updated something. Since I was merging several weeks of one branch to another this touched hundreds of bugs, sending bugmail to untold numbers of people. Mercifully, Bugzilla cut this off after some limit was reached. It was so embarrassing I could feel myself turning beet red as I thought of how many people just saw my mistake and wondered what the heck I was doing. And yet just had to forge onwards, fix my config file, and try it again. Super nicely, Clint Talbert told me that the first time he tried pushing some change he broke all branches of every product and had no idea what had happened. Little did he know I would blog about his sad story to make myself feel less silly….. That was years ago and I think Mozilla was still using cvs at that point! I merged mozilla-central to holly again, did not break anything this time, and watched the tests run and gradually appear across the screen. Very cool.

I also ended up realizing that the changes to mozharness to turn these batches of tests on again were not super obvious and to figure it out I needed to read a 2000-line configuration file which has somewhat byzantine logic. I’m not judging it, it is clearly something that has grown organically over time and someone else probably in release engineering is an expert on it and can tweak it casually to do whatever is needed.

Back to our story. For a batch of tests on holly that have failures and thus are showing up on treeherder as orange, it should be possible to go through the logs for the failing tests, figure out how to turn them off with some skip-if statements, filing bugs for each skipped failing test. Then, keep doing that till a batch of tests is green and it is ready to be moved over.

That seems like a reasonable plan for improving the automated test landscape, which should help developers to know that their code works in Firefox whether e10s is enabled or not. In effect, having the tests should mean that many problems are prevented from ever becoming bugs. The effect of this test coverage is hard to measure. How do you prove something didn’t happen? Perhaps by looking at which e10s tests fail on pushes to the try server. Another issue here is that there are quite a lot of tests that have been around for many years. It is hard too know how many of them are useful, whether there are a lot of redundant tests, in short whether there is a lot of cruft and there probably is. With a bit more experience in the code and fixing and writing tests it would be easier to judge the usefulness of these tests.

I can also see that, despite this taking a while to figure out (and to even begin to explain) it is a good entry point to contributing to Firefox. It has more or less finite boundaries. If you can follow what I just described in this post and my last post, and you can read and follow a little python and javascript, then you can do this. And, if you were to go through many of the tests, over time you would end up understanding more about how the codebase is structured.

As usual when I dive into anything technical at Mozilla, I think it’s pretty cool that most of this work happens in the open. It is a great body of data for academics to study, it’s an example of how this work actually happens for anyone interested in the field, and it’s something that anyone can contribute to if they have the time and interest to put in some effort.

This post seems very plain with only some screenshots of Treeherder for illustration. Here, have a photo of me making friends with a chicken.

Liz with chicken