Alternate title ideas: “It’s not all s/Tryserver/Try”  or “What I should have done, and didn’t”

I bet you get the point by now. Today I caused a fairly lengthy, unnecessary downtime on Try.  Now that I’m writing this, things are under control again and there’s a few small niggly bits left but nothing that will keep me up at night.

It all started with a bug about graphserver posts from tests not getting through because they were looking for MozillaTry (the tinderbox name for Try) but instead the graphserver only knew about Tryserver (the branch name for Try) and nothing was using Try (except the repo for Try) which is what it ought to have been doing in the first place.

Now that I’ve been adding a lot of project branches in a short amount of time, certain things have become more streamlined and so I felt that the best option was to go through and rename Tryserver/MozillaTry to Try everywhere so that from the repo going forward, everything was the same. This has been working extremely well for our project branches and helps make setup a snap.

Here’s where it gets all broken. I approached this bug with a quick swipe at this problem was superficial and ended up causing some preventable burning.  I shall now list for you (and future me) what I did and what I should have done:

Did:

• hg rename on configs for desktop
• branch configs for s/tryserver/try
• updated graphserver branch name to Try
• a quick downtime window from 10am – 12pm in order to prevent builds from getting split into two different upload dirs

Should have done:

• hg rename on configs for mobile
• grep of buildbotcustom for “tryserver” as we have special casing for it in several files
• log uploader and post_upload scripts to make sure everything about the try build was going to the right place
• updated the dir permissions on ftp for the new upload location and ensure that the archive is on nfs mount
• edited cronjobs on staging to catch the new try builds
• updated graphserver machines table for each try platform’s builder name
• more notice for downtime, with a 4 hour window that would have allowed a test push to make sure everything was wired up correctly  
• updated the treeclosure hook to include the new tinderbox page

Some of the things I should have done didn’t have an impact on the burning/try closure but it’s fair to say that if I had done a staging round of all my plans first I would have caught more of the obvious things that I missed. I would have then planned the downtime better and been prepared to ensure the disturbance would have been minimal since this was, after all, a really low priority bug.

Aki told me that he had a manager who said “you don’t learn til you break something”.  Well I broke everything try-related today and here’s hoping that I have learned something because the stress of this whole day is not something I want to experience often. It’s that feeling you get when you realize you’ve started something that you can’t back out of and there’s no way to go but forward, even though everything in front of you now appears hopeless and messy.

So here’s some lessons to take away:

1. Staging is not to be underestimated even for just renaming things that are already working
2. Taking the time to search with grep/mxr and find the terms you are replacing before starting the upgrade in production will help find wiring you might have overlooked in your preparations
3. Prepare more thoroughly and have a clear idea of the env. you started in and what it will take to have that env. back when you’re done. Leaving dangly bits is not ideal.

Happy Friday.
(and many thanks to Aki)

Dear BBC,

Today on the front page of your technology section you said that downloads for Firefox 4 have been lower than they were for Firefox 3 and that:

The lower figure may be explained by the widespread availability of pre-release versions of Firefox 4 in the months ahead of its launch.

First of all, you forgot that we’ve had 3.5 and 3.6 between those two and so we now have users spread out a bit across versions. Second, here’s an overview of how we’re organizing the release of Firefox 4:

• We put out the RC and picked up users from outside of our usual beta testing pool in order to give our final candidate some solid tire kicking
• Firefox 4 went live but our users on 3.5 and 3.6 are not offered the update automatically yet, they must “Check for updates” in order to be asked if they want to upgrade to Firefox 4 
• Once we have more coverage of the new release for a couple of weeks and are even more confident that we’ve got an amazing browser out there we will turn on the Major Update notification which will offer our 400+ million users the chance to come on up and experience the next level of the web

According to W3C school’s stats(which are measured by visits to their site) the browser distribution of their visitors looks like this:

2011	Internet Explorer	Firefox	Chrome	Safari	Opera
February	26.5 %	42.4%	24.1%	4.1%	2.5%
January	26.6 %	42.8%	23.8%	4.0%	2.5%

2011	Total	FF 4.0	FF 3.6	FF 3.5	FF 3.0	Other
February	42.4 %	1.9 %	35.8 %	2.9 %	1.5 %	0.3 %
January	42.8 %	1.5 %	36.1 %	3.1 %	1.7 %	0.4 %

What this says to me is that our more than 8 million downloads since yesterday morning PDT only shows us how many people are paying attention to the fact that Firefox 4 has launched and is available for download. It’s not representative of our 400+ million active daily user base (the people who just use the browser but perhaps don’t read your blog or mine).  These people will soon learn about Firefox 4 through their browser’s update notification window. We’ll be seeing a spike in downloads in a couple of weeks and I hope you’ll report on that.

Last night was the first meeting of The Factory and Mozilla. The partnership if a result of work between Mozilla Drumbeat and Web Made Movies. Ben Moskowitz and Brett Gaylor invited myself and Atul Varma to what is to be the first of three sessions helping teens learn about the budding open web technologies that can be integrated with video.

About 13 kids streamed into a computer lab at 4:30pm and we began with some introductions detailing who we were and why this stuff is interesting. The group were very engaged and eager to dive right in with whatever we had for them. So we started with Atul’s Web X-Ray Goggles so the students could see what exactly the web was made of. The idea was to just grab parts of whatever websites you liked and change them right on the page so you could see how easy it is to “hack” the web. Some of the teens went even further and started building their own pages with Etherpad by grabbing snippets of code from sites. About 4 kids said they had done a View Source on a page prior to this class and 30 minutes into this workshop they were all doing it like pros.

Once they had a chance to remix a web page we moved on to the next exercise which was to select 4 popular sites of their choosing (Etherpad democracy!) and those sites were printed out on paper, the teens were split into 3 teams, and each team did paper prototyping of a new site using elements from the originals. I was very impressed with how the students took the idea and ran with it. Each team worked fast (they had 7 minutes) and no one was hanging back keeping their opinions to themselves. The teams produced 3 new site mock-ups that each had a very simple look, with a video as the prominent element on the page but they also took time to add site navigation and social media integration by putting Facebook and Twitter in the sidebar or footer.

In the last 30 minutes of our time Brett and Ben demonstrated Butter also explaining how very “caveman” the technology is right now. With only a glance at the interface and a basic explanation of how it’s wired up the teens jumped right in with suggestions and ideas about what they would like to be able to do:
* Hide popcorn elements when nothing is showing in them
* Be able to zoom in or out on Google maps while the video is playing
* Click on something in the video (example: coffee mug) and have it trigger an event

Ben made a really great point about how it’s also important to look at something like Butter and think “How can you go beyond the interface?”. How do you make your story more interesting from the beginning knowing you can use this tool throughout instead of just tacking on events and additional information to a completed video that is done in a standard format?

We’ll be working with them again tonight, with chunks of a film they made last summer called “The List”. More updates and more potential bugs and feature requests coming soon.

Also, if anyone is planning to teach a class like this you might want a few things in a “kit”:
* Portable printer (and paper)
* Scissors
* Tape or glue
* Handout with links to the tools/sites

Just to save some time 🙂