So my patch was backed out because it broke make buildsymbols on the Mac and Linux platforms. From looking closer at the error message and the patch, I deduce that it’s GetVCSFilename containing the offending line – a return file where a tuple (two return values in python) should be returning, so I change it to return (file, None) and after testing the patch on a clean Mac build I am presented with the successful making of buildsymbols.

Now enter evil Linux (cue evil music)- the one platform I have yet to build on in any serious fashion. I should test this patch on Linux to make sure that it works there too. Here’s the bullet form of what happened (as best I can remember at this point in head bashing on keyboard):

1. I ssh to ‘Liberia’, one of the Quad’s running Fedora in the ORI lab
2. Check out, apply patch and build a debug build on it – fast! It takes ~3 mins to checkout and ~13 to build
3. Run make buildsymbols and I get “make: nothing to do for ‘buildsymbols'” (thanks.)
4. Clobber build, tweak .mozconfig adding –enable-application=browser
5. Build – run make buildsymbols – still nothing to do for buildsymbols
6. After attempting to bike home, getting a flat tire from a massive pothole on Keele St and having to bus it I ssh into Liberia again and start from scratch. At this point both Armen and Dave start builds too, all of us using –enable-debug and other assorted .mozconfig settings
7. No one can make buildsymbols. Without the patch there are errors and with the patch – same errors.
8. In objdir/config/autoconf.mk I check and see that MOZ_CRASHREPORTER = 1 so make should be recognizing this.
9. I try a clean checkout and can’t even build because of errors with dump_symbols.cc

Swimming in a sea of red herrings is not fun. Tomorrow’s another day and another stab at getting this going. At this point I’m convinced that there’s something wrong with locally building and not in fact with the patch since it does not touches dump_symbols.cc and since today’s Linux symbols are up and fine (ted checked) which means that make buildsymbols does in fact work somewhere…just not for me (or Dave).

Continuing along with the education of future Build and Release engineers, Armen and I were introduced to automation with Robert Helmer. Rob’s leaving Mozilla in a couple of weeks and it’s too bad we won’t get to work with him this summer. He’s really got this all figured out and most of his presentation was way over my head. I said to Ben Hearsum later that it feels like once I’m actually touching the build system it will probably make more sense, but just talking about it? That’s a bit confusing. In any case, Rob presented for about an hour on the automation process from past to present and then there was a lively discussion about what’s coming next.

The next day Ben Hearsum talked about Try server and showed us the current set up. I’m really excited to learn more about Try server, and also to having permissions to use it myself. Considering that my recent patch had to be backed out for breaking buildsymbols on the Mac and Linux platforms the Try server could really help me out. I’ll be talking with Ben about how I can get in on it. I have an LDAP account now…

Following Ben, Nick Thomas showed us the flow of releases from tinderbox to virus scanning to their final resting place on the ftp servers. In his diagram you can see pre Sept 2007, how it is currently and then on the right – what we’re aiming for.

The last presentation of the day was Rob Campbell with his presentation on unit tests. Again we were treated to a thorough explanation of how things were done, how they’re being done now and where things are heading.

The most exciting part of the week was realising that there is still a lot of room in Mozilla to have an impact on how things are done. On Wednesday morning we went for breakfast and Armen and I were lucky enough to be sitting next to Mike Schroepfer – VP of Engineering for Mozilla. He shared a lot of information with us and also pointed out how every single person at the table (the Build, Release and Automation team) has contributed significantly. That Mozilla has been able to grow as quickly as it has without falling down is a testament to all their hard work (and that of many others) and I am inspired to be part of that.

Thanks to Dave and to John for helping organize this opportunity. I think that my transition to interning this summer will be much smoother as I have met many of the people with whom I’ll be working, plus the overviews of each area of Build were invaluable.

Armen and I are in California attending Build and Release team meetings this week. Over the next two days we’ll be introduced to the many facets of the Build and Release workflow.

Todays first session was about Talos with Alice.

Here is the diagram of Talos (copied from the diagram Alice drew – yes Talos is a robot):

Alice walked us through how Talos gets its information from browser builds by having buildbot read for new builds from quickparse which is a text file. Buildbot has a script that knows what to look for in order to find new builds and there is a 5 minute delay before Talos is deployed because the information can get into quickparse before the build is finished and so therefore does not technically exist.

Currently there are 30 Production and 20 Stage Talos machines running, this past December there was only 1 Production machine and the stage machines.

This huge increase of Talos machines has led to an insane amount of data being gathered and a database which is in serious need of some help.

After Alice’s presentation we all tried a standalone Talos so we could see the tests at work.

Anyone can try them, just follow these directions. If you are using a recent nightly you might need to add security.fileuri.strict_origin_policy : false to your sample.config file preferences because of new security features. Also, you can comment out the tjss tests because those are kind of boring – the fun test is the svg since you’ll see a lot of graphics tests running on your browser. This standalone runs on a new profile so it’s okay if you already have Firefox running when you run this script.

So the information that’s generated is good for recognizing regression like in this bug, where if you look at the graph you can see how the build was chugging along, something got checked in that affected performance and then it was backed out and the performance went back to normal.

Pic here, see bug #425941 for more info:

More information on Talos Machines.