Chasing rainbows is easier

I was so thrilled to discover Splunk that I installed it on one of the buildbot masters – qm-rhel02 – without realizing that in fact, Splunk starts to quickly eat up disk space and hogs memory usage. Yesterday afternoon some Talos boxes started to go down because of this, and once I stopped the Splunk server everything started to right itself.

Lessons learned:

     Do not play with the buildbot master. 
     Do not look directly at the buildbot master. 
     Do not taunt the buildbot master.

So today’s tasks include getting access to the samba share that was set up, creating a cron job that will rsync the buildbot master logs to said share and then finding a safe place to set up Splunk again.

We really need to have a way to look at data from the buildbot master over a long period of time – otherwise filing bugs on these intermittent failures is just a shot in the dark. Take yesterday for example. qm-win2k3-pgo01 is being “unreliable” and had the same errors in refest for two consecutive builds. I file a bug, and the response is “grab me a copy of the offending objdir so we can poke at it”. Wouldn’t you know that the very next build does not have the same error output – this time it has mochitest issues that are seemingly unrelated. This morning I check again and it’s had a compile failure, an exception (the most hideous purple) and then a completely green run.

Intermittent failures == needle in a haystack

sha1sum on Mac OSX

Getting ready to assemble the Firefox 3 CD and came upon the glitch that Mac OS X doesn’t provide sha1sum tools. Quick Google search turned up a great comment on this blog post which suggests using openssl and by putting alias sha1sum="openssl dgst -sha1" in my .profile I can now do sha1sum $app_name.iso to my heart’s content.

The Firefox 3 CD will now have all the supported locales on it, which is a step up from the Firefox 2 CD. Look for it in a Mozilla Store near you.

Splunk – Where IT’s at

So, you know that script I was working on to parse error logs? Well, it turns out that there is already an amazing, free, graphical program that does the work for me. Excellent.

It’s called Splunk, John O’Duinn mentioned it in passing last night and today I got it running on the unittest-staging build master in about 2 seconds flat.

Installation (on linux) is a breeze, simply:
wget 'http://www.splunk.com/index.php/download_track?file=3.2.6/linux/splunk-3.2.6-38259-Linux-i686.tgz∾=&wget;=true&name;=wget&typed;=releases'

Then unpack and bin/splunk start --accept-license

It creates and starts a server on port 8000 which you can then access to use all the graphical features as well as an admin dashboard. The site is clean and well laid out. I look forward to going deeper into what this app can offer us.

I also installed it locally on my Mac and the steps are exactly the same. After starting up the Splunk server, simply point it to the directory where your log files live and Bob’s your uncle.

See how it beautifully transforms log files into searchable fields with a graphical display? This will be extremely useful as we shift our machines around, play with the difference between VMs and hardware as well as put all our unittest machines up to Buildbot 0.7.7.

With only a few minutes on the dashboard I found it easy to navigate, add several input streams from the various build slaves that run on unittest-staging and also noticed that you can create and save specific search requests.

Can’t wait to see how much this helps others, now off to install it on the other masters.