Interning at Mozilla has so far provided me with many opportunities to learn (and re-learn) some of the finer points of the command line interface.

Today I have spent most of my time working on some scripts (both shell and python) that will assist in parsing error logs from the unittest builbot masters.

Here’s how I would like it to work:

• The script will live in the master directory and when run followed by the 1…n slave directory names it will create a folder with files containing all lines which match particular error messages that are in the stdio log files. Each file is specific to the pattern searched (eg: reftest, browser, check)
• A python script then reads through each of the files and breaks the lines of error statements up into sqlite insert statements and throws them into a sqlite db
• The DB will have a front end which will allow for easy searching and sorting of the data to see if there is a particular test that fails more frequently than others, on which platforms, etc…

This is an interesting side project for me as we wait for release to start bringing the unittest boxes up to Buildbot 0.7.7. I’m enjoying writing python (esp. compared to shell scripting).

Finally – many thanks to Chris Tyler for today’s little success. I needed to put the time and date of the log file into the pattern matches so that the database can be more useful and I was having difficulty figuring it out, Chris is the expert at the one line solution which was ( >> means continued on next line):

for file in *-log*    # traverse all files in $DIRNAME
  do 
    grep -HnA 2 "$string1" $file >>
  | sed "s~^~|$(date -r $file '+$%D|%T|')~" >>
    > "$OUTPUT_FILE.reftest"
  done

A quick (< 4 minutes) overview of some of the new features in the soon-to-be-released Firefox 3. Check it out, then head over to http://www.spreadfirefox.com/worldrecord/ and sign up to be notified when the new version comes out! read more | digg story

The other day two of our Moz2 unittest buildbots – one Linux and one Windows – were both failing tests intermittently. We have all these logs but no way to parse the data to look for patterns and try to figure out what is going on. In an attempt to scratch the surface of this issue, I was tasked to look at the error messages and put together something for bug 435064.

It took a little while to come up with the right approach but in the end I had some grep statements that did the trick. Thinking that this will come up again, I packaged them into a little shell script to do the dirty work for me next time:

#!/bin/bash
# parseError.sh
# simple script for gathering up errors in log files

if [ -z "$1" ] 
then 
  echo "Usage: '$0' [directory]"
  exit 1
fi

string1="UNEXPECTED FAIL"
string2="ERROR FAIL"
string3="command timed out"
string4="FAIL"

DIRNAME=$1
OUTPUT_FILE="../output"

echo "Looking for UNEXPECTED, ERROR and command time outs..."
cd $DIRNAME
for file in *-log*    # traverse all log files in $DIRNAME
do 
  grep -Hn "$string1" $file >> $OUTPUT_FILE
  grep -Hn "$string2" $file >> $OUTPUT_FILE
  grep -Hn "$string3" $file >> $OUTPUT_FILE
done

# these two searches include 5 lines of context
echo "Looking for Check and Browser Fails...."
grep -HnC 5 $string4 *-log-check* >> $OUTPUT_FILE 
grep -HnC 5 $string4 *-log-browser* >> $OUTPUT_FILE

echo "Sorting......"
sort -n $OUTPUT_FILE | sed /--/d > "$OUTPUT_FILE.sorted"

echo "Search complete."
exit 0

Thanks to dchen and humph for helping with the finer points of writing shell scripts.