Interning at Mozilla has so far provided me with many opportunities to learn (and re-learn) some of the finer points of the command line interface.
Today I have spent most of my time working on some scripts (both shell and python) that will assist in parsing error logs from the unittest builbot masters.
Here’s how I would like it to work:
- The script will live in the master directory and when run followed by the 1…n slave directory names it will create a folder with files containing all lines which match particular error messages that are in the stdio log files. Each file is specific to the pattern searched (eg: reftest, browser, check)
- A python script then reads through each of the files and breaks the lines of error statements up into sqlite insert statements and throws them into a sqlite db
- The DB will have a front end which will allow for easy searching and sorting of the data to see if there is a particular test that fails more frequently than others, on which platforms, etc…
This is an interesting side project for me as we wait for release to start bringing the unittest boxes up to Buildbot 0.7.7. I’m enjoying writing python (esp. compared to shell scripting).
Finally – many thanks to Chris Tyler for today’s little success. I needed to put the time and date of the log file into the pattern matches so that the database can be more useful and I was having difficulty figuring it out, Chris is the expert at the one line solution which was ( >> means continued on next line):
for file in *-log* # traverse all files in $DIRNAME
do
grep -HnA 2 "$string1" $file >>
| sed "s~^~|$(date -r $file '+$%D|%T|')~" >>
> "$OUTPUT_FILE.reftest"
done
