sha1sum on Mac OSX

Getting ready to assemble the Firefox 3 CD and came upon the glitch that Mac OS X doesn't provide sha1sum tools. Quick Google search turned up a great comment on this blog post which suggests using openssl and by putting alias sha1sum="openssl dgst -sha1" in my .profile I can now do sha1sum $app_name.iso to my heart's content.

The Firefox 3 CD will now have all the supported locales on it, which is a step up from the Firefox 2 CD. Look for it in a Mozilla Store near you.

Splunk - Where IT's at

So, you know that script I was working on to parse error logs? Well, it turns out that there is already an amazing, free, graphical program that does the work for me. Excellent.

It's called Splunk, John O'Duinn mentioned it in passing last night and today I got it running on the unittest-staging build master in about 2 seconds flat.

Installation (on linux) is a breeze, simply:
wget 'http://www.splunk.com/index.php/download_track?file=3.2.6/linux/splunk-3.2.6-38259-Linux-i686.tgz&ac=&wget=true&name=wget&typed=releases'


Then unpack and bin/splunk start --accept-license

It creates and starts a server on port 8000 which you can then access to use all the graphical features as well as an admin dashboard. The site is clean and well laid out. I look forward to going deeper into what this app can offer us.




I also installed it locally on my Mac and the steps are exactly the same. After starting up the Splunk server, simply point it to the directory where your log files live and Bob's your uncle.



See how it beautifully transforms log files into searchable fields with a graphical display? This will be extremely useful as we shift our machines around, play with the difference between VMs and hardware as well as put all our unittest machines up to Buildbot 0.7.7.

With only a few minutes on the dashboard I found it easy to navigate, add several input streams from the various build slaves that run on unittest-staging and also noticed that you can create and save specific search requests.

Can't wait to see how much this helps others, now off to install it on the other masters.

Robot War - Firefox 3 vs The Other Browser

Just in time for Download Day, a little video produced and directed by Marcia Knous and edited by me. I'm posting it on several sites youTube, Flickr and blip.tv - please link to it, digg it, favourite it, whathaveyou - just pass it along.

What did I do today?

Interning at Mozilla has so far provided me with many opportunities to learn (and re-learn) some of the finer points of the command line interface.

Today I have spent most of my time working on some scripts (both shell and python) that will assist in parsing error logs from the unittest builbot masters.

Here's how I would like it to work:

• The script will live in the master directory and when run followed by the 1...n slave directory names it will create a folder with files containing all lines which match particular error messages that are in the stdio log files. Each file is specific to the pattern searched (eg: reftest, browser, check)


• A python script then reads through each of the files and breaks the lines of error statements up into sqlite insert statements and throws them into a sqlite db


• The DB will have a front end which will allow for easy searching and sorting of the data to see if there is a particular test that fails more frequently than others, on which platforms, etc...

This is an interesting side project for me as we wait for release to start bringing the unittest boxes up to Buildbot 0.7.7. I'm enjoying writing python (esp. compared to shell scripting).

Finally - many thanks to Chris Tyler for today's little success. I needed to put the time and date of the log file into the pattern matches so that the database can be more useful and I was having difficulty figuring it out, Chris is the expert at the one line solution which was ( >> means continued on next line):

for file in *-log*    # traverse all files in $DIRNAME
  do 
    grep -HnA 2 "$string1" $file >>
  | sed "s~^~|$(date -r $file '+$%D|%T|')~" >>
    > "$OUTPUT_FILE.reftest"
  done

What's new in Firefox 3? Check out this demo!

A quick (< 4 minutes) overview of some of the new features in the soon-to-be-released Firefox 3. Check it out, then head over to http://www.spreadfirefox.com/worldrecord/ and sign up to be notified when the new version comes out!

read more | digg story

Parsing for errors in Buildbot log files

The other day two of our Moz2 unittest buildbots - one Linux and one Windows - were both failing tests intermittently. We have all these logs but no way to parse the data to look for patterns and try to figure out what is going on. In an attempt to scratch the surface of this issue, I was tasked to look at the error messages and put together something for bug 435064.

It took a little while to come up with the right approach but in the end I had some grep statements that did the trick. Thinking that this will come up again, I packaged them into a little shell script to do the dirty work for me next time:

#!/bin/bash
# parseError.sh
# simple script for gathering up errors in log files

if [ -z "$1" ] 
then 
  echo "Usage: '$0' [directory]"
  exit 1
fi

string1="UNEXPECTED FAIL"
string2="ERROR FAIL"
string3="command timed out"
string4="FAIL"

DIRNAME=$1
OUTPUT_FILE="../output"

echo "Looking for UNEXPECTED, ERROR and command time outs..."
cd $DIRNAME
for file in *-log*    # traverse all log files in $DIRNAME
do 
  grep -Hn "$string1" $file >> $OUTPUT_FILE
  grep -Hn "$string2" $file >> $OUTPUT_FILE
  grep -Hn "$string3" $file >> $OUTPUT_FILE
done

# these two searches include 5 lines of context
echo "Looking for Check and Browser Fails...."
grep -HnC 5 $string4 *-log-check* >> $OUTPUT_FILE 
grep -HnC 5 $string4 *-log-browser* >> $OUTPUT_FILE

echo "Sorting......"
sort -n $OUTPUT_FILE | sed /--/d > "$OUTPUT_FILE.sorted"

echo "Search complete."
exit 0

Thanks to dchen and humph for helping with the finer points of writing shell scripts.