Thursday, June 19, 2008

Splunk - Where IT's at


So, you know that script I was working on to parse error logs? Well, it turns out that there is already an amazing, free, graphical program that does the work for me. Excellent.

It's called Splunk, John O'Duinn mentioned it in passing last night and today I got it running on the unittest-staging build master in about 2 seconds flat.

Installation (on linux) is a breeze, simply:
wget 'http://www.splunk.com/index.php/download_track?file=3.2.6/linux/splunk-3.2.6-38259-Linux-i686.tgz&ac=&wget=true&name=wget&typed=releases'


Then unpack and bin/splunk start --accept-license

It creates and starts a server on port 8000 which you can then access to use all the graphical features as well as an admin dashboard. The site is clean and well laid out. I look forward to going deeper into what this app can offer us.




I also installed it locally on my Mac and the steps are exactly the same. After starting up the Splunk server, simply point it to the directory where your log files live and Bob's your uncle.



See how it beautifully transforms log files into searchable fields with a graphical display? This will be extremely useful as we shift our machines around, play with the difference between VMs and hardware as well as put all our unittest machines up to Buildbot 0.7.7.

With only a few minutes on the dashboard I found it easy to navigate, add several input streams from the various build slaves that run on unittest-staging and also noticed that you can create and save specific search requests.

Can't wait to see how much this helps others, now off to install it on the other masters.


3 comments:

pd said...

not a bad tool if you dont mind supporting proprietary code like Flash, which Splunk appears to heavily rely on

Bob said...

Uhm

It uses Ajax, not flash. And Splunk is not open Source, even thought the basic version is free. It only allows you to index 300 MB of data per day if you don't pay, which is less than useful for me.

Spanky said...

Actually, it's both Flash and AJAX. The reporting charts are done in Flash. The data entry & log searching is AJAX though. You can check it by right clicking on the graphs on the Admin Dashboard -- you'll get the Adobe Flash menu. Also, it's 500MB of data in the free license, not 300MB of data - http://www.splunk.com/article/2018