Feeding the ELSA database with sphinx

There are diffrent tools for munging syslog data and feeding some database to make bigdata searches possible.
I have written som crappy ones myself which can store less than 100 events per second before they drop dead.

I really like SPLUNK, but a hefty pricetag is attached and 500Mb a day is the maximum if you want to testdrive it.
It is easy to fill that up in an enterprise environment. But I still admit if SPLUNK was for free, this blog would probably not exist.
Time to test the open source logparsing SPLUNK lookalile ELSA.
Enter your syslogserver and fire it up:

wget "http://enterprise-log-search-and-archive.googlecode.com/svn/trun/elsa/contrib/install.sh" && sudo sh install.sh node

The install will take quite a while, and if you don't populate /etc/elsa_vars.sh with your database settings(or at least your password) it will fail miserably. When you are done, a file called /etc/elsa_node.conf should exist.

The solution to the problem i sphinx that allows you to do fulltextsearch darn fast.
http://sphinxsearch.com/

You can still search the mysqldatabase like you always have done:

mysql -u root -p

use syslog_data;
select msg from syslogs_archive_1 where msg LIKE  "%rsyslog%" ;

But it will take plenty of time.