How many ISP:s are you using?

This one requires some basic programming skills but python is easy to learn so lets get going.
GeoIP can produce more than just countries, but the support for ASN is a little bit tricky since the normal binary GeoIP module is quite old even in Ubuntu 12.04.
That is why we will use the pythonbased one that is much slower instead, this matters if you want to map thousands of ipadresses.
First install install the latest module by issuing:
sudo pip install pygeoip
Download http://download.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz from the maxmind homepage, extract it and put it among the others.

Start up the python interpeter.
import pygeoip
_ipaddr="198.101.241.245"
gia = pygeoip.GeoIP('/usr/share/GeoIP/GeoIPASNum.dat')
_asn = gia.org_by_addr(_ipaddr)
print _asn

Ok now parse your favourite logs for ipadresses an map how many ISP:s which users are using....
Which combinations seems to unlikely to be true?
Which ISP do you find to be least trusworthy?

Feeding the ELSA database with sphinx

There are diffrent tools for munging syslog data and feeding some database to make bigdata searches possible.
I have written som crappy ones myself which can store less than 100 events per second before they drop dead.

I really like SPLUNK, but a hefty pricetag is attached and 500Mb a day is the maximum if you want to testdrive it.
It is easy to fill that up in an enterprise environment. But I still admit if SPLUNK was for free, this blog would probably not exist.
Time to test the open source logparsing SPLUNK lookalile ELSA.
Enter your syslogserver and fire it up:

wget "http://enterprise-log-search-and-archive.googlecode.com/svn/trun/elsa/contrib/install.sh" && sudo sh install.sh node

The install will take quite a while, and if you don't populate /etc/elsa_vars.sh with your database settings(or at least your password) it will fail miserably. When you are done, a file called /etc/elsa_node.conf should exist.

The solution to the problem i sphinx that allows you to do fulltextsearch darn fast.
http://sphinxsearch.com/

You can still search the mysqldatabase like you always have done:

mysql -u root -p

use syslog_data;
select msg from syslogs_archive_1 where msg LIKE  "%rsyslog%" ;

But it will take plenty of time.

Using rsyslog to monitor files

This feature is quite new to me but it is really neat for monitoring applications that does not write to syslog.
But you still want to pass on the data in "realtime" without to much fuzz.

Create a file under /etc/rsyslog.d/ that does the magic.

cd /usr/local/bro/logs/
logs=$(find . | awk -F\/ '{print $3}'|awk -F\. '{print $1}'|sort -u)

echo $ModLoad imfile # > bro.conf
for i in $logs
do
echo '$InputFileName /usr/local/bro/logs/current/'$i >> bro.conf
echo '$InputFileTag bro_'$i >> bro.conf
echo '$InputFileStateFile stat-bro_'$i >> bro.conf
echo '$InputFileSeverity info' >> bro.conf
echo '$InputFileFacility local7' >> bro.conf
echo '$InputRunFileMonitor' >> bro.conf
done

echo 'local7.* @your_syslog_server_to_use_for_analyzing' >> bro.conf

An example file

$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor
$InputFileName /usr/local/bro/logs/current/ssl.log
$InputFileTag bro_ssl:
$InputFileStateFile stat-bro_ssl
$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor
$InputFileName /usr/local/bro/logs/current/smtp.log
$InputFileTag bro_smtp:
$InputFileStateFile stat-bro_smtp
$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor
$InputFileName /usr/local/bro/logs/current/smtp_entities.log
$InputFileTag bro_smtp_entities:
$InputFileStateFile stat-bro_smtp_entities
$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor
$InputFileName /usr/local/bro/logs/current/notice.log
$InputFileTag bro_notice:
$InputFileStateFile stat-bro_notice
$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor
$InputFileName /usr/local/bro/logs/current/ssh.log
$InputFileTag bro_ssh:
$InputFileStateFile stat-bro_ssh
$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor
$InputFileName /usr/local/bro/logs/current/ftp.log
$InputFileTag bro_ftp:
$InputFileStateFile stat-bro_ftp
$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor
local7.* @127.0.0.1

 

The script assumes that your bro installation has been up and running for a couple of days to most types of logfiles. Edit your syslogserver and copy the file to /etc/rsyslog.d/ and restart rsyslog. Depending on where your suslogserver is, you should se the data flow, if not start digging with tcpdump.

tcpdump -n -r filename udp and dst port 514

bro basic setup

bro-ids is a nice tool to keep track of what is passing by on the wire.
Lets install it on an modern Ubuntu.

sudo apt-get install swig python-dev libmagic-dev libpcre3-dev libssl-dev cmake git-core subversion ruby-dev libgeoip-dev flex bison
# download bro http://www.bro-ids.org/downloads/release/bro-2.1.tar.gz
cd bro-2.1
./configure
make
make install
/usr/local/bro/bin/broctl > install > start

If you have plenty of data passing by and the system is loaded above 50% you should consider PF_RING and recomplile libpcap and bro.
You might find some performance improvements with ethtool tweaking, but first make sura that you have a decent intel nic or similar server equipment.
If this is still nog good enough, star removing checks from local.bro
Next item is about passing the data on for analysis...