Logtop is an awesome, albeit a little quirky, real-time log analysis tool developed by Julien Palard. You should use logtop when time is of the essence. When you cannot wait for your cron job to run to analyze log files from last night. When you need to know if you’re being hacked now – not yesterday.

Here’s how I installed it on CentOS 6:

cd
yum -y install git ncurses-devel uthash uthash-devel
git clone https://github.com/JulienPalard/logtop.git
cd logtop
make
make install

And here’s a very basic example of how to use it. In this case I am counting the number of hits by individual IPs against by httpd server. This can be useful if you need to see who needs to be firewalled right now.

tail -f /var/log/httpd/access_log | awk {'print $1; fflush();'} | logtop

Note the fflush syntax allowing awk to flush its buffers in real time and thus be usable by toplog. The end result is simple: toplog counts frequency and number of occurrences of whatever you’re piping into it.

Another example shows how random /dev/random is:

tr -dc '0-9' </dev/urandom | fold -w 1 | head -n 10MB | logtop

Here’s an example showing DHCP server activity by call type. Note the --line-buffered option for the grep command.

tail -f /var/log/boot.log | grep -oP --line-buffered "DHCP(DISCOVER|OFFER|REQUEST|ACK)" | logtop

The thing to understand about logtop is that the tool is designed for real-time analysis. You can’t use it to produce a frequency report of historical log data, even if this data contains timestamps. So, real-time analysis – logtop. Historical data analysis – something else (and there’re plenty of choices here).

Things get a little tricky if you need to run logtop against real-time data for a defined period of time. Here’s an example of running the command for one minute:

timeout 61 tail -f /var/log/httpd/access_log | timeout 60 awk {'print $1; fflush();'} | logtop -q

Here’s a slightly more advanced example that will use iptables to block IPs that accessed your httpd at a rate of greater than 0.9 times per second over the past minute, Something like this can be useful to fend off a DOS attack or an overly aggressive Web crawler.

timeout 61 tail -f /var/log/httpd/access_log | timeout 60 awk {'print $1; fflush();'} | logtop -q | grep -E "([0-9]{1,3}\.){3}([0-9]{1,3})" | while read line
do
	c=$(echo $line | awk '{print $2}')
	f=$(echo $line | awk '{print $3}')
	i=$(echo $line | awk '{print $4}')
	if (( $(echo "$f > 0.9" | bc -l) ))
	then
		iptables -A INPUT -s $i -j DROP
	fi
done