Real-Time Log Stats With Logtop

Originally published December 30, 2016 @ 11:01 pm

Logtop is an awesome, albeit a little quirky, real-time log analysis tool developed by Julien Palard. You should use logtop when time is of the essence. When you cannot wait for your cron job to run to analyze log files from last night. When you need to know if you’re being hacked now – not yesterday.

Here’s how I installed it on CentOS 6:

cd
yum -y install git ncurses-devel uthash uthash-devel
git clone https://github.com/JulienPalard/logtop.git
cd logtop
make
make install

And here’s a very basic example of how to use it. In this case I am counting the number of hits by individual IPs against by httpd server. This can be useful if you need to see who needs to be firewalled right now.

tail -f /var/log/httpd/access_log | awk {'print $1; fflush();'} | logtop

Note the fflush syntax allowing awk to flush its buffers in real time and thus be usable by toplog. The end result is simple: toplog counts frequency and number of occurrences of whatever you’re piping into it.

Here’s a slightly more advanced variation of the above example. It will find and watch multiple access_log files and show the hit frequency for each IP that is not on your private network. If you think someone out there is killing your server with requests, this is how you find them.

tail -f $(find /var/log/httpd -maxdepth 2 -mindepth 2 -type f -name "access_log") | \
awk {'print $1; fflush();'} | grep -oP --line-buffered "([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})" | \
grep -vP --line-buffered "(^127\.)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)" | \
logtop

Seeing the IPs is nice, but it would be useful to see a bit more information about each IP. If you install geoiplookup, you can modify the command above to make use of it:

tail -f $(find /var/log/httpd -maxdepth 2 -mindepth 2 -type f -name "access_log") | \
awk {'print $1; fflush();'} | grep -oP --line-buffered "([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})" | \
grep -vP --line-buffered "(^127\.)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)" |
while read i; do echo -ne "${i}; "; geoiplookup ${i} | grep -oP --line-buffered "(?<=(ion|v [0-9]): ).*(?=$)" | \
cut -d "," -f 1-5 | paste -sd " " - | sed 's/ N\/A,//g'; done | logtop

The output might look something like this:

109 lines, 0.49 lines/s
RANK    CNT   LINE/S LINE
   1     26     0.12 182.73.212.174; IN, India IN, N/A AS9498 BHARTI Airtel Ltd.
   2      7     0.03 185.12.108.134; TR, Turkey TR, N/A AS62054 YNT Bilisim Danismanlik Iletisim Insaat Sanayai ve Ticaret Ltd. Sirketi
   3      6     0.03 203.133.170.51; KR, Korea, Republic of KR, N/A AS9764 Kakao Corp
   4      4     0.02 108.59.8.70; US, United States US, DE, Delaware, N/A AS30633 Leaseweb USA, Inc.
   5      4     0.02 5.255.250.108; US, United States US, MA, Massachusetts, Newburyport, 01950 AS13238 YANDEX LLC
   6      3     0.01 178.154.244.156; RU, Russian Federation RU, N/A AS13238 YANDEX LLC
...

Another example shows how random /dev/random is:

tr -dc '0-9' </dev/urandom | fold -w 1 | head -n 10MB | logtop

Here’s an example showing DHCP server activity by call type. Note the --line-buffered option for the grep command.

tail -f /var/log/boot.log | grep -oP --line-buffered "DHCP(DISCOVER|OFFER|REQUEST|ACK)" | logtop

The thing to understand about logtop is that the tool is designed for real-time analysis. You can’t use it to produce a frequency report of historical log data, even if this data contains timestamps. So, real-time analysis – logtop. Historical data analysis – something else (and there’re plenty of choices here).

Things get a little tricky if you need to run logtop against real-time data for a defined period of time. Here’s an example of running the command for one minute:

timeout 61 tail -f /var/log/httpd/access_log | timeout 60 awk {'print $1; fflush();'} | logtop -q

Here’s a slightly more advanced example that will use iptables to block IPs that accessed your httpd at a rate of greater than 0.9 times per second over the past minute, Something like this can be useful to fend off a DOS attack or an overly aggressive Web crawler.

timeout 61 tail -f /var/log/httpd/access_log | timeout 60 awk {'print $1; fflush();'} | logtop -q | grep -E "([0-9]{1,3}\.){3}([0-9]{1,3})" | while read line
do
	c=$(echo $line | awk '{print $2}')
	f=$(echo $line | awk '{print $3}')
	i=$(echo $line | awk '{print $4}')
	if (( $(echo "$f > 0.9" | bc -l) ))
	then
		iptables -A INPUT -s $i -j DROP
	fi
done