Originally published September 2, 2019 @ 6:09 pm
Most commonly iptables
is used to allow, block, or redirect connections. However, it also has a logging feature that can be very useful for network traffic analysis and system security.
In the example below I will get a list of network services listening on my server and create iptables
inbound connection logging rules. And then I’ll use some basic shell scripting to analyze collected data.
Here’s how you can get a list of listening network ports and service names:
for i in $(lsof -i -P -n | grep -oP "(?<=\*:)[0-9]{2,}(?= \(LISTEN)" | sort -nu); do lsof -i :${i} | grep -v COMMAND | awk -v i=$i '{print $1,i}' | sort -u; done | column -t sshd 22 httpd 80 rpcbind 111 httpd 443 xinetd 873 rpc.rquot 875 httpd 1701 mysqld 3306 splunkd 8000 splunkd 8089
Let’s say we want iptables
to log all inbound traffic for port 443
. Here’s the rule for this:
iptables -I INPUT -p tcp --dport 443 -m state --state NEW -j LOG --log-prefix "HTTPS-443 inbound: "
Save the rules and reload iptables
service and in /var/log/messages
you should now be able to see entries like these:
Aug 30 16:28:09 ncc1701 kernel: [6205722.091794] HTTPS-443 inbound: IN=bond0 OUT= MAC=a0:48:1c:b8:17:b8:5c:f4:ab:67:c4:4b:08:00 SRC=192.0.101.11 DST=192.168.122.137 LEN=52 TOS=0x00 PREC=0x00 TTL=56 ID=5926 DF PROTO=TCP SPT=14784 DPT=443 WINDOW=29200 RES=0x00 SYN URGP=0 Aug 30 16:28:09 ncc1701 kernel: [6205722.092482] HTTPS-443 inbound: IN=bond0 OUT= MAC=a0:48:1c:b8:17:b8:5c:f4:ab:67:c4:4b:08:00 SRC=192.0.102.70 DST=192.168.122.137 LEN=52 TOS=0x00 PREC=0x00 TTL=54 ID=18456 DF PROTO=TCP SPT=56268 DPT=443 WINDOW=29200 RES=0x00 SYN URGP=0
Seeing the source IP is useful but, perhaps, seeing the source hostname, country of origin, and organization would be even better. Here’s how this can be accomplished:
grep 'HTTPS-443 inbound:' /var/log/messages | while read line do src_ip="$(echo ${line} | grep -oP "(?<=SRC=)([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})(?= DST=)")" src_desc="$(geoiplookup ${src_ip} 2>/dev/null | grep -v 'not found' | awk -F: '{$1=""; print $0}' | sed 's/\//_/g' | xargs)" if [ -z "${src_desc}" ] then src_desc="IP origin unknown" fi echo "${line}" | sed -r "s/SRC=${src_ip} /SRC=${src_ip} ${src_desc} /g" done
The result might look something like this:
Aug 30 16:11:13 ncc1701 kernel: [6204705.454169] HTTPS-443 inbound: IN=bond0 OUT= MAC=a0:48:1c:b8:17:b8:5c:f4:ab:67:c4:4b:08:00 SRC=104.129.204.86 US, United States US, GA, Georgia, Atlanta, 30301, 33.749001, -84.388000, 524, 404 AS22616 ZSCALER, INC. DST=192.168.122.137 LEN=60 TOS=0x00 PREC=0x00 TTL=58 ID=6674 DF PROTO=TCP SPT=19707 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0 Aug 30 16:11:14 ncc1701 kernel: [6204706.192422] HTTPS-443 inbound: IN=bond0 OUT= MAC=a0:48:1c:b8:17:b8:5c:f4:ab:67:c4:4b:08:00 SRC=45.12.32.102 IP origin unknown DST=192.168.122.137 LEN=60 TOS=0x00 PREC=0x00 TTL=53 ID=26339 DF PROTO=TCP SPT=56736 DPT=443 WINDOW=14480 RES=0x00 SYN URGP=0 Aug 30 16:11:15 ncc1701 kernel: [6204706.636596] HTTPS-443 inbound: IN=bond0 OUT= MAC=a0:48:1c:b8:17:b8:5c:f4:ab:67:c4:4b:08:00 SRC=192.168.122.1 IP origin unknown DST=192.168.122.137 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=37548 DF PROTO=TCP SPT=38372 DPT=443 WINDOW=14600 RES=0x00 SYN URGP=0
Useful stuff, except you would probably want to exclude your private network addresses from this list. So here’s an enhanced version of the previous command:
grep 'HTTPS-443 inbound:' /var/log/messages | while read line do src_ip="$(echo ${line} | grep -oP "(?<=SRC=)([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})(?= DST=)")" if [ $(echo ${src_ip} | grep -cE "(^0\.)|(^127\.)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)") -eq 0 ] then src_desc="$(geoiplookup ${src_ip} 2>/dev/null | grep -v 'not found' | awk -F: '{$1=""; print $0}' | sed 's/\//_/g' | xargs)" if [ -z "${src_desc}" ] then src_desc="IP origin unknown" fi echo "${line}" | sed -r "s/SRC=${src_ip} /SRC=${src_ip} ${src_desc} /g" fi done
So, what practical uses does this information have? Let’s imagine that for some reason I don’t want Zscaler connecting to my server. I run the previous command and save the output to a temporary file /tmp/iplog
, and the do something like this:
wc -l /tmp/iplog && grep -c ZSCALER /tmp/iplog && grep ZSCALER /tmp/iplog | grep -oP "(?<=SRC=)([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})(?= )" | sort -Vu 817 /tmp/iplog 413 104.129.204.86
In this data sample, this one external IP is responsible for more than half of all requests to my Web server. So blocking it may be a good idea. This command here will give you a list of organizations accessing your server:
grep -v 'origin unknown' /tmp/iplog | grep -oP "(?<= AS)[0-9]{3,} .*(?=DST=)" | awk '{$1=""; print $0}' | sort -u | sed 's/^ //g' Amazon.com, Inc. Automattic, Inc Bell Canada bet-at-home.com International Ltd. Case Western Reserve University CHINA UNICOM China169 Backbone China Unicom Guangdong IP network City Network Hosting AB Cogent Communications ColoCrossing Combell NV Comcast Cable Communications, LLC ...
And this command will count how many time each organization accessed your server:
grep -v 'origin unknown' /tmp/iplog | grep -oP "(?<= AS)[0-9]{3,} .*(?=DST=)" | awk '{$1=""; print $0}' | sed 's/^ //g' | sort | uniq -c | sort -k1rn 413 ZSCALER, INC. 61 Rostelecom 60 China Unicom Guangdong IP network 36 Automattic, Inc 24 Google LLC 15 OVH SAS 12 DataWeb Global Group B.V. 9 Host NIT Inc 9 Microsoft Corporation
You can also get a count by country of origin:
grep -v 'origin unknown' /tmp/iplog | awk -F, '{print $2}' | sed 's/^ //g' | sort | uniq -c | sort -k1rn 588 United States US 68 China CN 67 Russian Federation RU 19 France FR 11 Canada CA 8 Ukraine UA 7 Poland PL 6 Argentina AR 6 Czech Republic CZ
So when they tell you it’s the Russians and the Chinese who are breaking your server, take such information with a grain of salt. Most attacks originate from the US networks.
Perhaps unrelated to our topic of logging, here are a couple of iptables
tricks I find useful – both having to do with cleaning up your rules from time to time. Let’s say I banned a particular IP or range and now I want let those guys back in. Here’s an example:
# These IPs and ranges I want to take off my blacklist: iptables -S | grep DROP | grepcidr 104.129.204.0/24 -A INPUT -s 104.129.204.0/24 -j DROP -A INPUT -s 104.129.204.86/32 -j DROP # Here's one way of doing this: iptables -S | grep DROP | grepcidr 104.129.204.0/24 | while read i; do j="$(echo ${i} | sed 's/\-A/\-D/g')" iptables $(eval echo $j) done && service iptables save
Now, let’s imagine you blacklist some IPs, but you want them to be removed from your blacklist in a few days. Automatically, of course. Well, iptables
rules have a handy comment
option, allowing you to insert some arbitrary text into the rule without affecting its functionality. So our comment will be a date in epoch
format that will show when this particular rule is supposed to expire. Here’s what I mean:
# Let's add this IP range to blacklist with a comment # showing epoch timestamp seven days from now iptables -A INPUT -m comment --comment "$(date -d'now + 7 days' +'%s')" -s 104.129.204.0/24 -j DROP # This is what it will look like: iptables -S | grepcidr 104.129.204.0/24 -A INPUT -s 104.129.204.0/24 -m comment --comment "1568065502" -j DROP
The command below will go through your iptables
rules, pick those with what looks like an epoch
timestamp, and delete any expired rules:
iptables -S | egrep '[0-9]{10,}' | while read i do if [ $(echo ${i} | egrep -o '[0-9]{10,}') -lt $(date +'%s') ] then echo "Dropping expired rule: ${i}" j="$(echo ${i} | sed 's/\-A/\-D/g')" iptables $(eval echo $j) fi done && service iptables save
![](https://www.igoroseledko.com/wp-content/uploads/2020/11/Photo-2020-10-31-07-24-22_8181.jpg)
Experienced Unix/Linux System Administrator with 20-year background in Systems Analysis, Problem Resolution and Engineering Application Support in a large distributed Unix and Windows server environment. Strong problem determination skills. Good knowledge of networking, remote diagnostic techniques, firewalls and network security. Extensive experience with engineering application and database servers, high-availability systems, high-performance computing clusters, and process automation.