Generating IP Whitelists

Originally published November 30, 2020 @ 12:09 pm

I have several scripts that scan various log files for signs of suspicious activity and block the offending IPs on my Web servers – pretty standard stuff. The trick, of course, is not to block yourself or someone you care about.

Here’s an example of what I am talking about:

# Extract IPs responsible for a particular error message in the
# Apache log and block them via iptables
block $(grep error.*ModSecurity /var/log/httpd/igoroseledko.com/error_log | grep -oE "([0-9]{1,3}\.){3}([0-9]{1,3})" | sort -uV | egrep -vf "${whitelist}" | xargs)

# The 'block' command is a small script:
#!/bin/bash
if [ ! -z "" ]; then
  for i in ${@}; do
    iptables -A INPUT -s "${i}" -j DROP
  done
  /sbin/service iptables save 2>/dev/null 1>$2
  tmpfile=$(mktemp)
  /sbin/iptables-save | awk '/^COMMIT$/ { delete x; }; !x[$0]++' > ${tmpfile}
  /sbin/iptables -F
  /sbin/iptables-restore < ${tmpfile}
  /sbin/service iptables save
  /sbin/service iptables reload
  /bin/rm -f ${tmpfile}
fi

The key here is the ${whitelist} containing IP addresses, I definitely don’t want to block no matter what. One way to generate such a whitelist is to scan the contents of /etc/hosts.allow and /etc/hosts. Because these two files can use either subnet or CIDR notation, we need to expand those into individual IPs. This should work OK if you don’t have any /8 or wider ranges.

I am extracting IPs and IP ranges from the two aforementioned config files in the example below. I am also using a netmast2cidr function to convert individual IPs and any ranges that use netmask notation to use CIDR notation. I then use nmap to expand the resulting ranges into a large list of individual IPs. And finally, I use the generated whitelist with my block command.

whitelist="$(mktemp)"
p="([0-9]{1,3}\.){3}([0-9]{1,3})"

netmask2cidr() {
 IFS=/ read i j
 if [ ! -z "${j}" ]; then
   k="$(ipcalc -p 1.1.1.1 ${j} 2>/dev/null | sed -n 's@^PREFIX=\(.*\)@@p')"
   if [ ! -z "${k}" ]; then echo "${i}/${k}"; else echo "${i}/${j}"; fi
 else echo "${i}/32"
 fi
}

while read line ; do
 echo ${line} | netmask2cidr
done < <(grep -hoE "(${p})(/((${p})|[0-9]{1,2}))?" /etc/hosts.allow /etc/hosts | sort -uV) | \
xargs -n1 -I% nmap -sL -n % | grep -oE "([0-9]{1,3}\.){3}([0-9]{1,3})" > "${whitelist}"

block $(grep error.*ModSecurity /var/log/httpd/igoroseledko.com/error_log | grep -oE "${p}" | sort -uV | egrep -vf "${whitelist}" | xargs)

This works fine as long as you keep the whitelist relatively short. But what if you want to add all of the private networks (you know, ranges like 10.0.0.0/8) to your whitelist? All the private network ranges contain 34,668,544 IPs – this would make one hell of a pattern file for grep to parse.

The solution is to use grepcidr with a whitelist pattern file that uses CIDR notation. Here’s an example similar to the previous one, but with a couple of notable differences: I am adding the private networks, my external IP address (the wget bit below), I am no longer using nmap, and now I use grepcidr instead of grep.

whitelist="$(mktemp)"
p="([0-9]{1,3}\.){3}([0-9]{1,3})"

netmask2cidr() {
 IFS=/ read i j
 if [ ! -z "${j}" ]; then
   k="$(ipcalc -p 1.1.1.1 ${j} 2>/dev/null | sed -n 's@^PREFIX=\(.*\)@@p')"
   if [ ! -z "${k}" ]; then echo "${i}/${k}"; else echo "${i}/${j}"; fi
 else echo "${i}/32"
 fi
}

while read line ; do
 echo ${line} | netmask2cidr
done < <(grep -hoE "(${p})(/((${p})|[0-9]{1,2}))?" /etc/hosts.allow /etc/hosts | sort -uV) > "${whitelist}"
echo -e "127.0.0.0/8\n10.0.0.0/8\n172.16.0.0/12\n192.168.0.0/16" >> "${whitelist}"
wget http://ipecho.net/plain -O - -q 2>/dev/null | awk '{print $0"/32"}' >> "${whitelist}"
sort -uV "${whitelist}" | sponge "${whitelist}"

block $(grep error.*ModSecurity /var/log/httpd/igoroseledko.com/error_log | grep -oE "${p}" | sort -uV | grepcidr -vf "${whitelist}" | xargs)