Most commonly iptables is used to allow, block, or redirect connections. However, it also has a logging feature that can be very useful for network traffic analysis and system security.

In the example below I will get a list of network services listening on my server and create iptables inbound connection logging rules. And then I’ll use some basic shell scripting to analyze collected data.

Here’s how you can get a list of listening network ports and service names:

for i in $(lsof -i -P -n | grep -oP "(?<=\*:)[0-9]{2,}(?= \(LISTEN)" | sort -nu); do lsof -i :${i} | grep -v COMMAND | awk -v i=$i '{print $1,i}' | sort -u; done | column -t

sshd       22
httpd      80
rpcbind    111
httpd      443
xinetd     873
rpc.rquot  875
httpd      1701
mysqld     3306
splunkd    8000
splunkd    8089

Let’s say we want iptables to log all inbound traffic for port 443. Here’s the rule for this:

iptables -I INPUT -p tcp --dport 443 -m state --state NEW -j LOG --log-prefix "HTTPS-443 inbound: "

Save the rules and reload iptables service and in /var/log/messages you should now be able to see entries like these:

Aug 30 16:28:09 ncc1701 kernel: [6205722.091794] HTTPS-443 inbound: IN=bond0 OUT= MAC=a0:48:1c:b8:17:b8:5c:f4:ab:67:c4:4b:08:00 SRC=192.0.101.11 DST=192.168.122.137 LEN=52 TOS=0x00 PREC=0x00 TTL=56 ID=5926 DF PROTO=TCP SPT=14784 DPT=443 WINDOW=29200 RES=0x00 SYN URGP=0
Aug 30 16:28:09 ncc1701 kernel: [6205722.092482] HTTPS-443 inbound: IN=bond0 OUT= MAC=a0:48:1c:b8:17:b8:5c:f4:ab:67:c4:4b:08:00 SRC=192.0.102.70 DST=192.168.122.137 LEN=52 TOS=0x00 PREC=0x00 TTL=54 ID=18456 DF PROTO=TCP SPT=56268 DPT=443 WINDOW=29200 RES=0x00 SYN URGP=0

Seeing the source IP is useful but, perhaps, seeing the source hostname, country of origin, and organization would be even better. Here’s how this can be accomplished:

grep 'HTTPS-443 inbound:' /var/log/messages | while read line
do  
  src_ip="$(echo ${line} | grep -oP "(?<=SRC=)([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})(?= DST=)")"
  src_desc="$(geoiplookup ${src_ip} 2>/dev/null | grep -v 'not found' | awk -F: '{$1=""; print $0}' | sed 's/\//_/g' | xargs)"
  if [ -z "${src_desc}" ]
  then
    src_desc="IP origin unknown"
  fi
  echo "${line}" | sed -r "s/SRC=${src_ip} /SRC=${src_ip} ${src_desc} /g"
done

The result might look something like this:

Aug 30 16:11:13 ncc1701 kernel: [6204705.454169] HTTPS-443 inbound: IN=bond0 OUT= MAC=a0:48:1c:b8:17:b8:5c:f4:ab:67:c4:4b:08:00 SRC=104.129.204.86 US, United States US, GA, Georgia, Atlanta, 30301, 33.749001, -84.388000, 524, 404 AS22616 ZSCALER, INC. DST=192.168.122.137 LEN=60 TOS=0x00 PREC=0x00 TTL=58 ID=6674 DF PROTO=TCP SPT=19707 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0
Aug 30 16:11:14 ncc1701 kernel: [6204706.192422] HTTPS-443 inbound: IN=bond0 OUT= MAC=a0:48:1c:b8:17:b8:5c:f4:ab:67:c4:4b:08:00 SRC=45.12.32.102 IP origin unknown DST=192.168.122.137 LEN=60 TOS=0x00 PREC=0x00 TTL=53 ID=26339 DF PROTO=TCP SPT=56736 DPT=443 WINDOW=14480 RES=0x00 SYN URGP=0
Aug 30 16:11:15 ncc1701 kernel: [6204706.636596] HTTPS-443 inbound: IN=bond0 OUT= MAC=a0:48:1c:b8:17:b8:5c:f4:ab:67:c4:4b:08:00 SRC=192.168.122.1 IP origin unknown DST=192.168.122.137 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=37548 DF PROTO=TCP SPT=38372 DPT=443 WINDOW=14600 RES=0x00 SYN URGP=0

Useful stuff, except you would probably want to exclude your private network addresses from this list. So here’s an enhanced version of the previous command:

grep 'HTTPS-443 inbound:' /var/log/messages | while read line
do  
  src_ip="$(echo ${line} | grep -oP "(?<=SRC=)([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})(?= DST=)")"
  if [ $(echo ${src_ip} | grep -cE "(^0\.)|(^127\.)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)") -eq 0 ]
  then
    src_desc="$(geoiplookup ${src_ip} 2>/dev/null | grep -v 'not found' | awk -F: '{$1=""; print $0}' | sed 's/\//_/g' | xargs)"
    if [ -z "${src_desc}" ]
    then
      src_desc="IP origin unknown"
    fi
    echo "${line}" | sed -r "s/SRC=${src_ip} /SRC=${src_ip} ${src_desc} /g"
  fi
done

So, what practical uses does this information have? Let’s imagine that for some reason I don’t want Zscaler connecting to my server. I run the previous command and save the output to a temporary file /tmp/iplog, and the do something like this:

wc -l /tmp/iplog && grep -c ZSCALER /tmp/iplog && grep ZSCALER /tmp/iplog | grep -oP "(?<=SRC=)([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})(?= )" | sort -Vu
817 /tmp/iplog
413
104.129.204.86

In this data sample, this one external IP is responsible for more than half of all requests to my Web server. So blocking it may be a good idea. This command here will give you a list of organizations accessing your server:

grep -v 'origin unknown' /tmp/iplog | grep -oP "(?<= AS)[0-9]{3,} .*(?=DST=)" | awk '{$1=""; print $0}' | sort -u | sed 's/^ //g'

Amazon.com, Inc.
Automattic, Inc
Bell Canada
bet-at-home.com International Ltd.
Case Western Reserve University
CHINA UNICOM China169 Backbone
China Unicom Guangdong IP network
City Network Hosting AB
Cogent Communications
ColoCrossing
Combell NV
Comcast Cable Communications, LLC
...

And this command will count how many time each organization accessed your server:

grep -v 'origin unknown' /tmp/iplog | grep -oP "(?<= AS)[0-9]{3,} .*(?=DST=)" | awk '{$1=""; print $0}' | sed 's/^ //g' | sort | uniq -c | sort -k1rn
    413 ZSCALER, INC.
     61 Rostelecom
     60 China Unicom Guangdong IP network
     36 Automattic, Inc
     24 Google LLC
     15 OVH SAS
     12 DataWeb Global Group B.V.
      9 Host NIT Inc
      9 Microsoft Corporation

You can also get a count by country of origin:

grep -v 'origin unknown' /tmp/iplog | awk -F, '{print $2}' | sed 's/^ //g' | sort | uniq -c | sort -k1rn
    588 United States US
     68 China CN
     67 Russian Federation RU
     19 France FR
     11 Canada CA
      8 Ukraine UA
      7 Poland PL
      6 Argentina AR
      6 Czech Republic CZ

So when they tell you it’s the Russians and the Chinese who are breaking your server, take such information with a grain of salt. Most attacks originate from the US networks.

Perhaps unrelated to our topic of logging, here are a couple of iptables tricks I find useful – both having to do with cleaning up your rules from time to time. Let’s say I banned a particular IP or range and now I want let those guys back in. Here’s an example:

# These IPs and ranges I want to take off my blacklist:
iptables -S | grep DROP | grepcidr 104.129.204.0/24
-A INPUT -s 104.129.204.0/24 -j DROP
-A INPUT -s 104.129.204.86/32 -j DROP

# Here's one way of doing this:
iptables -S | grep DROP | grepcidr 104.129.204.0/24 | while read i; do 
j="$(echo ${i} | sed 's/\-A/\-D/g')"
iptables $(eval echo $j)
done && service iptables save

Now, let’s imagine you blacklist some IPs, but you want them to be removed from your blacklist in a few days. Automatically, of course. Well, iptables rules have a handy comment option, allowing you to insert some arbitrary text into the rule without affecting its functionality. So our comment will be a date in epoch format that will show when this particular rule is supposed to expire. Here’s what I mean:

# Let's add this IP range to blacklist with a comment
# showing epoch timestamp seven days from now
iptables -A INPUT -m comment --comment "$(date -d'now + 7 days' +'%s')" -s 104.129.204.0/24 -j DROP

# This is what it will look like:
iptables -S | grepcidr 104.129.204.0/24
-A INPUT -s 104.129.204.0/24 -m comment --comment "1568065502" -j DROP

The command below will go through your iptables rules, pick those with what looks like an epoch timestamp, and delete any expired rules:

iptables -S | egrep '[0-9]{10,}' | while read i
do
  if [ $(echo ${i} | egrep -o '[0-9]{10,}') -lt $(date +'%s') ]
  then
    echo "Dropping expired rule: ${i}"
    j="$(echo ${i} | sed 's/\-A/\-D/g')"
    iptables $(eval echo $j)
  fi
done && service iptables save