Networking

Unix and Linux network configuration. Multiple network interfaces. Bridged NICs. High-availability network configurations.

Applications

Reviews of latest Unix and Linux software. Helpful tips for application support admins. Automating application support.

Data

Disk partitioning, filesystems, directories, and files. Volume management, logical volumes, HA filesystems. Backups and disaster recovery.

Monitoring

Distributed server monitoring. Server performance and capacity planning. Monitoring applications, network status and user activity.

Commands & Shells

Cool Unix shell commands and options. Command-line tools and application. Things every Unix sysadmin needs to know.

Home » Commands & Shells, Processes

Monitoring Application Network Connections

Submitted by on February 11, 2016 – 3:12 pm

A Java application on one of my servers has been causing problems by opening up too many network connections and now closing them quickly enough. I needed to be notified when the number of connections opened by a particular process exceeded a certain limit. I also needed a record of open connections over time. Below is a quick script I wrote for that.

There’re a couple of things I should point out. First, the options used with the netstat command:

netstat -tunap

The “p” is for “- -process”, which takes PID as argument. Also, I did not want to be notified more than once per day, so with every email alert the script touches a lock file and then compares the mtime on the file to the current time. If mtime is more than 24 x 60 x 60 seconds in the past (24 hours), the another email will be sent and timestamp on the lock file will be updated.
if [ $(echo $(( (`date +%s` - `stat -L --format %Y ${lockfile}`) < (24*60*60) ))) -eq 0 ]

This syntax will return 1 as long as the lock file is less than 86400 seconds old. The output will be 0 when the lock file is older than 24 hours, at which point another email alert may be sent. Also, note how the first letter of the “[p]rocess_string” is enclosed in square brackets. This is avoid matching “grep” in “ps -ef | grep” output. Finally, I use “grep -m1” to make sure I have only one PID. If you need to match multiple PIDs, you would need to use a “for” loop.

And here’s the script:

#!/bin/bash
lim=1000
email="email@domain.com"
this_script=$(basename "$(test -L "$0" && readlink "$0" || echo "$0")")
time_db=$(date +'%Y-%m-%d %H:%M:%S')
lockfile="/tmp/${this_script}.lock"
logfile="/var/tmp/${this_script}.log"
this_host=$(echo ${HOSTNAME} | awk -F'.' '{print $1}')

pid=$(ps -ef | grep -m1 "[p]rocess_string" | awk '{print $2}')
c=$(netstat -tunap | grep -c ${pid})

echo -e "${time_db}\t${this_host}\t${pid}\t${c}" >> "${logfile}"

notify() {
	echo "process_string PID ${pid} has opened ${c} TCP connections on ${HOSTNAME}" | \
	mailx -s "process_string network alert from ${HOSTNAME}" "${email}"
	touch "${lockfile}"
}

if [ ${c} -gt ${lim} ]
then
	if [ -f "${lockfile}" ]
	then
		if [ $(echo $(( (`date +%s` - `stat -L --format %Y ${lockfile}`) < (24*60*60) ))) -eq 0 ]
		then
			notify
		fi
	else
		notify
	fi
fi

 

Print Friendly, PDF & Email

Leave a Reply