Recently I ran into a situation where I needed to reboot several HPC cluster compute nodes. A couple of systems were still running user jobs that I did not want to interrupt. I also didn’t want to sit around and watch those jobs, as there was no way of telling how long they could take to complete. The solution was to write a short script that looks something like this:

#!/bin/ksh
while [ `uptime | awk '{print $10}' | awk -F'.' '{print $1}'` -ge 1 ]
do
        wait
done
echo "Autorebooting node `hostname` on low CPU utilization." | mailx -s "Autoreboot of `hostname` in progress" my.cell@att.com
/sbin/reboot -f

The script should be executed as follows:

nohup /full/path/to/script/script.ksh &

The idea is simple: the script runs in the background even after you log out; it monitors system load, which in our case will always be above 1 as long as user jobs are running. As soon as system load falls below 1, the script executes a forced reboot. You don’t need to do a forced reboot unless you have some persistent problems with your systems doing a graceful shutdown. In my case the problem has to do with PANFS filesystems hanging on unmount. A forced reboot is a sure way to restart the system and any disk corruption is usually fixed automatically during boot up. But your situation may be different, so don’t use “reboot -f” unless you have to.