The following is a brief troubleshooting guide for an InfiniBand network found in common HPC Linux clusters. Running these commands requires OFED 1.5.2 or later package installed on your systems. Additionally, the “pdsh” (parallel shell) command is part of the HP CMU cluster management software (version 4.2.1 used in our example) installed on the head node. If you don’t have CMU installed, below you will find a simple scripted alternative for running commands on multiple cluster nodes. In our cluster compute nodes are named “node1″ through “node32″.

Identify hardware module used for the IB interface:

Sample output:

Check the state of the IB port:

Sample output:

Check the status of the subnet manager and start it if necessary:

Sample output:

Check the state of the IB port on the compute nodes:

Sample output:

Here’s a way of running the command above without pdsh:

The next step is to check the speed of IB ports on the head node:

Sample output:

… and on the compute nodes:

And here’s how you do this without pdsh:

More detailed analysis of the IB connection can be performed with the ibdiagnet command:

Sample output:

cat /sys/class/infiniband/mlx4_0/ports/1/lid
pdsh –w node[1-32] /sys/class/infiniband/mlx4_0/ports/1/lid
ibdiagnet –pc –c 1000

The final step is to check the error state of each port:

Sample output:

In our case, the “ibcheckerrors” command revealed a problem with the IB switch. This turned out to be a hardware problem and the switch needed to be replaced.