Networking

Unix and Linux network configuration. Multiple network interfaces. Bridged NICs. High-availability network configurations.

Applications

Reviews of latest Unix and Linux software. Helpful tips for application support admins. Automating application support.

Data

Disk partitioning, filesystems, directories, and files. Volume management, logical volumes, HA filesystems. Backups and disaster recovery.

Monitoring

Distributed server monitoring. Server performance and capacity planning. Monitoring applications, network status and user activity.

Commands & Shells

Cool Unix shell commands and options. Command-line tools and application. Things every Unix sysadmin needs to know.

Home » Networking

Troubleshooting InfiniBand Connections

Submitted by on May 4, 2011 – 10:36 am 6 Comments

The following is a brief troubleshooting guide for an InfiniBand network found in common HPC Linux clusters. Running these commands requires OFED 1.5.2 or later package installed on your systems. Additionally, the “pdsh” (parallel shell) command is part of the HP CMU cluster management software (version 4.2.1 used in our example) installed on the head node. If you don’t have CMU installed, below you will find a simple scripted alternative for running commands on multiple cluster nodes. In our cluster compute nodes are named “node1” through “node32”.

Identify hardware module used for the IB interface:

Sample output:

Check the state of the IB port:

Sample output:

Check the status of the subnet manager and start it if necessary:

Sample output:

Check the state of the IB port on the compute nodes:

Sample output:

Here’s a way of running the command above without pdsh:

The next step is to check the speed of IB ports on the head node:

Sample output:

… and on the compute nodes:

And here’s how you do this without pdsh:

More detailed analysis of the IB connection can be performed with the ibdiagnet command:

Sample output:

cat /sys/class/infiniband/mlx4_0/ports/1/lid
pdsh –w node[1-32] /sys/class/infiniband/mlx4_0/ports/1/lid
ibdiagnet –pc –c 1000

The final step is to check the error state of each port:

Sample output:

In our case, the “ibcheckerrors” command revealed a problem with the IB switch. This turned out to be a hardware problem and the switch needed to be replaced.

Print Friendly, PDF & Email

6 Comments »

  • Avatar kass9191 says:

    I’m taking a couple Pre-IB (International Baccalaureate) classes right now, and I’d like to move down from those classes next year, which will then become IB, so that I can have time to actually practice music at home. For a Music Major my primary focus should be my knowledge of music theory and the quality of my performance, but I’m afraid that moving down from two pre-IB classes one level might hurt my chances.

  • Avatar Dr Hank says:

    i’m trying out for 2 ib schools in my area
    i’m really nervous because in 1 of the schools.. it states that only 100 students are accepted
    how should i prepare myself?
    how difficult are the test?
    thanks
    I’m trying out for 2 ib highschools in my area…
    its stated that in 1 of the schools only 100 are accepted
    how difficult are the test?
    how should I prepare for it?
    thanks!

  • Avatar Kevin says:

    I want to go to Cambridge in England when I am older.
    Would IB classes be a good thing to take?

    I’m in middle school right now going on to High School soon.
    By the way I’m just finishing 7th grade right now and I just finished Algebra 1 Honors and going onto Geometry 1 Honors in 8th grade.

  • Avatar cardskid22 says:

    The IB program is the international baccalaureate program and running start is taking classes at the community college nearby. Which would a college like yale prefer on an application??

    Pleease help!! Thanks!!!

  • Avatar Smashing Pumpkins says:

    I am planning on going to an american university when i finish the IB programme.
    Now… wtf is the difference between graduate and undergraduate degrees and which one will i have to apply to?

    Also do IB students get any extra credit or financial aid in american universities? I am also an international student (i study in Malaysia)

    please enlighten me i am very confused :(
    thank you guys :)
    please answer the question.

  • Avatar zigg3ns says:

    I just got a notice from the school saying that i need to pay for my IB Exams.

    I tought only those who are getting the IB Diploma need to take the exams or is it for both?
    Please answer.

1 Pingbacks »

Leave a Reply

%d bloggers like this: