Networking

Unix and Linux network configuration. Multiple network interfaces. Bridged NICs. High-availability network configurations.

Applications

Reviews of latest Unix and Linux software. Helpful tips for application support admins. Automating application support.

Data

Disk partitioning, filesystems, directories, and files. Volume management, logical volumes, HA filesystems. Backups and disaster recovery.

Monitoring

Distributed server monitoring. Server performance and capacity planning. Monitoring applications, network status and user activity.

Commands & Shells

Cool Unix shell commands and options. Command-line tools and application. Things every Unix sysadmin needs to know.

Home » Commands & Shells

Installing and Configuring Ganglia

Submitted by on March 25, 2008 – 4:19 pm 18 Comments

Ganglia is a distributed performance monitoring application used primarily for tracking status of high-performance compute (HPC) clusters. Ganglia is a royal pain in the ass to install and configure even for a seasoned Unix sysadmin. Ganglia is a nice tool – very functional and free. It is a bit outdated in its design. It is poorly documented and there isn’t much useful information available online about installing and configuring Ganglia. First, some basics about Ganglia-related pieces of the puzzle:

rrdtool

The glorious Round Robin Database package. This needs to be installed on the Ganglia master server (the one collecting and presenting all the data; see diagram below).

core

When installing from RPM, Ganglia core is a separate package that needs to be installed first. When building from source – don’t worry about it.

gmond

Ganglia monitoring daemon – runs at boot time on all monitored nodes.

gmetad

Ganglia Meta Daemon – runs only on the Ganglia master server. Collects data from gmond instances on the monitored cluster nodes.

web-frontend

A set of PHP files for your Web server to display Ganglia data and graphs.

SAMPLE LAYOUT

ganglia_server – the master Ganglia server which holds all the collected performance data and runs the Web server.
head_node1 – the head node of CLUSTER1.
in1, in2, … , in41 – compute nodes of CLUSTER1.
head_node2 – the head node of CLUSTER2.
node1, node2, … , node60 – compute nodes of CLUSTER2.

In most cases, individual compute nodes are locked into cluster’s internal network. They cannot directly communicate with the outside network. Only the head node can. As the result, the Ganglia master server, which is located outside of the cluster, cannot access these nodes directly. So, data collected by each compute node is passed on to the head node and from there it goes to the Ganglia server. Nothing to it.

INSTALLING RRDTOOL

RRDTool needs to be installed ONLY on the Ganglia master server (see diagram above). This app is usually installed by default in most Linux distros. Unfortunately, most of these default installations are bastardized and cannot be used by Ganglia. I would strongly recommend downloading rrdtool source and compiling it. You don’t need to uninstall the existing version of rrdtool before compiling a new version.

Let’s say the new version of rrdtool was compiled and installed in /usr/local/rrdtool-1.2.27/. Now you need to relink three RRDTool-related binary files in /usr/bin to point to the new version:

for i in rrdcgi rrdtool rrdupdate
do
mv /usr/bin/${i} /usr/bin/${i}.orig
ln -s /usr/local/rrdtool-1.2.27/bin/${i} /usr/bin/${i}
done

That should be it for the RRDTool.

INSTALLING GMOND

Gmond needs to be installed on all monitored nodes and on the cluster’s head node, even if you don’t want to monitor the head node. No need to install gmond on the Ganglia master server. Download the latest source code for Ganglia and put it in /tmp. Unzip, untar, cd to the resulting directory and…

sh ./configure
make
make install
which gmond

Repeat on every freaking compute node and head node on both CLUSTER1 and 2. Get rid of the source code directory.

Now, to get gmond started at boot time:

vi /etc/init.d/gmond and paste the following nonsense:

#! /bin/sh

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/sbin/gmond
NAME=gmond
DESC="Ganglia Monitor Daemon"

test -x $DAEMON || exit 0
set -e

case "$1" in
start)
echo -n "Starting $DESC: "
start-stop-daemon --start --quiet --pidfile /var/run/$NAME.pid 
--exec $DAEMON
echo "$NAME."
;;
stop)
echo -n "Stopping $DESC: "
start-stop-daemon --stop --quiet --oknodo 
--exec $DAEMON 2>&1 > /dev/null
echo "$NAME."
;;
reload)
;;
restart|force-reload)
$0 stop
$0 start
;;
*)
N=/etc/init.d/$NAME
# echo "Usage: $N {start|stop|restart|reload|force-reload}" >&2
echo "Usage: $N {start|stop|restart|force-reload}" >&2
exit 1
;;
esac

exit 0

Save and exit from vi and do the following:

chmod 755 /etc/init.d/gmond
chkconfig --add gmond

CONFIGURING GMOND

This is where the fun begins. First, generate a default configuration file on the head node AND one of the compute nodes in CLUSTER1:

gmond --default_config > /etc/gmond.conf

Now open this file on one of the compute nodes and make changes to the following sections (see the diagram above):

cluster {
name = "CLUSTER1"
}

udp_send_channel {
host = head_node1
port = 8649
}

tcp_accept_channel {
port = 8649
}

Remove any sections about joining multicast or udp_recv_channel. Put this gmond.conf into /etc/ on every compute node of CLUSTER1.

Before you get up for a cup of coffee, edit the /etc/gmond.conf on the head node of CLUSTER1 (head_node1). Make changes to the following sections:

cluster {
name = "CLUSTER1"
}

udp_recv_channel {
port = 8649
}

tcp_accept_channel {
port = 8649
}

Remove any sections about joining multicast or udp_send_channel. Save the file and now you may get that coffee, ’cause there’s much more.

Configure CLUSTER2 in the same way, but replace “CLUSTER1” with “CLUSTER2” and “head_node1” with “head_node2”, correspondingly.

INSTALLING GMETAD

Once again: install gmetad ONLY on the Ganglia master server (see diagram above). Download the latest source code for Ganglia and put it in /tmp. Unzip, untar, cd to the resulting directory and…

sh ./configure 
CFLAGS"-I/usr/local/rrdtool-1.2.27/include" 
CPPFLAGS="-I/usr/local/rrdtool-1.2.27/include" 
LDFLAGS="-L/usr/local/rrdtool-1.2.27/lib" --with-gmetad

Naturally, you will replace “rrdtool-1.2.27” with the correct version of rrdtool you just installed. And then the usual:
make
make install
which gmetad

Don’t delete source code directory just yet but proceed to configuring gmetad.

CONFIGURING GMETAD

Edit the default configuration file: vi /etc/gmetad.conf and add the following lines:

data_source "CLUSTER1" head_node1
data_source "CLUSTER2" head_node2

RRAs "RRA:AVERAGE:0.5:1:105408"

To get gmetad started at boot time, vi /etc/gmetad and copy/paste this stuff into it:
#! /bin/sh PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/sbin/gmetad
NAME=gmetad
DESC="Ganglia Meta Daemon"

test -x $DAEMON || exit 0
set -e

case "$1" in
start)
echo -n "Starting $DESC: "
start-stop-daemon --start --quiet --pidfile /var/run/$NAME.pid 
--exec $DAEMON
echo "$NAME."
;;
stop)
echo -n "Stopping $DESC: "
start-stop-daemon --stop --quiet --oknodo 
--exec $DAEMON 2>&1 > /dev/null
echo "$NAME."
;;
reload)
;;
restart|force-reload)
$0 stop
$0 start
;;
*)
N=/etc/init.d/$NAME
# echo "Usage: $N {start|stop|restart|reload|force-reload}" >&2
echo "Usage: $N {start|stop|restart|force-reload}" >&2
exit 1
;;
esac

exit 0

Save and exit from vi and do the following:

chmod 755 /etc/init.d/gmetad
chkconfig --add gmetad

INSTALLING WEB-FRONTEND

In the source code directory there will be a “web” subfolder. Move it over to the htdocs folder of you apache2 server (oh, you also need to run apache2 and PHP). So, as an example:

mv /tmp/ganglia_3.0.5/web /srv/www/htdocs/ganglia
chown -R nobody:nobody /srv/www/htdocs/ganglia

That should do it.

STARTING EVERYTHING

On both head_node1 and head_node2 do “/etc/init.d/gmond start” and do the same on all compute nodes on both clusters.

On the Ganglia master server do “/etc/init.d/gmetad start”

Open your browser and go to http://ganglia_server.domain/ganglia/

You may or may not see something like this:

If you see nothing, just relax, have a drink, take a three-day weekend and on Tuesday start from the top of this page. And this time pay attention to the stupid details.

Print Friendly, PDF & Email

18 Comments »

  • ArianaCigo says:

    hi, thanks,The article was very well written, very helpful to me

  • Jeff Moon says:

    Really good article. I was wondering if you could add the config file for gmetad. It looks like there was a stub for it, but it’s not there. Thanks.

  • Cameron says:

    It’s 2012. Ganglia is still poorly documented. It still doesn’t start reliably. There’s still zero useful troubleshooting info. (This includes “start over and pay more attention. Not useful.) The developers still don’t acknowledge these problems. Before wasting time on this mess, consider alternatives.

  • Ryan Z says:

    How long does it take to surgically remove a Ganglion Cyst within the wrist?

  • sarah w says:

    Most cysts are some what soft and squishy, but I’ve never had a ganglion cyst before, and I honestly thought it was a bone in my wrist that was dislocated and was now protruding upwards, but apparently it’s a ‘ganglion cyst’. It looks like one too, but it doesn’t feel like a cyst, it feels like a bone.

    Some help would be appreciated please?

  • Cole says:

    Today in my nervous systems class we were talking about the cranial nerves. My professor said that only ANS nerves have ganglia in the PNS, and all sensory and somatic motor nerves synapse within the CNS. When we were looking at the vestibulcochlear nerve (cranial nerve 9) I noticed in the diagram the vestibular ganglia appeared to be in the PNS, but since this is a sensory only nerve, the ganglia should be in the CNS. I asked my professor about it after class, and he said it was a good question and that I should research it. I was wondering if anyone here has taken a neurobiology class and can help me out. Thanks in advance.

  • rndmaktn says:

    What is a way to get a small-medium ganglion cyst on my right wrist to go away naturally? Thanks.
    I’d like surgery or aspiration to be my last resort.

  • sarah w says:

    She stopped breathing after birth and has has a MRI diagnosing an insult to the basal ganglia,Does anyone know much about this condition and what should i expect..Anyone know of anyone with this condition …Any help greatly appreciated..

  • Hannah says:

    What is the recommended after care for a ganglion cyst that was,aspirated

  • Austin says:

    I have a ganglion on my foot just below my ankle. I want it removed and yes i know all the risks and treatment options etc. I have my appointment next week and i want to know how to make it sound like i have compressed nerves so that he will remove it. Whats the best thing to say?
    There is nothing you can tell me about ganglions that i dont already know, i know all the treatment options etc. All i want to know is what to say to fake nerve compression, thats it.

  • Mackenzie P says:

    i know of other people who have had their ganglion cyst surgically removed and they had to wear a bulky dressing but a little after that they had to wear a cast?

    i don’t know, very confused.

  • ScRSC says:

    I have this soft mass the size of walnut in my left groin. No pain just discomfort, like tightening. If I stretch my hips it hurts but if I press on it it doesn’t at all. When I lie down it feels all right. I had a infection not long ago si it could be a ganglia but it be hernia also! Same symptoms!

  • tjpimpin says:

    i recently got a ganglion cyst removal and am wondering how long will it take to heal. And i believe i have internal stitches. i don’t know how big the cut is or what type of stitches it is. next week i have an appointment for a check up and to take off the cast type thingy i’m wearing…any answer is appreciated…
    does anyone know a picture of the a possible scar that will form?

  • Kristian says:

    I have had a ganglion cyst for about a year, but its as hard as a bone. I play many sports and I also enjoy swimming during they summer. Is there anything I have to wear after getting my cyst removed?

  • D3ZZY says:

    4 years ago i had a ganglion cyst on my right wrist. I had it surgically removed. I have another starting to develop on the same wrist. What do they do the second time? do they do the same thing or do they do something new?

  • Phillip123 says:

    I was planning on eating a raw oyster, and I know they are alive when you eat them. Of course, it is almost certain that they can not feel pain and are not conscious (being that they lack a central nervous system and a brain), but just to be safe, I want to quickly destroy the ganglia to remove the possibility and to further make sure that it doesn’t squiggle in my mouth. Where on the oyster is it located?

  • Sun star says:

    Great Article… Works as explained.. :)

2 Pingbacks »

Leave a Reply to rndmaktn Cancel reply

%d bloggers like this: