OpenVZ DIsk Monitors on OpenNMS

At work, we run a lot of Parallels Virtuozzo, which is a commercially supported version of OpenVZ. There are a couple of challenges monitoring OpenVZ, since it abstracts out a lot of the OS and snmpd can’t get the data.

Thankfully, there is the ability to extend snmpd with scripts. The basic extend functionality is covered in the man pages, so I’m going to focus more on the OpenNMS side of things.

First up, create a script to grab the disk stats from the OpenVZ host. The script I use to get the disk usage stats is :
#!/bin/bash
DF_DATA="$(df / | grep / | tr '%' ' ')"
TOTAL=$(echo $DF_DATA | awk '{ print $2 }')
USED=$(echo $DF_DATA | awk '{ print $3 }')
FREE=$(echo $DF_DATA | awk '{ print $4 }')
PCNT=$(echo $DF_DATA | awk '{ print $5 }')
echo $TOTAL
echo $USED
echo $FREE
echo $PCNT

Then we add the following to snmpd.conf and restart the daemon:
extend vzdisks /usr/local/sbin/vzdisks.sh

At this point, you shoudl be able to run snmpwalk and see the data :
ksonney:~# snmpwalk -c nocstats -v 2c localhost .1.3.6.1.4.1.8072.1.3.2.4.1.2.7.118
NET-SNMP-EXTEND-MIB::nsExtendOutLine."vzdisks".1 = STRING: 188743680
NET-SNMP-EXTEND-MIB::nsExtendOutLine."vzdisks".2 = STRING: 15437520
NET-SNMP-EXTEND-MIB::nsExtendOutLine."vzdisks".3 = STRING: 173306160
NET-SNMP-EXTEND-MIB::nsExtendOutLine."vzdisks".4 = STRING: 9

(you can see all the extends you have running by using the OID .1.3.6.1.4.1.8072.1.3.2.4.1 – everything after that is the unique identifier for the particular entend)

At that point, you can start working on the OpenNMS configuration. Since I run more than one extension, I have a file called etc/datacollection/extensions.xml. The clause I use for the vz disks :

<datacollection-group name="extensions">
<group name="vzdisk-stats" ifType="ignore">
<mibObj oid=".1.3.6.1.4.1.8072.1.3.2.4.1.2.7.118.122.100.105.115.107.115" instance="1" alias="totalDisk" type="octetstring" />
<mibObj oid=".1.3.6.1.4.1.8072.1.3.2.4.1.2.7.118.122.100.105.115.107.115" instance="2" alias="usedDisk" type="octetstring" />
<mibObj oid=".1.3.6.1.4.1.8072.1.3.2.4.1.2.7.118.122.100.105.115.107.115" instance="3" alias="freeDisk" type="octetstring" />
<mibObj oid=".1.3.6.1.4.1.8072.1.3.2.4.1.2.7.118.122.100.105.115.107.115" instance="4" alias="percentDisk" type="octetstring" />
</group>
<systemDef name="Net-Snmp Extensions">
<sysoidMask>.1.3.6.1.4.1.8072.3.</sysoidMask>
<collect>
<includeGroup>vzdisk-stats</includeGroup>
<includeGroup>mysql-backup-stats</includeGroup>
</collect>
</systemDef>

</datacollection-group>

Add the following line to etc/datacollection-config.xml :
<include-collection dataCollectionGroup="extensions"/>

And then add in the OpenNMS Configuration for graphing the stats (etc/snmp-graph.properties.d/vzdisk.properties) :
# Virtuozzo reports
reports=netsnmp.vzdisks, netsnmp.vzdiskuse
# Custom NET-SNMP Script extend: Disk stats from Containers
report.netsnmp.vzdisks.name=Virtuozzo Disk Status
report.netsnmp.vzdisks.columns=percentDisk
report.netsnmp.vzdisks.type=nodeSnmp
report.netsnmp.vzdisks.command=--title="VZDisk Percent Used" \
--vertical-label="Percent Used" \
DEF:pdisk={rrd1}:percentDisk:AVERAGE \
LINE2:pdisk#0066FF:"Percent Used " \
GPRINT:pdisk:AVERAGE:"Current \\: %8.0lf" \
GPRINT:pdisk:MAX:" Max \\: %8.0lf\\n"
report.netsnmp.vzdiskuse.name=Virtuozzo Disk Use
report.netsnmp.vzdiskuse.columns=totalDisk,usedDisk,freeDisk
report.netsnmp.vzdiskuse.type=nodeSnmp
report.netsnmp.vdiskuse.command=--title="VZDisk Usage Stats" \
--vertical-label="Disk Space in Bytes" \
DEF:tdisk={rrd1}:totalDisk:AVERAGE \
DEF:udisk={rrd2}:usedDisk:AVERAGE \
DEF:fdisk={rrd3}:freeDisk:AVERAGE \
AREA:udisk#0066FF:"Used " \
GPRINT:udisk:AVERAGE:"Current \\: %8.0lf" \
GPRINT:udisk:MAX:" Max \\: %8.0lf\\n" \
STACK:fdisk#00aa00:"Free Space " \
GPRINT:fdisk:AVERAGE:"Current \\: %8.0lf" \
GRPINT:fdisk:MAX:" Max \\: %8.0lf\\n" \
GPRINT:tdisk:MAX:"Total Disk Size \\: %8.0lf\\n"

At this point, if you restart OpenNMS, it should start pulling the data. Give it about 15 mins for the data to start being pulled in, and you should see a graph for percent disk usage showing up.

For a little advanced fun, you can add a threshold to the threshd-configuration.xml and get notifications when it goes over :

<package name="vzDisk">
<filter>IPADDR != '0.0.0.0' &amp; (nodeSysOID LIKE '.1.3.6.1.4.1.8072.%')</filter>
<include-range begin="1.1.1.1" end="254.254.254.254"/>
<include-range begin="::1" end="ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff"/>
<service name="SNMP" interval="300000" user-defined="false" status="on">
<parameter key="thresholding-group" value="vzDisk"/>
</service>
</package>

And for thresholds.xml :
<group name="vzDisk" rrdRepository="/opt/opennms/share/rrd/snmp">
<threshold description="Virtuozzo Disk High Threshold"
type="high" ds-type="node" value="85.0" rearm="84.0"
trigger="2" ds-label="VZ DIsk usage" filterOperator="or" ds-name="percentDisk"/>
</group>

Restart OpenNMS, and you should be good to go.

About Kevin Sonney

Kevin Sonney - who, contrary to popular opinion was NOT raised by wolves - grew up in central North Carolina. He fell into the technology field by accident in 1991, when he gave up the wild and crazy lifestyle of an on-air AM radio DJ to become a mundane technical support monkey. The technology industry has never really recovered from this. Kevin has worked for such names as IBM, Red Hat, webslingerZ, and Lulu Technologies (we won't mention the ones that didn't survive the experience). He currently works as a Linux Administrator for Apptio. In his spare time he rescues stray animals and plays video games with his two sons. His wife, we're sad to say, helps him get past the really hard bits. Kevin is still not very mundane, he just got better at hiding it.
This entry was posted in code, Projects, SysAdmin, Work. Bookmark the permalink.