Nagios

mikeshn · 08-14-2009, 08:58 AM

I install Nagios and few servers with nrpe and pnp4nagios.
Is there a way to check Memory usage through nrpe?

Thanks

MensaWater · 08-14-2009, 09:21 AM

Yes.

Since Linux precaches memory just doing a simple check doesn't help so I wrote this script:

Code:

#!/bin/ksh

# Determine memory usage percentage on Linux servers.
# Original write for RHEL3 for PC1 Project - jlightner 05-Jul-2005
#
# Modified for RHEL5 on mailservers.
# -Some of the escapes previously required for RHEL3's ksh not needed on RHEL5.
# -Changed comparisons to allow for decimal rather than integer values.
# jlightner 23-Jan-2009
#

# Usage:  check_mem.sh WARNING CRITICAL
#         Where WARNING and CRITICAL are the integer only portions of the
#         percentage for the level desired.
#         (i.e. 85% Warning & 95% Critical should be input only as "85 95".)

# Define Levels based on input
#
WARNLEVEL=$1
CRITLEVEL=$2

# Setup standard Nagios/NRPE return codes
#
UNKNOWN_STATE=3
CRITICAL_STATE=2
WARNING_STATE=1
OK_STATE=0

# Give full paths to commands - Nagios can't determine location otherwise
#
BC=/usr/bin/bc
GREP=/bin/grep
AWK=/bin/awk
FREE=/usr/bin/free
TAIL=/usr/bin/tail
HEAD=/usr/bin/head

# Get memory information from the "free" command - output of top two lines
# looks like:
#                 total       used       free     shared    buffers     cached
#    Mem:       8248768    6944444    1304324          0     246164    5647524
# The set command will get everything from the second line and put it into
# posiional variables $1 through $7.
#
set `$FREE |$HEAD -2 |$TAIL -1`

# Now give variable names to the positional variables we set above
#
MEMTOTAL=$2
MEMUSED=$3
MEMFREE=$4
MEMBUFFERS=$6
MEMCACHED=$7

# Do calculations based on what we got from free using the variables defined
#
REALMEMUSED=`echo $MEMUSED - $MEMBUFFERS - $MEMCACHED | $BC`
USEPCT=`echo "scale=3; $REALMEMUSED / $MEMTOTAL * 100" |$BC -l`
#USEPCT=`echo scale=3 "\n" $REALMEMUSED \/ $MEMTOTAL \* 100 |$BC -l |$AWK -F\. '{print $1}'`

# Compare the Used percentage to the Warning and Critical levels input at
# command line.  Issue message and set return code as appropriate for each
# level.  Nagios web page will use these to determine alarm level and message.
#
#if [ `echo "5.0 > 5" |bc` -eq 1 ]
#then echo it is greater
#else echo it is not greater
#fi
if [ `echo "$USEPCT > $CRITLEVEL" |bc` -eq 1 ]
then echo "CRITICAL - Memory usage is ${USEPCT}%"
     exit ${CRITICAL_STATE}
elif [ `echo "$USEPCT > $WARNLEVEL" |bc` -eq 1 ]
then echo "WARNING - Memory usage is ${USEPCT}%"
     exit ${WARNING_STATE}
elif [ `echo "$USEPCT < $WARNLEVEL" |bc` -eq 1 ]
then echo "OK - Memory usage is ${USEPCT}%"
     exit ${OK_STATE}
else echo "Unable to determine memory usage."
     exit ${UNKNOWN_STATE}
fi
echo "Unable to determine memory usage."
exit ${UNKNOWN_STATE}

Then in nrpe.cfg on each host I have something like:

Code:

# MEMORY Check
# check_mem <WARN%> <CRIT%> = MEMORY at defined warning and critical use %.
command[check_mem]=/usr/local/nagios/libexec/check_mem.sh 85 95

/usr/local/nagios/libexec is where I put the above script (named check_mem.sh).

On your Nagios master you would then need to modify service.cfg to include something like:

Code:

define service{
        use                     generic-service
        host_name               BILLYBOB
        service_description     # Memory Use Pct
        contact_groups          ux-admins, noc-op
        check_command           check_nrpe!check_mem
        }

The above assumes check_nrpe is defined in checkcommands.cfg.

mikeshn · 08-14-2009, 10:27 AM

check_mem.sh should be place on nagios and remote servers?

MensaWater · 08-14-2009, 10:36 AM

On the remote Linux server which runs NRPE plugin.

Nagios runs on the master but NRPE plugin is used on UNIX/Linux hosts to monitor those systems. Nsclient is used on Windows hosts to monitor those. Nagios must then be configured to interrogate those hosts with the appropriate commands.

mikeshn · 08-14-2009, 10:47 AM

It's working now. Do you use pnp4nagios?
I cannot generate graphs from the request.

Also, How I can monitor traffic on remote server (inbound/outbound)?

Thanks

MensaWater · 08-14-2009, 12:04 PM

I'm not using pnp4nagios but will likely be exploring it within the next few months. I'm in the middle of migrating from an older version of Nagios on FreeBSD to a newer one on RHEL.

I'm not sure what you mean by "monitor traffic". I assume you mean network traffic. If so you could check for a Nagios plugin that already does that or write one of your own similar to the one I did for memory.

You could start by examining the information provided under /proc/net (type "man proc" and search within that for /proc/net to get details of what can be found there). Or it may be details you want are in the ifconfig output.

Making your own scripts with Nagios is easy because all it really wants is certain exit statuses as seen in the script I posted earlier:
UNKNOWN_STATE=3
CRITICAL_STATE=2
WARNING_STATE=1
OK_STATE=0

It doesn't have to be ksh - it could be bash, perl, python or whatever so long as the end result is a single line of output and one of the above exit statuses (0, 1, 2 or 3). Nagios recognizes those statuses and will display appropriate value. The single line of ouput is also displayed. (Nagios won't display multiple lines for a single item.)

cristian_vidu · 01-05-2011, 11:01 AM

For anyone who finds the script useful (as I did) I added the option to output performance data and use it with PNP4Nagios for visual graphs.

The output is in MB but can be easily changed.

Thanks MensaWater for saving me the time to write my own script.

code:

Code:

#!/bin/ksh

# Determine memory usage percentage on Linux servers.
# Original write for RHEL3 for PC1 Project - jlightner 05-Jul-2005
#
# Modified for RHEL5 on mailservers.
# -Some of the escapes previously required for RHEL3's ksh not needed on RHEL5.
# -Changed comparisons to allow for decimal rather than integer values.
# jlightner 23-Jan-2009
#
# Modified to add support for PNP4Nagios
# - Added support for graphs, the performance data output is in MB 
#   as specified in the output
# cristian_vidu 2011-01-05

# Usage:  check_mem.sh WARNING CRITICAL
#         Where WARNING and CRITICAL are the integer only portions of the
#         percentage for the level desired.
#         (i.e. 85% Warning & 95% Critical should be input only as "85 95".)

# Define Levels based on input
#
WARNLEVEL=$1
CRITLEVEL=$2

# Setup standard Nagios/NRPE return codes
#
UNKNOWN_STATE=3
CRITICAL_STATE=2
WARNING_STATE=1
OK_STATE=0

# Give full paths to commands - Nagios can't determine location otherwise
#
BC=/usr/bin/bc
GREP=/bin/grep
AWK=/bin/awk
FREE=/usr/bin/free
TAIL=/usr/bin/tail
HEAD=/usr/bin/head

# Get memory information from the "free" command - output of top two lines
# looks like:
#                 total       used       free     shared    buffers     cached
#    Mem:       8248768    6944444    1304324          0     246164    5647524
# The set command will get everything from the second line and put it into
# posiional variables $1 through $7.
#
set `$FREE |$HEAD -2 |$TAIL -1`

# Now give variable names to the positional variables we set above
#
MEMTOTAL=$2
MEMUSED=$3
MEMFREE=$4
MEMBUFFERS=$6
MEMCACHED=$7

# Do calculations based on what we got from free using the variables defined
#
REALMEMUSED=`echo $MEMUSED - $MEMBUFFERS - $MEMCACHED | $BC`
USEPCT=`echo "scale=3; $REALMEMUSED / $MEMTOTAL * 100" |$BC -l`
#USEPCT=`echo scale=3 "\n" $REALMEMUSED \/ $MEMTOTAL \* 100 |$BC -l |$AWK -F\. '{print $1}'`

# Compare the Used percentage to the Warning and Critical levels input at
# command line.  Issue message and set return code as appropriate for each
# level.  Nagios web page will use these to determine alarm level and message.
#
#if [ `echo "5.0 > 5" |bc` -eq 1 ]
#then echo it is greater
#else echo it is not greater
#fi

# Support for PNP4Nagios. WARN/CRIT MEM is computed from WARN/CRIT LEVEL
#
WARNMEM=`echo "scale=3; $WARNLEVEL / 100 *$MEMTOTAL" |$BC -l`
CRITMEM=`echo "scale=3; $CRITLEVEL / 100 *$MEMTOTAL" |$BC -l`

# Modify output to MB instead of KB as reported by free
#
WARNMEMMB=`echo "scale=0; $WARNMEM / 1024" |$BC -l`
CRITMEMMB=`echo "scale=0; $CRITMEM / 1024" |$BC -l`
MEMTOTALMB=`echo "scale=0; $MEMTOTAL / 1024" |$BC -l`
REALMEMUSEDMB=`echo "scale=0; $REALMEMUSED / 1024" |$BC -l`

if [ `echo "$USEPCT > $CRITLEVEL" |bc` -eq 1 ]
then echo "CRITICAL - Memory usage is ${USEPCT}% |'Mem_Use'=${REALMEMUSEDMB}MB;${WARNMEMMB};${CRITMEMMB};0;${MEMTOTALMB}"
     exit ${CRITICAL_STATE}
elif [ `echo "$USEPCT > $WARNLEVEL" |bc` -eq 1 ]
then echo "WARNING - Memory usage is ${USEPCT}% |'Mem_Use'=${REALMEMUSEDMB}MB;${WARNMEMMB};${CRITMEMMB};0;${MEMTOTALMB}"
     exit ${WARNING_STATE}
elif [ `echo "$USEPCT < $WARNLEVEL" |bc` -eq 1 ]
then echo "OK - Memory usage is ${USEPCT}% |'Mem_Use'=${REALMEMUSEDMB}MB;${WARNMEMMB};${CRITMEMMB};0;${MEMTOTALMB}"
     exit ${OK_STATE}
else echo "Unable to determine memory usage. |'Mem_Use'=${REALMEMUSEDMB}KB;${WARNMEMMB};${CRITMEMMB};0;${MEMTOTALMB}"
     exit ${UNKNOWN_STATE}
fi
echo "Unable to determine memory usage."
exit ${UNKNOWN_STATE}

MensaWater · 01-06-2011, 08:11 AM

No problems. Thanks for the enhancement.

The_P · 08-03-2011, 07:53 AM

First, sorry for my bad english
I know this article is a little bit old but i have a problem with the script to get running with centos.
Running nrpe from my server (its an icinga server on centos) i get following error:

Code:

AWK=/bin/awk
BASH=/bin/bash
BASH_ARGC=([0]="2")
BASH_ARGV=([0]="90" [1]="50")
BASH_LINENO=([0]="0")
BASH_SOURCE=([0]="/usr/local/icinga/libexec/check_mem")
BASH_VERSINFO=([0]="3" [1]="2" [2]="25" [3]="1" [4]="release" [5]="x86_64-redhat-linux-gnu")
BASH_VERSION='3.2.25(1)-release'
BC=/usr/bin/bc
CRITICAL_STATE=2
CRITLEVEL=90
DIRSTACK=()
EUID=501
FREE=/usr/bin/free
GREP=/bin/grep
GROUPS=()
G_BROKEN_FILENAMES=1
HEAD=/usr/bin/head
HISTSIZE=1000
HOME=/root
HOSTNAME=someserver
HOSTTYPE=x86_64
IFS=$' \t\n'
INPUTRC=/etc/inputrc
LANG=en_US.UTF-8
LESSOPEN='|/usr/bin/lesspipe.sh %s'
LOGNAME=root
LS_COLORS='no=00:fi=00:di=00;34:ln=00;36:pi=40;33:so=00;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=00;32:*.cmd=00;32:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.cpio=00;31:*.jpg=00;35:*.gif=00;35:*.bmp=00;35:*.xbm=00;35:*.xpm=00;35

It doesent matter if i use root or icinga account i always get this error.
I reinstalled icinga (on both server and centos host).

Im also using the check_mem script on a proxmox server and some opensuse-vms and everything is working fine.
Has someone an idea what my problem could be?

The_P · 08-04-2011, 02:44 AM

Solved it.
If someoneelse got this kind of problem here is my resolution (i think the same way is also right for nagios)

First: Run the command with sudo. To let nrpe use sudo ad following line to /etc/sudoers on the centos systems:
icinga ALL=(ALL) NOPASSWD:/usr/local/icinga/libexec/check_mem.sh

Second: Also in /etc/sudoers on the centos system comment Defaults requiretty. This will prevent the NRPE: Unable to read output error on the icinga/nagios server.

Again.....sorry for my english