Email notification of system reboot

benr77 · 09-27-2004, 12:31 PM

Hi,

I'm running a Fedora Core 2 system, and want to set up the system to send me an email whenever the system goes down for a reboot or a power off.

So, I've tried creating a script containing:

Code:

echo "Host is going down for a reboot NOW" | mail -s "Host going for a reboot at `date`" me@domain.com

I saved this script as /etc/init.d/RebootAlertEmail

Then I created a symlink to this script as /etc/rc6.d/S00RebootEmailAlert so that this script would be triggered anytime the system enters runlevel 6 for a reboot.

This seems to work ok, except that what happens is the email doesn't actually get sent until the system comes back online after the reboot (if at all in the event of a problem). Obviously I don't want to receive the email once the system is operational again, I need it as soon as the machine knows its going down.

Does anyone know where I'm going wrong, or can point out a better otr simpler solution? This must be a common requirement for monitoring a server machine surely?

Thanks

meblost · 09-27-2004, 12:47 PM

The way I understand it is that scripts with an 'S' prefix are run when the system is booting and scripts with a 'K' prefix are run when the system shuts down. Try giving your script a 'K' prefix.

benr77 · 09-28-2004, 02:36 AM

Is it not that S scripts are executed when the system enters that particular runlevel, and K scripts executed when the system leaves that runlevel ??

So if I'm correct my S00 script should be executed when the system enters runlevel 6 whilst going for a reboot. As far as I can see this is happening correctly.

What is not happening is the email send. The email seems to be created at the correct time by the script, but not sent. Maybe sendmail queues all jobs it has if it's in runlevel 0 or 6 ??

meblost · 09-28-2004, 10:14 AM

I was wrong. But after reading /etc/rc.d/rc, it looks like kill scripts are run first, then start scripts. Maybe something vital to sendmail is stopped before the email is sent. Or maybe you are right, maybe sendmail queues all jobs in 6. Check out /etc/rc.d/rc.

benr77 · 09-28-2004, 11:24 AM

Yes, I've now tried switching it to a kill script K02RebootEmailAlert but it's not working at all now.... I might have to look into a different solution as this doesn't seem like the most reliable approach...

Is there any other way to achieve this email send?

meblost · 09-28-2004, 09:47 PM

If you use bash, you could try creating an alias in /etc/bashrc or where ever your system wide alias' go:

alias reboot='/path/to/RebootAlertEmail; reboot'
alias halt='/path/to/RebootAlertEmail; halt'

Or you could change the line in /etc/inittab that controls what action is taken when init is asked to change to runlevel 6 from
l6:6:wait:/etc/rc.d/rc 6 to
l6:6:wait:'/path/to/RebootAlertEmail;/etc/rc.d/rc 6'.

Not sure how smart that is though. Changing inittab is uncharted territory for me. That would be a bad file to mess up. Make sure you know what you're doing first.

SciYro · 09-28-2004, 10:04 PM

i think that by the time the script is executed, the Internet has already been shut down, leaving sendmial (or whatever) to either scrap the message, or send it when the Internet becomes operational again (O, say when it boots up)

anyways, i dint know how red hat/mandrake order its run-level scripts, so i wouldn't know how to get it to work with them ... but one thing you could do is create a separate script, and tell sysvinit (/etc/init.conf) that when it enter the run level, to execute your script, and not the mandrake default, your script will first send the email, then execute mandrakes shutdown script just as it normally would (copy the execution command for run level 6 from /etc/init/conf before you change it)

benr77 · 09-29-2004, 09:02 AM

Yes, I've been looking in /etc/inittab and don't really want to mess around inside that file too much !!

Using an alias is not suitable, as I want to be informed by email if the machine reboots from a power failure or some other cause, as well as by any shutdown statement issued by a user.

Maybe it would be simpler just to skip this idea and install some monitoring software on a separate machine??

twantrd · 09-29-2004, 03:11 PM

I dont know if FC has this but on Redhat AS in /etc/rc6.d I see "S01reboot". Try inserting your 'echo' command after the commented statements (right before it even calls the runcmd function) in that script. I would try this myself but this is on a production box and so I can't

.

-twantrd

benr77 · 10-01-2004, 06:14 AM

twantrd - thanks - I just tried this and its giving the same results as my previous attempts - essentially sending the email ok, but the email doesn't actually get sent until the machine comes back online...

I suppose this functionality will have to do for the time being - it won't let me know when the machine is halted, which is what I wanted, but I can make do with alerts on reboots for the moment.

Thanks for everybody's input.

atom · 10-01-2004, 07:09 AM

I found that if something is not done when it should, it's usualy because the program did not have the time.

For some reason, httpd --restart did not work for me. I had to do httpd --stop;sleep 1; httpd --start;

Maybe, you should do the same. Give the program 3 seconds to send the mail and clean up. Maybe with a for loop like this:

Code:

echo -e "\n-------------------------------------------------------\n\nWaiting for sendmail:"
for ((i=5; i >= 0; i--))
do
     echo "$i seconds..."
     sleep 1
done
echo -e "Continuing reboot... \n\n-------------------------------------------------------"

doc.nice · 10-17-2004, 05:47 PM

I had the same problem some time ago, i think the problem was that exim or whatever MTA you use dooesn't deliver mails immediately but in special intervals. To force the MTA to (try to) deliver immediately, you can run 'runq' (exim -q). On my system (Debian with 2.4.27 kernel), i use the following script linked in those runlevels and all works great:

Code:

/etc/rcS.d/S42mailnotify (must be after S??networking, so the LAN is up and ready)
/etc/rc6.d/K01mailnotify (must be before K??networking, so the LAN is still up)

(By the way, to wipe out the confusion about K and S above:
K means, the script is called with $1=stop and S means $1=stop,
The scripts are all executed when entering a runlevel.
In many distros, there's a heuristics to execute only those scripts
which indeed change on the change from the old runlevel to the new one.)

So here's the script (/etc/init.d/mailnotify):

Code:

#! /bin/sh
# mailnotify    sends mail on system startup/shutdown

export PATH=/usr/sbin:$PATH

case "$1" in
    start)
      echo -n"sending mail to sysadmin..."
      echo "Host is getting up at $(date)" | mail -s "Host getting up at `date`" someone@home.net
      sleep 2
      runq
      sleep 3
      echo "OK"
        ;;
    stop)
      echo -n "sending mail to sysadmin..."
      echo "Host is going down for a reboot at $(date)" | mail -s "Host going fo
r a reboot at `date`" someone@home.net
      sleep 2
      runq
      sleep 3
      echo "OK"
        ;;
    *)
        echo "Usage: $0 {start|stop}"
        exit 1
        ;;
esac

exit 0

benr77 · 10-18-2004, 04:23 PM

Thanks doc. I've just tried this script and it works fine for giving me the email on system bootup, but the email for when the system is going down is not being sent. It's not even sending it and then failing to flush the queue.

I'm using postfix so used 'postqueue -f' rather than 'runq', but the error I'm getting in the logs is:

postfix/postqueue[960]: fatal: Cannot flush mail queue - mail system is down

I have the script running before the network or postfix are taken down, so it should be fine. Any ideas ??

Also, can you explain more about what you mean by "In many distros, there's a heuristics to execute only those scripts which indeed change on the change from the old runlevel to the new one" etc

Thanks

doc.nice · 10-18-2004, 06:34 PM

Well, concerning the postfix problem I can't help you at the moment, sorry.
Check your syslog for more information, maybe there's some more specific information
about what isn't running any more...
or, try searching the web

http://archives.neohapsis.com/archiv...hread.html#436
http://www.linux-club.de/viewtopic.php?p=23413#23413

About the runlevels and sys-v you can read the debian-policy, also I can show you some excerpt from the debian runlevel control script, '/etc/init.d/rc'

Code:

#[snip...snap]

#               Optimization feature:
#               A startup script is _not_ run when the service was
#               running in the previous runlevel and it wasn't stopped
#               in the runlevel transition (most Debian services don't
#               have K?? links in rc{1,2,3,4,5} )

#[snip...snap] (function startup() basically runs its parameters (optionally with a debug flag))

  # Is there an rc directory for this new runlevel?
  if [ -d /etc/rc$runlevel.d ]
  then
        # First, run the KILL scripts.
        if [ $previous != N ]
        then
                for i in /etc/rc$runlevel.d/K[0-9][0-9]*
                do
                        # Check if the script is there.
                        [ ! -f $i ] && continue

                        # Stop the service.
                        startup $i stop
                done
        fi
        # Now run the START scripts for this runlevel.
        for i in /etc/rc$runlevel.d/S*
        do
                [ ! -f $i ] && continue

                if [ $previous != N ]
                then
                        #
                        # Find start script in previous runlevel and
                        # stop script in this runlevel.
                        #
                        suffix=${i#/etc/rc$runlevel.d/S[0-9][0-9]}
                        stop=/etc/rc$runlevel.d/K[0-9][0-9]$suffix
                        previous_start=/etc/rc$previous.d/S[0-9][0-9]$suffix
                        #
                        # If there is a start script in the previous level
                        # and _no_ stop script in this level, we don't
                        # have to re-start the service.
                        #
                        [ -f $previous_start ] && [ ! -f $stop ] && continue
                fi
                case "$runlevel" in
                        0|6)
                                startup $i stop
                                ;;
                        *)
                                startup $i start
                                ;;
                esac
        done
  fi

IIRC, SuSe has a even more complicated version, but I only have this one at hand...

HTH, Flo