bash: do not rerun script that fails

kenneho · 02-16-2009, 10:23 AM

Hi.

I have a couple of scripts I'd like to put in a cron job. I'd like to make sure that scripts that fails doesn't get rerun before I've investigated the error. I believe that PID files will be needed here.

Basically, this is what I think I need:

If the script fails, send me an error containing the error message
Do not rerun the script until I've investigated the problem and manually reset the script somehow

Has anyone implemented such a "wrapper" script before, and would like to share some ideas on how to do this?

I'm quite a newbie when it comes to scripting, so please keep this in mind.

colucix · 02-16-2009, 10:57 AM

Just a couple of ideas: check the exit status of any command inside the script and if the status is not 0 just touch a file with a unique name (for example ${0}_abort, where $0 is expanded by the shell with the name of the script itself). Also embed all the code in a if condition like this:

Code:

#!/bin/bash
if [ ! -f /path/to/${0}_abort]
then
  <the whole script code here>
fi

In this way, the code will be execute if and only if the file ${0}_abort does not exist, that is only if the previous run was successful. In case of failure and manual reset, don't forget to delete the ${0}_abort file to let the script run again the next time.

Moreover, be sure to redirect the standard error of each command to a file, then at the end of the script you can check if that file is not empty and send an e-mail to your personal address:

Code:

if [ $(stat -c %s /path/to/errorfile) -ne 0 ]
then
  /usr/bin/sendmail your.address@domain.com <<-EOF
  Subject: Error in script $0
  $(cat /path/to/errorfile)
  EOF
fi

GazL · 02-16-2009, 11:44 AM

Code:

#!/bin/bash

mkdir -m 700 /var/tmp/scriptdir 2>/dev/null  ||  exit 1

echo "$(date) Starting run: PID=$$" >/var/tmp/scriptdir/run.log

#########################################################

# Include the real guts of the script here. 
#
# Make sure you 'exit' before cleanup if something goes 
# wrong.
#

#########################################################


## Success: Cleanup
rm -r /var/tmp/scriptdir
exit 0

That's the way I tend to do this sort of thing. Just replace scriptdir with a unique name for each of the scripts you want to control this way.

Basically, if the directory on the mkdir already exists, then a prior run either hasn't completed yet, or completed but not successfully, so we exit before doing anything.

Not only does this protect you from a failed run, it also protects against accidentally running your script multiple times concurrently owing to an over-running or hung run. The directory also makes a convenient place to dump error logs or other run control files that you may want to examine in the event of a failure, but don't really care about upon a successful completion.

Hope that gives you some ideas.

kenneho · 02-17-2009, 06:20 AM

Thank you both for your help. I've managed to implement a solution based on your feedback. Thanks again.