[Not for all as its just a messy type of temporary work] This script was made for an operator who was facing strange problem that in random timings , the mysql or some times radiusd service stopped automatically, unfortunately the error was not sorted out as it occurs in awkward timings when no one is at NOC, so I made an workaround in the form of this script so that we can diagnose the issue later in details and in the meanwhile manually startup of services should not be required. The script was scheduled to run after every 10 minutes.
Note: This is not a solution, Just a workaround for very small period of time.
This script will check one or more services status, if found stopped, it will try to re-start them, and then check there status again and output there status in email.
In this example I took MYSQL and RADIUS services.
Sections:
- Create Required Scripts
- Download sendEmail tool to send email alerts
- Check Service status
- Another version of script [spicework] with prevention of repeated email in case of scheduler
1- Create Required Scripts
Create New script checkstatus.sh which will be check the required services status and will output there result in text file which will be later used to send email (only if services found stopped)
mkdir /temp touch /temp/checkstatus.sh chmod +x /temp/checkstatus.sh nano touch /temp/checkstatus.sh
Now add following contents in it
#!/bin/bash SRV1="radiusd" SRV2="mysqld" SRV2D="mysql" pid1=`pidof $SRV1` pid2=`pidof $SRV2` CURDATE=`date` # Delete existing mail file rm -fr /temp/radiusstatus # Check for SRV1 - RADIUSD - If not running then add its entry to local file for mail if [ "$pid1" == "" ]; then echo -e "$SRV1 service is NOT running, Trying to start . . ." echo -e "$CURDATE $SRV1 service is NOT running, Trying to start . . ." >> /temp/radiusstatus var1=$SRV1 service $SRV1 restart # If SRV1 is running then Ignore and check for next service else echo $SRV1 service is Running OK , no further action required, EXITING. Script by Syed Jahanzaib fi # Check for SRV2 - MYSQL - If not running then add its entry to local file for mail if [ "$pid2" == "" ]; then echo -e "$SRV2D service is NOT running, Trying to start . . ." echo -e "$CURDATE $SRV2D service is NOT running, Trying to start . . ." >> /temp/radiusstatus var2=$SRV2 service $SRV2D restart # If SRV2 is running then Ignore and check for next service else echo $SRV2 service is Running OK , no further action required, EXITING. Script by Syed Jahanzaib fi # Emal the results to admin if file exists if [ ! -f /temp/radiusstatus ] then echo -e "$SRV1 and $SRV2D Services seems to be running ..." else echo -e "ALERT ... Services seems to be stopped, check email " # Check Services status after starting them by calling external script echo -e "=========================================" >> /temp/radiusstatus echo -e "Current Status After running this script" >> /temp/radiusstatus /temp/checkservice.sh $SRV1 >> /temp/radiusstatus /temp/checkservice.sh $SRV2 >> /temp/radiusstatus #Finally send email with all the data gathered /temp/sendEmail-v1.56/sendEmail -t aacable@hotmail.com -u "$CURDATE / Billing Alert: $var1 $var2 Services found stopped and restarted now" -o tls=yes -s smtp.gmail.com:587 -xu YOURGMAIL_ID@gmail.com -xp YOURGMAILPASSWORD -f YOUTGMAIL_ID@gmail.com -o message-file=/temp/radiusstatus -o message-content-type=text fi
Save and Exit.
Now create another script checkservice.sh which will be called by above parent script to get the final status of services.
touch /temp/checkservice.sh chmod +x /temp/checkservice.sh nano /temp/checkservice.sh Now add following contents in it
#!/bin/bash PID=`pidof $1` if [ `pgrep $1` ] then echo "$1 = OK with pid $PID" else echo "$1 = NOT RUNNING" fi
Save & Exit.
2- Download sendEmail tool to send email alerts
Install sendEmail Tool
wget http://caspian.dotconf.net/menu/Software/SendEmail/sendEmail-v1.56.tar.gz tar zxvf sendEmail-v1.56.tar.gz cd sendEmail-v1.56/
ADD SUPPORTING LIBRARY
for ubuntu
apt-get -y install libio-socket-ssl-perl libnet-ssleay-perl perl
for centos
yum -y install perl perl-Crypt-SSLeay perl-IO-Socket-SSL
TESTING SEND EMAIL
Try to send email using command line: Example
/temp/sendEmail-v1.56/sendEmail -t YOURMAIL@hotmail.com -u "Test Email" -s smtp.gmail.com:587 -xu YOURMGAILID@gmail.com -xp YOURGMAILPASSWORD -f YOURMGAILIDgmail.com -o tls=yes
if you get message something like “sendEmail[xxxx]: Email was sent successfully!”, then you are good to GO LIVE !
3- Check Service status
Now run the script like (make sure mysql and radiusd services are running)
/temp/checkstatus.sh
and you will see following result if servies are already running
and if any service found to be stopped, it will try to start it and will send you the result.
(If one service found stopped it will only try to start that service, if both services found stopped it will try to start them both)
now for test stop the mysql or radiusd service and run the script again, this time you will see the result as follows
and at email you will see the following result
You can then schedule the service to run after every xx minutes in CRON
crontab -l
# Run after every 10 minutes , Check for stopped services of mysql and radius */10 * * * * /temp/checksrv.sh
4- Another version of script [spicework] with prevention of repeated email in case of scheduler
#!/bin/bash if pgrep "mysql" > /dev/null then echo "MYSQL Running" rm -f /var/run/.mysql_mail_sent else echo "mysqld ALERT Stopped" if [ ! -f /var/run/.mysql_mail_sent ]; then echo -e "Send your mysql mail here" date > /var/run/.mysql_mail_sent fi fi if pgrep "radiusd" > /dev/null then echo "radiusd Running" rm -f /var/run/.radiusd_mail_sent else echo "radiusd ALERT Stopped" if [ ! -f /var/run/.radiusd_mail_sent ]; then echo -e "Send your radiusd mail here" date > /var/run/.radiusd_mail_sent fi fi
To DO Task:
Add check to prevent re-sending of email alerts if service have permanent failure or cannot be started in any case, the script will keep sending the email :D , so a check is required. Will add it later in some free time.
Regard’s
Syed Jahanzaib
Filed under: Linux Related
