Scenario:
We have Ubuntu 12.x installed and serial modem is attached (Teltonika G10). Kannel is installed and serving as SMS gateway.
Problem:
Sometimes modem stops responding and when we restart the modem, kannel starts giving error of Routing Failed. After restart of KANNEL service, the kannel starts responding to sms request and works fine.
Requirement:
This little issue can be really annoyed if you are doing remote administration OR if sms alerts are important for you to keep informed about various aspects of the network. We want some automatic mechanism that can detect this specific error and then act accordingly.
Example …
- If KANNEL service is down , try to start it
- If failed to start the service, then print error, and sends email about incident, (one email per incident), and exit script
- If service starts successfully, then print info, and sends email about successful attempt. (one email per incident), and continue further
- Test Kannel log last entries, and look for word Routing failed, if found, then restart kannel service and email. (one email per incident).
- If it does not found any entry in LOG for specific word, then Print OK Result for all.
Solution!
Disclaimer:
This may or may not work for you. Because I made it for some very particular situation, and this script helped. So this script checks for two things only. 1st kannel service, 2nd kannel log error for routing failed. I am sure this may or may not help you in same situation because its possible that error can be due to something else. I carefully examined my situation and made script for it. You can get some idea by it and modify the script as required.
Schedule Script to run after every 15 minutes or whatever.
Cron Example
# Run KANNEL MONITOR Script after every 15 minutes */15 * * * * /temp/kanneltest.sh
the SCript !
mkdir /temp cd /temp touch /temp/kanneltest.sh chmod +x /temp/kanneltest.sh nano /temp/kanneltest.sh
Copy paste following script. Make sure you read it carefully and modify things accordingly.
#!/bin/bash # No part of this script is copied from anywhere, you are free to use it, modify or distribute. # Linux BASH script to check KANNEL service Status and internal component system (example kannel service is ok,but not responding to request giving error in logs # If KANNEL not responding, then send EMail alert to admin (trigger one time for each status changed) # Useful for remote admins, who want to be informed when KANNEL have any responding problem. # Created: 9th-JUNE-2016 # Syed Jahanzaib # aacable at hotmail dot com / https://aacable.wordpress.com #set -x # Colors Config . . . [[ JZ . . . ]] ESC_SEQ="\x1b[" COL_RESET=$ESC_SEQ"39;49;00m" COL_RED=$ESC_SEQ"31;01m" COL_GREEN=$ESC_SEQ"32;01m" # Service name to monitor ... KANNEL="kannel" KANNEL_SERVICE_ERROR="kannel_main_service_down" KANNEL_BEARERBOX="bearerbox" # Text that will be checked in kannel logs output file, # So we will simply use the grep to catch word **Routing failed**. You may use other technique that may work for you. Chill / zaib TEXT_TO_CATCH="Routing failed" # Hostname and other Variables HOSTNAME=`hostname` COMPANY="zaib (Pvt) Ltd." FOOTER="Powered By Syed.Jahanzaib" DATE=`date` # Temporary file holder for KANNEL status KANNEL_STATUS_TMP_HOLDER="/tmp/KANNELstatus.txt" KANNEL_SERVICE_TMP_HOLDER="/tmp/kannelservice.txt" KANNEL_SERVICE_TMP_HOLDER_ERR="/tmp/kannel_servic_error.txt" KANNEL_QUERY_RESULT="/tmp/KANNEL_query_result.txt" KANNEL_LOG_FILE="/var/log/kannel/bearerbox.log" # Create temp file if not already present, usually for 1st time execution touch $KANNEL_STATUS_TMP_HOLDER touch $KANNEL_SERVICE_TMP_HOLDER touch $KANNEL_SERVICE_TMP_HOLDER_ERR touch $KANNEL_QUERY_RESULT # EMAIL RELATED and KANNEL INFO # for down status, we have to use GMAIL to send email KANNELURL="127.0.0.1:13013" KANNELID="kannel" KANNELPASS="KANNELPASS" CELL1="03333021909" # GMAIL Section GMAILID="YOURGMAILID@gmail.com" GMAILPASS="YOURGMAILPASS" ADMINMAIL1="aacableAThotmail.com" ########################################################################################### # Testing KANNEL Service bearerbox status by its PID echo -e "$COL_RED Step 1:$COL_RESET INFO: Testing $KANNEL Service status , testing its PID ..." PID=`pgrep $KANNEL_BEARERBOX` if [ -n "$PID" ]; then echo -e "$KANNEL Service Status = $COL_GREEN OK $COL_RESET with pid $PID" sed -i "/$KANNEL_SERVICE_ERROR/d" "$KANNEL_SERVICE_TMP_HOLDER" else echo -e "$COL_RED$KANNEL Service = NOT RUNNING, trying to restarting it ...$COL_RESET" service $KANNEL stop > /dev/null 2>&1 sleep 5 killall -9 $KANNEL_BEARERBOX > /dev/null 2>&1 sleep 2 service $KANNEL start > /dev/null 2>&1 sleep 5 # IF KANNEL MAIN SERVICE found DOWN, and email/SMS not already sent, then send it one time to $CELL1/x PID=`pgrep $KANNEL_BEARERBOX` if [ -z "$PID" ]; then KSRVAFTRES="DOWN" echo -e "Script tried to restart $KANNEL MAIN SERVICE and final status is = $COL_RED $KSRVAFTRES $COL_RESET..." else KSRVAFTRES="UP/OK" echo -e "Script tried to restart $KANNEL MAIN SERVICE and final status is = $COL_GREEN $KSRVAFTRES .$COL_RESET..." if [ $(grep -c "$KANNEL_SERVICE_ERROR" "$KANNEL_SERVICE_TMP_HOLDER") -eq 0 ]; then echo -e "$COL_RED Sending SMS/EMAIL for KANNEL MAIN SERVICE...$COL_RESET" PID=`pgrep $KANNEL_BEARERBOX` if [ -n "$PID" ]; then KSRVSTATUS="UP/OK" echo -e "$KANNEL sevice was not running, After script attempt to restart it, its status is $COL_GREEN $KSRVSTATUS $COL_RESET with PID $PID" KSRVUPMSG="/tmp/ksrvup.msg" echo "$KANNEL sevice was not running, After script attempt to restart it, its status is $KSRVSTATUS with PID $PID" > $KSRVUPMSG /temp/sendEmail-v1.56/sendEmail -u "ALERT: $KANNEL service was not working and finally its $KSRVSTATUS @ $DATE" -o tls=yes -s smtp.gmail.com:587 -t $ADMINMAIL1 -xu $GMAILID -xp $GMAILPASS -f $GMAILID -o message-file=$KSRVUPMSG -o message-content-type=text else KSRVSTATUS="DOWN" echo -e "After kannel restart attempt, its status is still $COL_RED $KSRVSTATUS, $COL_RESET... SMS/EMAIL Already sent ..." #Updating its down log, so script should not repeat sms/email sending echo "$KANNEL_SERVICE_ERROR" > $KANNEL_SERVICE_TMP_HOLDER fi fi fi fi ########################################################################################### # Testing again if above step have started the service or not , if not EXIT the script with the error # Because if kannel is not running, then checking modem is useless, kannel must be running first in order to proceed further PID=`pgrep $KANNEL_BEARERBOX` if [ -n "$PID" ] then echo "" else echo -e "$COL_REDALERT ALERT: $KANNEL service failed to respond on Service restart request .... please check$COL_RESET $COL_RED Script cannot continue, exiting ...$COL_RESET" #exit 0 fi ############## zaib final down PID=`pgrep $KANNEL_BEARERBOX` if [ -z "$PID" ]; then if [ $(grep -c "$KANNEL_SERVICE_ERROR" "$KANNEL_SERVICE_TMP_HOLDER_ERR") -eq 0 ]; then KSRVSTATUS="DOWN" KSRVDOWNMSG="/tmp/ksrvdown.msg" echo "After kannel restart attempt by the script, its status is still $KSRVSTATUS. You need to check it manualy. Now only Human can see what is going on ... " echo "After kannel restart attempt by the script, its status is still $KSRVSTATUS. You need to check it manualy. Now only Human can see what is going on ... " > $KSRVDOWNMSG /temp/sendEmail-v1.56/sendEmail -u "ALERT: $KANNEL service failed to restart. Status = $KSRVSTATUS @ $DATE / Check it manualy!" -o tls=yes -s smtp.gmail.com:587 -t $ADMINMAIL1 -xu $GMAILID -xp $GMAILPASS -f $GMAILID -o message-file=$KSRVDOWNMSG -o message-content-type=text #Updating its down log, so script should not repeat sms/email sending echo "$KANNEL_SERVICE_ERROR" > $KANNEL_SERVICE_TMP_HOLDER_ERR exit 0 fi else sed -i "/$KANNEL_SERVICE_ERROR/d" "$KANNEL_SERVICE_TMP_HOLDER_ERR" fi ########################################################################################### PID=`pgrep $KANNEL_BEARERBOX` if [ -n "$PID" ] then echo "" else echo -e "$COL_REDALERT ALERT: $KANNEL service failed to respond on Service restart request .... please check$COL_RESET $COL_RED Script cannot continue, exiting ...$COL_RESET" exit 0 fi ########################################################################################### # Run the script to test kannel internal processing ... echo -e "$COL_RED Step 2:$COL_RESET Testing Kannel internal Service Status in LOGS for internal responding @DATE ..." for KANNEL in $KANNEL do # Match $TEXT_TO_CATCH in the log file grep query result STATUS=`tail -n 10 $KANNEL_LOG_FILE | grep "$TEXT_TO_CATCH"` if [ -n "$STATUS" ]; then # Print Result for information purposes echo "ALERT: text ** $TEXT_TO_CATCH ** found in $KANNEL_QUERY_RESULT" echo -e "$COL_RED $KANNEL service is UP but internal system is not responding to SMS ... Trying to restarting $KANNEL service ...$COL_RESET" # IF KANNEL found DOWN, and email/SMS not already sent, then send it one time to $CELL1/x if [ $(grep -c "$KANNEL" "$KANNEL_STATUS_TMP_HOLDER") -eq 0 ]; then # Restart the KANNEL service service kannel stop > /dev/null 2>&1 sleep 5 killall -9 bearerbox > /dev/null 2>&1 sleep 2 service kannel start > /dev/null 2>&1 sleep 5 echo -e "$COL_RED $KANNEL service is UP but internal system is not responding $DATE .. SENDING DOWN SMS/EMAIL for current incident / for one time only ...$COL_RESET" # Update KANNEL down status in a file so that sms/email should not be sent again and again echo "$KANNEL" > $KANNEL_STATUS_TMP_HOLDER # Sending DOWN SMS via KANNEL echo -e "$COL_REDSending SMS , if KANNEL SERVICE is down or internal system not responding, SMS will NOT be sent ...$COL_RESET " # Update down message to be sent via sms/email echo "$MSG_DOWN" > $MSGDOWNHOLDER # sen email using sendemail tool cat $MSGDOWNHOLDER | curl "http://$KANNELURL/cgi-bin/sendsms?username=$KANNELID&password=$KANNELPASS&to=$CELL1" -G --data-urlencode text@- ######## EMAIL SECTION ############## # Make sure you install sendEMAIL tool and test it properly before using email section. #SEND EMAIL Alert As well using sendEMAIL tool using GMAIL ADDRESS. # If you want to send email , use below ... echo -e "$COL_REDSending EMAIL ALERT to $ADMINMAIL1 ...$COL_RESET " /temp/sendEmail-v1.56/sendEmail -u "ALERT: $KANNEL not responding @ $DATE" -o tls=yes -s smtp.gmail.com:587 -t $ADMINMAIL1 -xu $GMAILID -xp $GMAILPASS -f $GMAILID -o message-file=$MSGDOWNHOLDER -o message-content-type=text fi ########################################################################################### # Else if $TEXT_TO_CATCH does not found in $KANNEL_QUERY_RESULT, then consider KANNEL is UP else echo -e "$COL_GREEN INFO:$COL_RESET Word ** $TEXT_TO_CATCH ** was NOT found in $KANNEL_LOG_FILE $COL_RED FINAL RESULT:$COL_RESET $KANNEL service = $COL_GREEN OK $COL_RESET $KANNEL iternal system = $COL_GREEN OK $COL_RESET / All seems to be responding OK! " # If KANNEL found UP and last status was DOWN, then send UP msg one time if [ $(grep -c "$KANNEL" "$KANNEL_STATUS_TMP_HOLDER") -eq 1 ]; then echo "$KANNEL internal system is now responding at $(date)... SENDING UP SMS for current incident one time only ... " echo "$MSG_UP" > $MSGUPHOLDER # Sending UP SMS via KANNEL echo -e "Sending $COL_GREENUP$COL_RESET SMS , if KANNEL process is down, sms will not be sent ... " cat $MSGUPHOLDER | curl "http://$KANNELURL/cgi-bin/sendsms?username=$KANNELID&password=$KANNELPASS&to=$CELL1" -G --data-urlencode text@- ############## EMAIL SECTION ############## # Make sure you install sendEMAIL tool and test it properly before using email section. #SEND EMAIL Alert As well using sendEMAIL tool using GMAIL ADDRESS. # If you want to send email , use below ... echo "Sending SEMAIL UP/OK to $ADMINMAIL1 & $ADMINMAIL2 ... " /temp/sendEmail-v1.56/sendEmail -u "INFO: $KANNEL internal system is now responding OK @ $DATE" -o tls=yes -s smtp.gmail.com:587 -t $ADMINMAIL1 -xu $GMAILID -xp $GMAILPASS -f $GMAILID -o message-file=$MSGDOWNHOLDER -o message-content-type=text # Update KANNEL UP status in a file so that sms/email should not be sent again and again sed -i "/$KANNEL/d" "$KANNEL_STATUS_TMP_HOLDER" fi fi done # Script Ends Here ... # z@iB
Script Results:
Regard’s
Syed Jahanzaib
Filed under: Linux Related
