Disclaimer: This post is shared just for reference & learning purposes.
Scenario:
We are using Freeradius server which uses mySQL as its backend DB. Ideally the mysql server should have replica server so that if Primary goes down dueto any fault, the secondary replica should come in action.
For high availability purposes we we want to have a standby server. Mysql Master-Slave or Master-Master replication is ideal for real time replication. We successfully implemented this model at few sites, but yes replication requires constant monitoring, and at one place the secondary replica server backfired & caused data loss.
For one particular Remote Site we wanted to avoid the complications of REPLICATION. What we wanted is a standby server, and the DB from primary should be exported to secondary replica server daily in morning and emails for the actions taken by the script should be emailed to us.
We made custom script that is running successfully from quite some time.
The BASH script performs following function …
- Checks secondary server PING response
- Check secondary server SSH access
- Checks primary server MYSQL DB access
- Checks secondary server MYSQL DB access
- Check if exported DB is of valid size, (I set it to min 10 KB, yes you may want to adjust it according to your setup)
- If all OK, then export primary server DB, and import it to secondary server
Script Requirements:
- Sendemail tool to send email alerts/info
- passwordless login to secondary server (using SSH keys)
BASH Script Code:
- touch /temp/update_radius_from_10.0.0.1__TO__192.168.0.1.sh
- chmod +x /temp/update_radius_from_10.0.0.1__TO__192.168.0.1.sh
- nano /temp/update_radius_from_10.0.0.1__TO__192.168.0.1.sh
#!/bin/bash
#set -x
# Version 1.0 / 10-July-2019
# Syed Jahanzaib / Web: https://aacable.wordpress.com / Email: aacable@hotmail.com
# This script exports mysqldb and restores it to second remote server
# Requires passwordless login on remote server using SSH keys
# Settings various VARIABLES for the script
# adding dns for resolving
echo "nameserver 8.8.8.8" > /etc/resolv.conf
#SET DATE TIME
set $(date)
time=`date |awk '{print $4}'`
YESTERDAY=`date --date='yesterday' +%Y-%m-%d`
IP1=10.0.0.1
IP2=192.168.0.1
IP2ROLE="RADIUS"
IP2_SSH_PORT=22
SQL_DIR="sql_replica"
#MYSQL DETAILS
SQLUSER="root"
SQLPASS="MYPASSWORD"
export MYSQL_PWD=$SQLPASS
CMD="mysql -u$SQLUSER --skip-column-names -s -e"
DB="radius"
FILE="/$SQL_DIR/$YESTERDAY.ip.$IP1.sql"
GMAILID="MYGMAILID@gmail.com"
GMAILPASS="PASSWD"
ADMINMAIL1="ADMINMAIL1@hotmail.com"
COMPANY="zaib (Pvt) Ltd."
RESULT="/tmp/$IP2.$IP2ROLE.txt"
PING_ATTEMPTS="2"
PING_RESULT="/tmp/$IP2.$IP2ROLE.ping.result.txt"
IP2_SSH_CHK="/tmp/$IP2.ssh.chk.txt"
touch $RESULT
touch $PING_RESULT
> $RESULT
> $PING_RESULT
rm /$SQL_DIR/*.sql
# Test PING to device
count=$(ping -c $PING_ATTEMPTS $IP2 | awk -F, '/received/{print $2*1}')
if [ $count -eq 0 ]; then
echo "- $COMPANY ALERT: $IP2 - $IP2ROLE is not responding to PING Attempts, cannot continue without it , Please check !"
echo "- $COMPANY ALERT: $IP2 - $IP2ROLE is not responding to PING Attempts, cannot continue without it , Please check !" > $PING_RESULT
sendemail -t $email -u "ALERT: $IP2 $IPROLE NOT RESPONDING!" -o tls=yes -s smtp.gmail.com:587 -t $ADMINMAIL1 -xu $GMAILID -xp $GMAILPASS -f $GMAILID -o message-file=$PING_RESULT -o message-content-type=text
exit 1
fi
echo "- PING Result to $IP2 : OK"
echo "- PING Result to $IP2 : OK" >> $RESULT
#Cehck if SSH is accessible
scp -q -P $IP2_SSH_PORT root@$IP2:/etc/lsb-release $IP2_SSH_CHK
# Verify if file is downloaded from remote server via ssh
if [ ! -f $IP2_SSH_CHK ]; then
echo -e "- $COMPANY ALERT: $IP2 - $IP2ROLE is not responding to SSH ACCESS, cannot continue without it , Please check !"
exit 1
fi
echo -e "- SSH Access to $IP2 Result: OK"
echo -e "- SSH Access to $IP2 Result: OK" >> $RESULT
# Check if $DB (in this case radius )is accessible or not, if NOT, then exit the script
RESULT_DB_CHK=`$CMD "SHOW DATABASES LIKE '$DB'"`
if [ "$RESULT_DB_CHK" != "$DB" ]; then
echo "- ALERT: $IP2 - DB $DB not accessible."
echo "- ALERT: $IP2 - DB $DB not accessible." >> $RESULT
sendemail -t $email -u "- ALERT: $IP2 - DB $DB not accessible" -o tls=yes -t $ADMINMAIL1 -xu $GMAILID -xp $GMAILPASS -f $GMAILID -o message-file=$RESULT -o message-content-type=text
exit 1
fi
echo "- $DB - Database accessed OK" >> $RESULT
#############################################
######## START the BACKUP PROCESS ... #######
#############################################
# Checking if $SQL_DIR folder is previously present or not . . .
{
if [ ! -d "/$SQL_DIR" ]; then
echo -e "- ALERT: /$SQL_DIR folder not found, Creating it MYSQL EXPORT/DUMP backup should be placed there . . ."
mkdir /$SQL_DIR
else
echo -e "- INFO: $SQL_DIR folder is already present , so no need to create it, Proceeding further . . ."
fi
}
mysqldump -u$SQLUSER -p$SQLPASS --ignore-table={radius.radacct} $DB > $FILE
# CHECK FILE SIZE AND COMPARE, IF ITS LESS , THEN ALERT
SIZE=`ls -lh $FILE | awk '{print $5}'`
SIZEB=`ls -l $FILE | awk '{print $5}'`
if [ $SIZEB -lt 1 ]
then
echo "- ALERT: DMA REPLICA failed on $IP1 - Size = $SIZE OR $SIZEB Bytes"
echo "- ALERT: DMA REPLICA failed on $IP1 - Size = $SIZE OR $SIZEB Bytes" >> $RESULT
sendemail -t $email -u "DMA REPLICA ALERT for $YESTERDAY / Size=$SIZE" -o tls=yes -s smtp.gmail.com:587 -t $ADMINMAIL1 -xu $GMAILID -xp $GMAILPASS -f $GMAILID -o message-file=$RESULT -o message-content-type=text
exit 1
else
echo "- BACKUP file name $FILE Size is $SIZE"
echo "- BACKUP file name $FILE Size is $SIZE" >> $RESULT
fi
#ssh -p $IP2_SSH_PORT root@$IP2 mkdir /$SQL_DIR
#scp -P $IP2_SSH_PORT $FILE_FINAL root@$IP2:/$SQL_DIR
#ssh -p $IP2_SSH_PORT root@$IP2 ls -lh /$SQL_DIR
# Import file in secondary radius
#ssh -p $IP2_SSH_PORT root@$IP2 "mysql -u$SQLUSER -p$SQLPASS $DB < $FILE
#mysql -h $IP2 -u$SQLUSER -p$SQLPASS $DB < $FILE
ssh -p $IP2_SSH_PORT root@$IP2 mysql -u$SQLUSER -p$SQLPASS $DB output
sendemail -t $email -u "INFO: DMA Replica Report OK: From $IP1 to $IP2 - $YESTERDAY" -o tls=yes -s smtp.gmail.com:587 -t $ADMINMAIL1 -xu $GMAILID -xp $GMAILPASS -f $GMAILID -o message-file=$RESULT -o message-content-type=text
rm $IP2_SSH_CHK
rm $RESULT
rm $PING_RESULT
rm $FILE
Email Report Sample:
Cron schedule to run the script Daily at 7am
00 07 * * * /temp/update_radius_from_10.0.0.1__TO__192.168.0.1.sh
Regard’s
Syed Jahanzaib