I have a bash script that reads 10 lines on a file, each line has the server name that I need to boot up, the boot process takes long and outputs a lot of information, I need to send a "control c" on the script so it can proceed on the next server, or else the loop process will take long on each server during boot process
here is the actual output screen during boot process, it takes more than 30 minutes to complete on each server, loop may continue if control c is sent to the script/automation
$ eb restore environment_id
INFO: restoreEnvironment is starting.
-- Events -- (safe to Ctrl+C) Use "eb abort" to cancel the command.
2017-01-01 12:00 restoreEnvironment is starting
2017-01-01 12:15 Environment health has transitioned to Pending. Initialization in progress (running for 28 seconds). There are no instance
2017-01-01 12:20 Created security group named: sg-123123123
2017-01-01 12:22 Created load balancer named: awseb-e-3-qweasd2-DSLFLSFJHLS
2017-01-01 12:24 Created security group named: sg-123123124
2017-01-01 12:26 Created Auto Scaling launch configuration named: awseb-e-DSLFLSFJHLS-stack-AWSEBAutoScalingLaunchConfiguration-DSLFLSFJHLS
2017-01-01 12:28 Added instance [i-01askjdkasjd123] to your environment.
2017-01-01 12:29 Created Auto Scaling group named: awseb-e-DSLFLSFJHLS-stack-AWSEBAutoScalingLaunchConfiguration-DSLFLSFJHLS
2017-01-01 12:30 Waiting for EC2 instances to launch. This may take a 30 minutes
2017-01-01 13:15 Successfully launched environment: pogi-server
Here is my working script
#!/bin/bash
DIR=/jenkins/workspace/restore-all
INSTANCE_IDS=/jenkins/workspace/environment-ids
EB_FILE=$DIR/server.txt
echo "PROCEEDING TO WORK DIRECTORY"
cd $DIR ; pwd
echo""
echo "CREATING A CODE FOLDER"
mkdir $DIR/code ; pwd ; ls -ltrh $DIR/code
echo""
for APP in `cat $EB_FILE | awk '{print $NF}' | grep -v 0`
do
echo "#########################################"
echo "RESTORING = "$APP
echo ""
echo "COPYING BEANSTALKFILES"
mkdir $DIR/code/$APP ; cd $DIR/code/$APP
cp -pr $DIR/beasntalk/$APP/dev/.e* $DIR/code/$APP
echo ""
echo ""
echo "TRIGGERING EB RESTORE"
cd $DIR/code/$APP
eb restore `tail -1 $INSTANCE_IDS/$APP-dev.txt`
echo ""
echo "REMOVE CODE FOLDER"
cd $DIR/code ; rm -rf $DIR/code/*
echo""
done
echo "REMOVE WORKSPACE FOLDER"
rm -rf $DIR/*
echo""
Have you tried putting the eb restore command in the background with &?
I took a stab a rewriting what you posted. I don't know if it will work due to a few uncertainties (ie what do the contents of server.txt look like, do you copy it into the restore-all directory every time, do you put beanstalk/*/dev/.e* into restore-all each time, etc... etc...), but here it is:
#!/bin/bash
restore-all_dir=/jenkins/workspace/restore-all
InstanceIDs=/jenkins/workspace/environment-ids
EBFile=${restore-all_dir}/server.txt
echo -ne " PROCEEDING TO ${restore-all_dir} ..."
cd ${restore-all_dir}
echo -e "\b\b\b - done\n"
pwd
echo -ne " CREATING FOLDER ${restore-all_dir}/code ..."
mkdir ${restore-all_dir}/code
echo -e "\b\b\b - done\n"
pwd
ls -ltrh ${restore-all_dir}/code
for i in `awk '!/0/ {print $NF}' ${EBFile}`; do
echo "#########################################"
echo -e "#### RESTORING: ${i}\n"
echo -en " COPYING BEANSTALKFILES ..."
mkdir ${restore-all_dir}/code/${i}
cd ${restore-all_dir}/code/${i}
cp -pr ${restore-all_dir}/beanstalk/${i}/dev/.e* ${restore-all_dir}/code/${i}
echo -e "\b\b\b - done\n"
echo -en " TRIGGERING EB RESTORE ..."
cd ${restore-all_dir}/code/${i}
echo "eb restore `tail -1 ${InstanceIDs}/${i}-dev.txt` > ${restore-all_dir}/${i}_output.out &" ## directs output to a file and puts the process into the background
echo -e "\b\b\b - put into background\n output saved in ${restore-all_dir}/${i}_output.out\n"
## Is this necessary in every iteration, or can it be excluded and just removed with the final rm -rf command?
## echo -en " REMOVE CODE FOLDER ..."
## cd ${restore-all_dir}/code
## rm -rf ${restore-all_dir}/code/*
## echo -e "\b\b\b - done\n"
done
## this will loop "working..." until all background eb restore jobs are no longer "Running"
echo -en "\n working..."
while `jobs | awk '/Running/ && /eb restore/ {count++} END{if (count>=1) print "true"; else print "false"}'`; do
echo -en "\b\b\b\b\b\b\b\b\b\b\b working..."
done
echo -e "\b\b\b\b\b\b\b\b\b\b\b - done \n"
## this will also remove the output files unless they are put somewhere else
echo -en " REMOVE WORKSPACE FOLDER ..."
rm -rf ${restore-all_dir}/*
echo -e "\b\b\b - done\n"
Related
I wrote this bash script to remove the old version of a Mac application and install the new version. It all works well, except it doesn't seem to delete the application before installing the new one - instead, it seems to write over the top of it, which is causing some issues when you try to launch the application. Wondering where I'm going wrong in my script - any help would be much appreciated.
#!/bin/sh
#Script to remove the old version of 8x8 Virtual Office and the install the latest version.
APP=/Applications/8x8\ -\ Virtual\ Office.app
VERSION=$(defaults read /Applications/8x8\ -\ Virtual\ Office.app/Contents/Info | grep CFBundleShortVersionString | cut -c35-39)
#See Script options in JSS for $4 value
LATEST="$4"
pid=$(ps -fe | grep '8x8 - Virtual Office' | grep -v grep | awk '{print $2}')
if test -e "$APP" ; then
if [ "$VERSION" \< "$LATEST" ] ; then
echo "8x8 Virtual Office $VERSION found"
echo "Application needs updating..."
if [[ -n $pid ]]; then
echo "Quitting application first..."
kill $pid
sleep 5s
echo "Removing old version..."
rm -f $APP
sleep 5s
echo "Installing new version..."
jamf policy -event install88
else
echo "Application not running, removing old version..."
rm -f $APP
sleep 5s
echo "Installing new version..."
jamf policy -event install88
exit 0
fi
else
echo "No update required"
exit 0
fi
else
echo "8x8 Virtual Office not found, installing..."
jamf policy -event install88
exit 0
fi
APP=/Applications/8x8\ -\ Virtual\ Office.app
Your path contains spaces. So remove command should be rm -f "$APP"
Demo:
:=>APP=/Applications/8x8\ -\ Virtual\ Office.app
:=>echo $APP
/Applications/8x8 - Virtual Office.app
:=>rm $APP
rm: cannot remove '/Applications/8x8': No such file or directory
rm: cannot remove '-': No such file or directory
rm: cannot remove 'Virtual': No such file or directory
rm: cannot remove 'Office.app': No such file or directory
:=>rm "$APP"
rm: cannot remove '/Applications/8x8 - Virtual Office.app': No such file or directory
:=>
In the above demo when we execute rm $APP rm command is taking 4 arguments Applications/8x8 , - , Virtual ,Office.app becasue of spaces in path.
`
I have made a script to run a backup from a local machine to a remote server. However when running this script I get the error:
/etc/run_backup: 173: /etc/run_backup: Syntax error: Unterminated quoted string
For the life of me I cannot see why I am getting this error. Below is the script (with some redactions):
#!/bin/sh
#
# this script must be run as root on machine with folders
# to be backed up.
#
# This script will attempt to back up several folders
# to a remote server which must have passwordless ssh
# access set up for the root users on the machines. If
# this is inaccessible a backup will be attempted in some
# local storage, e.g. a usb drive. Locations are defined
# below
set -ex
##### CONFIG VARIABLES #####
# backup server location
backup_server=<redacted>
# name of top level folder used to contain all backups on either
# the remote server or local backup location
backup_folder=backups
# the root folder where $backup_folder will be placed on the
# remote machine
remote_backup_storage_root=/mnt/reos-storage-2
# the root folder where $backup_folder will be placed on
# local storage if the remote machine is not available
local_backup_storage_root=/media/usbflash0
# the directory wher the results of the backup attempt by rdiff-backup
# will be placed for each folder backup, and also the where the backup
# log file generated by this script will be placed
error_report_dir=/home/pi/rdiff-backup-errors
# email to send any messages to. An email will be sent on success
# and under certain error conditions
email_receiver=<redacted>
# the location of the nextcloud data folder to backup
nc_data_dir=/opt/nextcloud-data
# root directory of the nextcloud site install to backup
nc_web_dir=/var/www/nextcloud
##### END CONFIG VARIABLES #####
# lock file will be created and used to ensure we don't try to run two
# backup processes at the same time
lockfilename=/var/run/remotebackup.pid
if test -e ${lockfilename}; then
echo "Backup is already running!"
echo "If it isn't, remove ${lockfilename} and try again."
echo "/etc/run_backup: unable to run backup as lock file ${lockfilename} is present" | mail -s "Backup Problem" ${email_receiver}
exit 3
else
echo $$ > ${lockfilename};
fi
remote_server_backup_folder=${remote_backup_storage_root}/${backup_folder}
logfile=${error_report_dir}/rdiff-backup.log
echo "[$(date "+%m%d%Y %T")] Starting backup" >> ${logfile}
# where the database will be backed up to. We don't store the date as we will
# then back up the whole nextcloud data directory using rdiff-backup onto the
# backup server which will allow us to step back in time if required
db_backup_file=${nc_data_dir}/nextcloud-db.bak
cd ${nc_web_dir}
echo "[$(date "+%m%d%Y %T")] Putting nextcloud in maintenance mode" >> ${logfile}
# turn on nextcloud maintenance mode
sudo -u www-data php occ maintenance:mode --on
echo "[$(date "+%m%d%Y %T")] Making nextcloud database dump on the server" >> ${logfile}
mysqldump -u root --single-transaction nextcloud > "${db_backup_file}"
echo "[$(date "+%m%d%Y %T")] Completed nextcloud database dump" >> ${logfile}
#nc -z ${backup_server} 22 > /dev/null
ssh -q root#${backup_server} exit
backup_machine_accessible=$?
echo "in here 0"
if [ "$backup_machine_accessible" -eq "0" ] ;
then
echo "in here 1"
backup_location="root#${backup_server}::${remote_server_backup_folder}"
echo "[$(date "+%m%d%Y %T")] Backing up to ${backup_location}" >> ${logfile}
else
# use local storage (flash drive)
echo "in here 2"
# check the USB drive is actually accesible, then make sure the
# backup directory on the drive actually exists
if test -e ${local_backup_storage_root}; then
echo "in here 3"
backup_location="${local_backup_storage_root}/${backup_folder}"
mkdir -p ${backup_location}
echo "[$(date "+%m%d%Y %T")] ${backup_server} was not accessible, backing up to ${backup_location}" >> ${logfile}
else
echo "in here 4"
echo [$(date "+%m%d%Y %T")] ${backup_server} was not accessible, and ${backup_location} was not available" >> ${logfile}
echo "${backup_server} was not accessible, and ${backup_location} was not also accessible" | mail -s "Backup Problem" ${email_receiver}
if test -e ${lockfilename}; then
rm ${lockfilename}
fi
exit 4
fi
fi
echo "out here"
# make the directory for storing errors reported during the backup process
mkdir -p ${error_report_dir}
# backup nextcloud instance
echo "[$(date "+%m%d%Y %T")] Starting nextcloud web backup" >> ${logfile}
rdiff-backup ${nc_web_dir} ${backup_location}/nextcloud/ 2> ${error_report_dir}/rdiff-backup-nextcloud-web_$(hostname).txt
# backup nextcloud user data
echo "[$(date "+%m%d%Y %T")] Starting nextcloud data backup" >> ${logfile}
rdiff-backup \
--ssh-no-compression \
--exclude "${nc_data_dir}/data/.opcache" \
--exclude "${nc_data_dir}/data/access.log" \
--exclude "${nc_data_dir}/data/error.log" \
--exclude "${nc_data_dir}/data/nextcloud.log" \
--exclude "${nc_data_dir}/data/access.log" \
${nc_data_dir} ${backup_location}/nextcloud-data/ 2> ${error_report_dir}/nextcloud-data-and-db_$(hostname).txt
# turn off nextcloud maintenance mode
cd ${nc_web_dir};
echo "[$(date "+%m%d%Y %T")] Turning off nextcloud maintnenance mode" >> ${logfile}
sudo -u www-data php occ maintenance:mode --off
# backup /etc which includes letsencrypt, fstab
echo "[$(date "+%m%d%Y %T")] /etc backup" >> ${logfile}
rdiff-backup --ssh-no-compression /etc/ ${backup_location}/etc_$(hostname)/ 2> ${error_report_dir}/rdiff-backup-etc_$(hostname).txt
# backup root
echo "[$(date "+%m%d%Y %T")] /root backup" >> ${logfile}
rdiff-backup --ssh-no-compression /root/ ${backup_location}/root_$(hostname)/ 2> ${error_report_dir}/rdiff-backup-root_$(hostname).txt
# backup home directories
echo "[$(date "+%m%d%Y %T")] /home backup" >> ${logfile}
rdiff-backup --ssh-no-compression /home/ ${backup_location}/home_$(hostname) 2> ${error_report_dir}/rdiff-backup-home_$(hostname).txt
echo "/etc/run_backup: backup of $(hostname) appears to have run successfully, or at least reached the end of the script without errors" | mail -s "Backup Report" ${email_receiver}
# Remove lock file and end script
#
if test -e ${lockfilename}; then
rm ${lockfilename}
else
echo "[$(date "+%m%d%Y %T")] Could not remove lock file ${lockfilename}!" >> ${logfile}
echo "/etc/run_backup was unable to remove lock file ${lockfilename} at end of the script as it did not exist" | mail -s "Backup Problem" ${email_receiver}
fi
And the output I get when running it is the following:
# /etc/run_backup
+ backup_server=<redacted>
+ backup_folder=backups
+ remote_backup_storage_root=/mnt/reos-storage-2
+ local_backup_storage_root=/media/usbflash0
+ error_report_dir=/home/pi/rdiff-backup-errors
+ email_receiver=<redacted>
+ nc_data_dir=/opt/nextcloud-data
+ nc_web_dir=/var/www/nextcloud
+ lockfilename=/var/run/remotebackup.pid
+ test -e /var/run/remotebackup.pid
+ echo 13057
+ remote_server_backup_folder=/mnt/reos-storage-2/backups
+ logfile=/home/pi/rdiff-backup-errors/rdiff-backup.log
+ date +%m%d%Y %T
+ echo [09162019 11:30:06] Starting backup
+ db_backup_file=/opt/nextcloud-data/nextcloud-db.bak
+ cd /var/www/nextcloud
+ date +%m%d%Y %T
+ echo [09162019 11:30:06] Putting nextcloud in maintenance mode
+ sudo -u www-data php occ maintenance:mode --on
Maintenance mode already enabled
+ date +%m%d%Y %T
+ echo [09162019 11:30:07] Making nextcloud database dump on the server
+ mysqldump -u root --single-transaction nextcloud
+ date +%m%d%Y %T
+ echo [09162019 11:30:12] Completed nextcloud database dump
+ ssh -q root#ed-mh-pi01.reoptimizesystems.com exit
+ backup_machine_accessible=0
+ echo in here 0
in here 0
/etc/run_backup: 173: /etc/run_backup: Syntax error: Unterminated quoted string
can anyone explain the error? It seems to me like it is something to do with the if statement if [ "$backup_machine_accessible" -eq "0" ] ;, but I can't see any problem with this.
I am trying to run the following cron job from bash (RHEL 7.4), An entry level postgres DB backup script I could write:
#!/bin/bash
# find latest file
echo $PATH
cd /home/postgres/log/
echo "------------ backup starts-----------"
latest_file=$( ls -t | head -n 1 | grep '\.log$' )
echo "latest file"
echo $latest_file
# find older files than above
echo "old file"
old_file=$( find . -maxdepth 1 -name "postgresql*" ! -newer $latest_file -mmin +1 )
if [ -f "$old_file" ]
then
echo $old_file
file_name=${old_file##*/}
echo "file name"
echo $file_name
# zip older file
tar czvf /home/postgres/log/archived_logs/$old_file.gz /home/postgres/log/$file_name
rm -rf /home/postgres/log/$file_name
else
echo "no old file found"
fi
Above is running correctly from shell and performing the intended tasks. It is also echoing needed info.
I have installed it with postgres user (not root) with crontab -e
*/2 * * * * /home/postgres/log/rollup.sh >> /home/postgres/log/logfile.csv 2>&1
It is correctly echoing (text which I have embedded for testing) but not the commands output to the .csv. Although it is not my concern. My concern is , it is not running those few commands at all.
I have given another try by changing the log file (.csv) path to /dev/null and commands in shell script are executing. I am not getting what I am missing here.
.csv file has 777 as permission , just to test
Crontab launches the script, the podcast updates correctly, but none of the file renaming (1st loop) or file moving (2nd loop).
If I run the script from the command-line, works perfectly.
I have added the "echo" lines to troubleshoot, the $file variable is consistent when run through command-line and crontab.
#/bin/sh
# Mad Money updates at 6:40 pm (timezone?) M-F
# At 6:30 pm CST it was ready to download
# http://podcast.cnbc.com/mmpodcast/lightninground.xml
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
echo "paths"
echo $PATH
podcast_folder=$"/home/zenon/podcasts/MAD_MONEY_W__JIM_CRAMER_-_Full_Episode"
episode_folder=$"/mnt/black-2tb-001/plex-server/shows/Mad-Money/Season-1"
hpodder update
sleep 1
hpodder download
sleep 1
cd ${podcast_folder}
for file in "$podcast_folder"/*.mp4; do
echo "Processing ${file}"
#"MadMoney-" Name
name=${file:60:9}
echo "podcast name is ${name}"
#"04" Month
month=${file:69:2}
echo "month is ${month}"
#"18" Day
day=${file:71:2}
echo "day is ${day}"
#"13" yr
yr=${file:73:2}
echo "year is 20${yr}"
title="${name}20${yr}.${month}.${day}.mp4"
echo "file ${file}"
echo "title ${title}"
# cp ${file} ${title}
mv ${file} ${title}
done
cd ${podcast_folder}
for file in "$podcast_folder"/*.mp4; do
chown zenon:plex ${file}
mv ${file} ${episode_folder}
done
# deletes any files older than 9 days
find ${episode_folder} -type f -mtime +9 -exec rm {} \;
exit
here is the debugging output from the script
cat cron.log
paths
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
1 podcast(s) to consider
Get: 4 MAD MONEY W/ JIM CRAMER - Full Episode
100% 1 B/s 0s
0 episode(s) to consider from 1 podcast(s)
0% 0 B/s 0s
Processing $/home/zenon/podcasts/MAD_MONEY_W__JIM_CRAMER_-_Full_Episode/*.mp4
you have a mistake in the first line, you should change
#/bin/sh to #!/bin/sh
Ive recently been introduced to bash scripting... So, Ive used my advanced theft course to throw together the attached script. it runs... and exits with "/xxx/ not mounted. You are not root! I have rdiff-backup and sshfs installed and working. The commands work fine on their own on the commandline, but in the script, well... Can you guys take a look and lemme know? PS I copied a LOT of this from scripts I found here and a few other places.
<code>
#!/bin/bash
# Version 1.5
# Prior to running this make sure you have ssh-keygen -t rsa to generate a key, then
# ssh username#target "mkdir .ssh/;chmod 700 .ssh"
# scp .ssh/id_rsa.pub username#target:.ssh/authorized_keys
#
# then check you can login and accept the ssh key
# ssh username#target "ls -la"
#
# Key things to remember, no spaces in pathnames, and try to use full paths (beginning with / )
#
# variables determine backup criteria
DATESTAMP=`date +%d%m%y`
USERNAME=username #remote site username here
TARGET=remote.ip.here #add the ip v4 address of the target
INCLUDES=/path/to/file/includes.txt #this is a txt file containing a list of directories you want backed up
EXCLUDES="**" #this is a list of files etc you want to skip
BACKUPLOG=/path/to/logfile/in/home/backuplog${DATESTAMP}.txt
OLDERTHAN=20W #change 20 to reflect how far back you want backups to exist
# to activate old backup expiry, uncomment the line below
#RMARGS=" --force --remove-older-than ${OLDERTHAN}"
TARGETMAIL="yourmailaddress#your.domain"
HOSTNAME=`hostname` #Dont change this!
TMPDIR=/backups Change this to the source folder
TARGETFOLDER=/backups change this to the TARGET folder
ARGS=" -v0 --terminal-verbosity 0 --exclude-special-files --exclude-other-filesystems --no-compression -v6"
# detecting distro and setting the correct path
if [ -e /etc/debian_version ];then
NICE=/usr/bin/nice
elif [ -e /etc/redhat-release ];then
NICE=/bin/nice
fi
if [ -e /tmp/backup.lock ];then
exit 0
fi
touch /tmp/backup.lock
touch -a ${BACKUPLOG}
cd /
/bin/mkdir -p ${TMPDIR}
/usr/bin/sshfs -o idmap=user -o ${USERNAME}#${TARGET}:/${TARGETFOLDER} ${TMPDIR} &>${BACKUPLOG}
# if you get errors mounting this then try
# mknod /dev/fuse -m 0666 c 10 229
for ITEMI in ${INCLUDES} ; do
ARGS="${ARGS} --include ${ITEMI} "
done
for ITEME in ${EXCLUDES} ; do
ARGS="${ARGS} --exclude-regexp '${ITEME}' "
done
# the --exclude ** / is a hack because it wont by default do multiple dirs, so use --include for all dirs then exclude everything else and sync / - if you dont understand dont worry
# ref: http://www.mail-archive.com/rdiff-backup-users#nongnu.org/msg00311.html
#echo /usr/bin/rdiff-backup ${ARGS} --exclude \'**\' / ${TMPDIR}/ &&
cat ${INCLUDES} | while read DIR; do
${NICE} -19 /usr/bin/rdiff-backup --exclude '**' ${DIR} ${TMPDIR}/ &>${BACKUPLOG}
if [ $? != 0 ]; then
echo "System Backup Failed" | mutt -s "Backup Log: System Backup Failed, Log attached!" -a ${BACKUPLOG} ${TARGETMAIL}
exit 1;
fi
done
#${NICE} -19 /usr/bin/rdiff-backup ${ARGS} --exclude '**' / ${TMPDIR}/ &>${BACKUPLOG} &&
echo Removing backups older than ${RMARGS}
${NICE} -19 /usr/bin/rdiff-backup -v0 --terminal-verbosity 0 ${RMARGS} ${TMPDIR}/ &>${BACKUPLOG}
/bin/umount ${TMPDIR} && /bin/rm -rf ${TMPDIR}/ &>${BACKUPLOG}
echo "System Backup Run" | mutt -s "Backup Log: System Backup Done!" -a ${BACKUPLOG} ${TARGETMAIL}
rm /tmp/backup.lock
rm ${BACKUPLOG}
Sorry, cannot paste, couldnt attach... BLIKSEM!
Thanks for ANY input... One HELL of a learning curve!!!
Regards,
B.