Bash Script to run a backup and upload to AWS S3 Failing - bash

Here's the code:
#!/bin/bash
# -------------------------------------------------
# -------------------------------------------------
# Use a comma-delimited list of single-quoted
# strings of the usernames to batch
# if it's empty it will backup all user directories
# in /home
# -------------------------------------------------
# -------------------------------------------------
USER_ACCOUNT=();
# -------------------------------------------------
# -------------------------------------------------
# Make sure the aws script is installed on the
# server, and the bucket name to upload these too
# are exact... case-sensitive
# -------------------------------------------------
# -------------------------------------------------
S3BUCKET='Kevs-Bucket/Test';
# Loop through the user array
# If it's empty, then get all users in the /home
# directory, based on each folder
# do not include root user
if [ ${#USER_ACCOUNT[#]} -eq 0 ]; then
# turn off dotglob and nullglob
cd /home;
shopt -s dotglob;
shopt -s nullglob;
DIRARR=(*/);
# we have our directories, now loop them and grab the user
# once we have the user, skip the root user
for d in ${!DIRARR[#]}; do
# Assign an account variable
ACCT=stat -c '%U' ${DIRARR[$i]}; #NOT WORKING HERE
if [ "$ACCT" == "root" ]; then
echo "ROOT";
else
run_backup $ACCT $S3BUCKET;
fi;
done;
else
# we have our list, now loop through them all
for i in ${!USER_ACCOUNT[#]}; do
# Assign an account variable
ACCT=${USER_ACCOUNT[$i]};
run_backup $ACCT $S3BUCKET;
done;
fi;
# -------------------------------------------------
# -------------------------------------------------
# Run the actual backup
run_backup(){
LOGFILE=/batch-move.log
# Package the account
./scripts/pkgacct $1;
echo '##########################################' >> $LOGFILE;
echo "# Start: date +'%T'" >> $LOGFILE;
echo "# Backing Up: $1" >> $LOGFILE;
echo '##########################################' >> $LOGFILE;
# Upload it to S3
s3put $2/cpmove-$1.tar.gz /home/cpmove-$1.tar.gz;
echo '##########################################' >> $LOGFILE;
echo "# Uploading Backup: $1" >> $LOGFILE;
echo '##########################################' >> $LOGFILE;
# Remove the file from the server
/bin/rm -f /home/cpmove-$1.tar.gz;
echo '##########################################' >> $LOGFILE;
echo "# Removing Backup Up: $1" >> $LOGFILE;
echo "# Finish: date +'%T'" >> $LOGFILE;
echo '##########################################' >> $LOGFILE;
}
I'm getting an error here ACCT=stat -c '%U' ${DIRARR[$i]}; #NOT WORKING HERE and the error is stating that -c is not a valid option for stat on my CentOS server
I have verified through other means that stat -c does work, so I assume that my code attempting to get the folders owner into a variable is incorrect.
Can you help me figure it out?

The line that's not working (below) contains the variable $i which you have not defined and the $() notation is not present, see added code below "EDIT - Try this".
ACCT=stat -c '%U' ${DIRARR[$i]}; #NOT WORKING HERE
You are looping through your arrays in an unusual way. Here is an example of how to loop through the elements of an array of filenames in Bash.
files=( "/home/User/FileName1" "/home/User/FileName2" "/home/User/FileName3" )
for fileName in "${files[#]}" ; do
echo "$fileName"
done
Also instead of using globbing to build your array – DIRARR=(*/); – you might want to consider using a loop to iterate over the files, for example:
for fileName in /home/* ; do
echo "$fileName"
done
Hope this helps.
EDIT - Try this:
Note: On my system the following ignores '.' and '..'.
# To avoid confusion 'ACCT' would be better named as 'OWNER'.
# Loop through the files in: /home/
for filename in /home/* ; do
# Get the owner of $filename.
ACCT=$(stat -c '%U' "$filename")
# If the file is a directory NOT owned by root, run backup.
if [ -d "$filename" -a "$ACCT" != "root" ]; then
# Uncomment when satisfied
# run_backup "$ACCT" "$S3BUCKET"
echo "Run backup - not owned by root:"
echo "filename: $filename owner: $ACCT"
# If the file is a directory owned by root, DO NOT run backup.
elif [ -d "$filename" -a "$ACCT" = "root" ]; then
# Remove elif clause when satisfied.
echo "Do not run backup - owned by root:"
echo "filename: $filename owner: $ACCT"
fi
done
Please note use of $() in the "ACCT=" line.

Related

Does converting this GPG encryption bash script to powershell make sense? Is there a more simple methodolgy?

I want to convert this BASH script into something a little more Windows friendly. I assume PowerShell is the way to go. So I found this Powershell script online that's a wrapper for the pgp2.exe. So I understand the function for encrypting the data, but I'm lost on how to code a temporary directory. So my question is what is the PowerShell logic that would follow to build a temporary sub-directory, create a path & file to then be encrypted. I am not familiar with PowerShell or the Windows 10 cmd terminal. So if anyone has any resources for converting this bash script into something more Windows 10 friendly, that would be helpful.
PS: I am not asking about syntax conversion. I'm looking more for the broad stroke logic and commands behind the Windows 10 language.
# -----------------------------------------------------------
# generate a file of random size with random bytes
# select a format generated file and encrypt/compress
# shred original generated file
# input: temp sub-directory path with filename
# -----------------------------------------------------------
do_a_file () {
filename="${1}"
filesize=0
while [[ $filesize -lt 1024 ]]; do
filesize=$(($RANDOM % 19999999))
done
dd if=/dev/urandom of="${filename}" bs=1 count=$filesize status=noxfer conv=notrunc,noerror 2> /dev/null
random_pass=$(dd if=/dev/urandom bs=128 count=1 status=noxfer 2> /dev/null | sha512sum - | cut -f 1 -d ' ')
extension=$(($RANDOM % 3))
case "${extension}" in
0)
# by default creates output file with .gpg extension
gpg -q --yes --passphrase "${random_pass}" -c "${filename}" &> /dev/null
;;
1)
tempfile="${filename}.rar"
rar a -o+ -hp"${random_pass}" "${tempfile}" "${filename}" &> /dev/null
;;
2)
tempfile="${filename}.7z"
7za a -y -mhe=on -p"${random_pass}" "${tempfile}" "${filename}" &> /dev/null
;;
esac
shred -n 1 -zu "${filename}"
}
# -----------------------------------------------------------
# split sub-directory path into array
# determine which subdirectory ($RANDOM % sizeof array)
# input: full sub-directory path
# -----------------------------------------------------------
pick_durr () {
subdir_path="${1}"
durr=""
d=0
durrs=()
while true; do
next_dir=${subdir_path%%/*}
subdir_path=${subdir_path#*/}
durrs[$d]=${next_dir}
((++d))
if [[ ${subdir_path} == ${next_dir} ]]; then
break;
fi
done
subdir_count=${#durrs[#]}
num_subdir=$(($RANDOM % $subdir_count))
d=0
while [[ $d -lt $num_subdir ]]; do
durr="${durr}/${durrs[$d]}"
((++d))
done
echo ${durr}
}
# -----------------------------------------------------------
# determine number of words in filename ($RANDOM % 5) + 1
# grab random words from wordlist
# concatenate together for filename
# -----------------------------------------------------------
build_filename () {
filename=""
nameword_count=0
num_namewords=$(($RANDOM % 5))
num_words=$(wc -l "${wordlist}" | cut -d ' ' -f 1)
((++num_namewords))
((++num_words))
while [[ $nameword_count -lt $num_namewords ]]; do
rand_line=$(($RANDOM % $num_words))
rand_word=$(sed -n "$rand_line p" "${wordlist}")
if [[ $nameword_count -eq 0 ]]; then
filename="${rand_word}"
else
filename="${filename}_${rand_word}"
fi
((++nameword_count))
done
echo "${filename}"
}
# -----------------------------------------------------------
# generate a sub-directory path
# select number [0, 12] of sub-directories to create
# grab $num_subdirs of random words from wordlist
# concatenate together for sub-directory path
# -----------------------------------------------------------
build_path () {
num_subdirs=$(($RANDOM % 13))
if [[ $num_subdirs -eq 0 ]]; then
echo
return 0
fi
subdir_count=0
subdir_path=""
num_words=$(wc -l "${wordlist}" | cut -d ' ' -f 1)
((++num_words))
while [[ $subdir_count -lt $num_subdirs ]]; do
rand_line=$(($RANDOM % $num_words))
rand_word=$(sed -n "$rand_line p" "${wordlist}")
subdir_path="${subdir_path}/${rand_word}"
((++subdir_count))
done
echo "${subdir_path}"
}
# -----------------------------------------------------------
# build out sub-directory structure and create path
# select number [9001, (9001 + $RANDOM)] of files to generate
# generate filename
# select temporary sub-directory path
# create the encrypted/compressed file
# repeat for specified number of iterations
# input: working directory, number of iterations
# -----------------------------------------------------------
main () { # main function, durr
num_files=0 # number of files to generate
loop_count=0 # count number of iterations passed
while [[ $loop_count -lt "${2}" ]]; do # iterations
subdurr_path=$(build_path) # build a sub-directory path
mkdir -p "${1}/${subdurr_path}" # create the sub directory path
while [[ $num_files -lt 9001 ]]; do # generate over 9000 files
num_files=$RANDOM # select number of files to generate
done # end set num_files loop
while [[ $num_files -gt 0 ]]; do # number of files
filename=$(build_filename) # create filename
subdurr=$(pick_durr ${subdurr_path}) # select sub-directory
do_a_file "${1}/${subdurr}/${filename}" # generate the file
((--num_files)) # decrement file count
done # end of number of files loop
((++loop_count)) # increment loop count
done # end of iterations loop
yes | rm -rf "${1}" &> /dev/null # unlink encrypted files
} # end of main function

How to run .sh using Cygwin in windows

I have a file named backup.sh, I have installed and set path variable too. but when i double click backup.sh file its saying something which I don't know. scenario is back mongo db database backup using script in grails.
#!/bin/bash
#
# Michael Mottola
# <mikemottola#gmail.com>
# December 18, 2011
#
# Creates backup files (bson) of all MongoDb databases on a given server.
# Default behaviour dumps the mongo database and tars the output into a file
# named after the current date. ex: 2011-12-19.tar.gz
#
### Set server settings
HOST="localhost"
PORT="27017" # default mongoDb port is 27017
USERNAME=""
PASSWORD=""
# Set where database backups will be stored
# keyword DATE gets replaced by the current date, you can use it in either path below
BACKUP_PATH="Desktop/path/to/backup/directory" # do not include trailing slash
FILE_NAME="DATE" #defaults to [currentdate].tar.gz ex: 2011-12-19.tar.gz
##################################################################################
# Should not have to edit below this line unless you require special functionality
# or wish to make improvements to the script
##################################################################################
# Auto detect unix bin paths, enter these manually if script fails to auto detect
MONGO_DUMP_BIN_PATH="$(which mongodump)"
TAR_BIN_PATH="$(which tar)"
# Get todays date to use in filename of backup output
TODAYS_DATE=`date "+%Y-%m-%d"`
# replace DATE with todays date in the backup path
BACKUP_PATH="${BACKUP_PATH//DATE/$TODAYS_DATE}"
# Create BACKUP_PATH directory if it does not exist
[ ! -d $BACKUP_PATH ] && mkdir -p $BACKUP_PATH || :
# Ensure directory exists before dumping to it
if [ -d "$BACKUP_PATH" ]; then
cd $BACKUP_PATH
# initialize temp backup directory
TMP_BACKUP_DIR="mongodb-$TODAYS_DATE"
echo; echo "=> Backing up Mongo Server: $HOST:$PORT"; echo -n ' ';
# run dump on mongoDB
if [ "$USERNAME" != "" -a "$PASSWORD" != "" ]; then
$MONGO_DUMP_BIN_PATH --host $HOST:$PORT -u $USERNAME -p $PASSWORD --out $TMP_BACKUP_DIR >> /dev/null
else
$MONGO_DUMP_BIN_PATH --host $HOST:$PORT --out $TMP_BACKUP_DIR >> /dev/null
fi
# check to see if mongoDb was dumped correctly
if [ -d "$TMP_BACKUP_DIR" ]; then
# if file name is set to nothing then make it todays date
if [ "$FILE_NAME" == "" ]; then
FILE_NAME="$TODAYS_DATE"
fi
# replace DATE with todays date in the filename
FILE_NAME="${FILE_NAME//DATE/$TODAYS_DATE}"
# turn dumped files into a single tar file
$TAR_BIN_PATH --remove-files -czf $FILE_NAME.tar.gz $TMP_BACKUP_DIR >> /dev/null
# verify that the file was created
if [ -f "$FILE_NAME.tar.gz" ]; then
echo "=> Success: `du -sh $FILE_NAME.tar.gz`"; echo;
# forcely remove if files still exist and tar was made successfully
# this is done because the --remove-files flag on tar does not always work
if [ -d "$BACKUP_PATH/$TMP_BACKUP_DIR" ]; then
rm -rf "$BACKUP_PATH/$TMP_BACKUP_DIR"
fi
else
echo "!!!=> Failed to create backup file: $BACKUP_PATH/$FILE_NAME.tar.gz"; echo;
fi
else
echo; echo "!!!=> Failed to backup mongoDB"; echo;
fi
else
echo "!!!=> Failed to create backup path: $BACKUP_PATH"
fi
its giving --host command not found. updated for later help.
bash FilePath/backup.sh
For example if you have backup.sh file in E:/SohamShetty/Files then you should do like this.
bash E:/SohamShetty/Files/backup.sh
You can run it as :
bash backup.sh
or
chmod +x backup.sh
./backup.sh

bash shell script to run imgcmp on two JPGs and store 'different' ones

I've got an IP camera that ftps files to a directory on my SuSE server.
I'm trying to write a shell script to do the following:
for every file in a directory;
use image compare to check this file against the next one
store the output in a file or variable.
if the next file is different then
copy the original to another folder
else
delete the original
end for
Running the following at the prompt generates this:
myserver:/uploads # imgcmp -f img_01.jpg -F img_02.jpg -m rmse > value.txt
myserver:/uploads # cat value.txt
5.559730
5.276747
6.256132
myserver:/uploads #
I know there's loads wrong with the code, the main issue I've got is with executing imgcmp from the script and extracting a value from it, so please point out the obvious as it may not be to me.
FILES=/uploads/img*
declare -i value
declare -i result
value = 10
shopt -s nullglob
# no idea what the above even does #
# IFS=.
# attempt to read the floating point number from imgcmp & make it an integer
for f in $FILES
do
echo "doing stuff w/ $f"
imgcmp -f 4f -F 4f+1 -m rmse > value.txt
# doesn't seem to find the files from the variables #
result= ( $(<value.txt) )
if [ $result > $value ] ; then
echo 'different';
# and copy it off to another directory #
else
echo 'same'
# and delete it #
fi
if $f+1 = null; then
break;
fi
done
when running the above, I get an error cannot open /uploads/img_023.jpg+1
and doing a cat of value.txt shows nothing, so all the files show as being the same.
I know where the issues are, but I've got no idea what I should actually be doing to extract the output of imgcmp (run from within a script) and then get it into a variable that I can compare it with.
FILES=/uploads/*
current=
for f in $FILES; do
if [ -z "$current" ]; then
current="$f"
continue
fi
next="$f"
echo "<> Comparing $current against $next"
## imgcmp will return non-0 if images cannot be compared
## and print an explanation message to stderr;
if result=$(imgcmp -f $current -F $next -m rmse); then
echo "comparison result: " $result
## Checking whether the first value returned
## is greater than 10
if [ "$(echo "$result" | awk '$1 > 10 {print "different"}')" = "different" ]; then
echo 'different';
# cp -v $current /some/other/folder/
else
echo 'same'
# rm -v $current
fi
else
## images cannot be compared... different dimensions / components / ...
echo 'wholly different'
# cp -v $current /some/other/folder/
fi
current="$next"
done

bash script prova.sh: line 76: syntax error: unexpected end of file

I have below script but I can't find the error. Somebody an help me ?
In concrete I split a big file in different then compress any file, move it and send by ftp rename destination filename.
Something not work :(
in line:
put ${file} ${7}.T${j}(+1)
I try to rename the file with (+1) ended new filename
Kind regards
#!/bin/bash
# configuration stuff
# ${1} absolute path file
# ${2} num_files
# ${3} output_filename
# ${4} ipMainframe ip to put files
# ${5} FTP username
# ${6} FTP password
# ${7} destination filename
if [ ! $# == 7 ]; then
#number of parameter different of two
echo "Number of parameter incorrect"
echo "Command use: LLP_split_gzip_sendFTPandTrigger.sh absolute_path_file number_of_pieces output_filename ipMainframe userFTP pwdFTP destinationFilename"
exit 1
fi
if [ -f ${1} ]; then
# If file exists
if [[ ${2} =~ ^[\-0-9]+$ ]] && (( ${2} > 0)); then
# if number of pieces is an integer > 0
#Remove old files
echo "home directory = $HOME"
CMD=`rm -f '"$HOME"/"$3"*'`
if [ $? != 0 ]; then
echo "Impossible to remove old files $home/llp_tmp* $home/"$3"* in home directory"
echo $CMD
fi
# Calculate line for every file splitted
total_lines=$(cat ${1} | wc -l)
((lines_per_file = (total_lines + ${2} - 1) / ${2}))
# Split the actual file, maintaining lines.
CMD=`split -l "$lines_per_file" "$1" "$HOME"/llp_tmp`
if [ $? != 0 ]; then
echo "SPLITTING FILE ERROR: problem to split file."
echo $CMD
exit 3
fi
#For every file splitted rename and zip it
i=1
for file in $HOME/llp_tmp*; do
CMD=`mv "$file" "$3"."$i"`
if [ $? != 0 ]; then
echo "Impossible to rename file"
echo $CMD
exit 5
fi
CMD=`gzip "$3"."$i"`
if [ $? != 0 ]; then
echo "Impossible to compress file $3.$i"
echo $CMD
exit 6
fi
i=`expr $i + 1`
done
ftp -n ${4} << EOF
j=1
user ${5} ${6}
for file in $3.*; do
put ${file} ${7}.T${j}(+1)
j=`expr $j + 1`
done
quit
else
echo "number of pieces second parameter must be more than 0."
exit 2
fi
else
echo "absolute path first paramater doesnt exist"
exit 1
fi
exit 0
You are not terminating your here document. When I run your script I get:
gash.sh: line 72: warning: here-document at line 54 delimited by end-of-file (wanted `EOF')
gash.sh: line 73: syntax error: unexpected end of file
ftp -n ${4} << EOF is the issue. Where is your here document?
The warning says it all, you don't have an EOF marker. Note that this MUST NOT BE INDENTED! The EOF must be in "column 0" and have no trailing characters, including whitespace.
Edit: It appears you want to use program constructs within a single FTP session - I don't know of a way of doing that in Bash. Perl has an easy to use FTP module where you can do it, simple example:
use strict;
use Net::FTP;
my $ftp = Net::FTP->new ("hostname");
$ftp->login ("username", "password");
$ftp->binary ();
for my $file (glob("$ENV{HOME}/llp_tmp*")) {
$ftp->put ($file);
}
$ftp->quit();
You don't need parentheses around +1.
Change it to:
put "${file}" "${7}.T${j}+1"
It's good practice to quote variables.
Another tip: Instead of j=`expr $j + 1`, you can simply use ((j++)).

multiple sed operations creating empty file

When as part of shell script only one line is operating on a file using sed command the redirected file contains the updated data, as below
cat ${PROP_PATH}/${PROP_FILE} | sed "s!${ISTR_KEY}=.*!${ISTR_KEY}=${SIM_ISTR_KEY_VAL}!" > ${UPDATEDPROPS_DIR}/${PROP_FILE}
whereas when it is executed as part of a shell script, where after this another sed command updates the same file as in the below script at the end what i get is an empty file, why ? ..... ideas please.
(check 'switchAll2Sim()' function below)
#!/bin/ksh
#
SIM_ICR_KEY_VAL="http://www.example.com/sim/http/icr"
SIM_ISTR_KEY_VAL="http://www.example.com/sim/http/istr"
SIM_GT_KEY_VAL="http://www.example.com/sim/http/gtr"
#
ICR_KEY="interface.url.icr"
ISTR_KEY="interface.url.istr"
GT_KEY="interface.ws.url.gt"
## Property Files
PROP_PATH=""
PROP_FILE="properties"
##
DATE=`date +%m%d%Y`
DATETIME=`date +%m%d%Y-%T`
BCKUP_DIR=_bckup
UPDATEDPROPS_DIR=_updatedprops
# ----------------------------------
pause(){
echo "Press [Enter] key to continue..."
read fackEnterKey
}
permissions(){
chmod 777 ${UPDATEDPROPS_DIR}
}
backup(){
if [ ! -d "${BCKUP_DIR}" ]; then
mkdir ${BCKUP_DIR}
fi
if [ ! -d "${UPDATEDPROPS_DIR}" ]; then
mkdir ${UPDATEDPROPS_DIR}
fi
permissions
## keep backup of properties
cp ${PROP_PATH}/${PROP_FILE} ${BCKUP_DIR}/${PROP_FILE}_${DATETIME}
echo "Backup of property files completed at: " ${DATETIME}
}
#-------------------------------------------------------------
# switch all properties to SIM
#-------------------------------------------------------------
switchAll2Sim(){
backup
#
# update files
cat ${PROP_PATH}/${PROP_FILE} | sed "s!${ISTR_KEY}=.*!${ISTR_KEY}=${SIM_ISTR_KEY_VAL}!" > ${UPDATEDPROPS_DIR}/${PROP_FILE}
cat ${UPDATEDPROPS_DIR}/${PROP_FILE} | sed "s!${ICR_KEY}=.*!${ICR_KEY}=${SIM_ICR_KEY_VAL}!" > ${UPDATEDPROPS_DIR}/${PROP_FILE}
cat ${UPDATEDPROPS_DIR}/${PROP_FILE} | sed "s!${GT_KEY}=.*!${GT_KEY}=${SIM_GT_KEY_VAL}!" > ${UPDATEDPROPS_DIR}/${PROP_FILE}
echo "Switch all to SIM completed at: " ${DATETIME}
pause
}
# switch all properties to real
#-------------------------------------------------------------
switchAll2Real(){
pause
}
#-------------------------------------------------------------
dispCurrentStats(){
echo "Displaying current properties..."
echo "*********************************"
echo " File: " ${PROP_PATH}/${PROP_FILE}
grep ${ICR_KEY} ${PROP_PATH}/${PROP_FILE}
grep ${ISTR_KEY} ${PROP_PATH}/${PROP_FILE}
grep ${GT_KEY} ${PROP_PATH}/${PROP_FILE}
#
echo "*********************************"
pause
}
show_menus() {
clear
echo "~~~~~~~~~~~~~~~~~~~~~"
echo " M E N U"
echo "~~~~~~~~~~~~~~~~~~~~~"
echo "1. Display current properties"
echo "2. Switch all to real"
echo "3. Switch all to simulator"
echo "4. Exit"
}
# read input from the keyboard and take a action
read_options(){
read option
case $option in
1) dispCurrentStats ;;
2) switchAll2Real ;;
3) switchAll2Sim ;;
4) exit 0;;
*) echo "Please insert options 1 ~ 4";;
esac
}
# -----------------------------------
# Main - infinite loop
# ------------------------------------
while true
do
show_menus
read_options
done
Thanks, using '-i, says [sed: illegal option -- i]
Then you have to work with tmp files.
cp foo foo.tmp
sed "s/x/y/" foo.tmp > foo
/bin/rm foo.tmp
OR
sed "s/x/y/" foo > foo.tmp
/bin/mv -f foo.tmp foo
is probably more efficient.
I hope this helps.
Your problem is that cat is reading from the same file that sed is writing to.
cat foo | sed "s/x/y/" > foo
Will not work because cat and sed run at the same time, not one after the other.
To fix this try the -i option to sed.
sed -i "s/x/y/" foo

Resources