Passing variables from external bash file not behaving as expected - bash

I have a bash script that’s using rsync to backup some files from my local desktop to a remote machine on my LAN. 
I have the main script with some customisable variables in a separate .sh file to make for easy maintenance, deployment and git management. 
So I have this dir structure
sync-backup-to-cp.sh
config/settings.sh
And the following code to include the config/settings.sh into the main sync-backup-to-cp.sh
#! /bin/bash
#load variables file
source /Users/enwhat/Dropbox/Flex/Scripts/mac/rysnc-backup-to-cp/config/settings.sh
However the imported variables aren’t behaving as expected. If I have any space in any of the variables it throws an error about the variables being invalid. It seems that bash is interpreting this oddly.
Ie. rsync_opts="--verbose --archive” will cause the script to break and run an error such as “invalid numeric arguments or unknown arguments supplied”. Where as rsync_opts="--verbose” runs perfectly.
To help illustrate the script I've taken some snippets of the code showing the flow with help so far
from: config / settings.sh
RSYNC_OPTS=( --bwlimit=1000 --verbose )
from my main script, there's a function call where these variables are passed in.
backup "$RSYNC_BIN" "$BACKUP_FILE_LIST" "$EXCLUDE_FILE_LIST" "$SSH_PORT" "$SSH_KEY" "$SOURCE" "$DESTINATION" "$RSYNC_OPTS[*]"
then the full function
function backup(){ #uses rsync to backup to server
#takes 8 args 1
#define local vars
local l_rsync_bin=$1
local l_rsync_backup_file_list=$2
local l_rsync_exclude_file_list=$3
local l_rsync_ssh_port=$4
local l_rsync_ssh_key=$5
local l_rsync_source=$6
local l_rsync_dest=$7
local l_rsync_opts=$8
#local l_time
#l_time=$(date)
#caffinate stops system from sleeping
echo ""$l_rsync_bin" "$l_rsync_opts" --verbose --archive --recursive --numeric-ids --human-readable --partial --progress --relative --itemize-changes --stats --rsync-path="sudo rsync" --delete-during --files-from="${l_rsync_backup_file_list}" --exclude-from="${l_rsync_exclude_file_list}" -e "ssh -q -p ${l_rsync_ssh_port} -i ${l_rsync_ssh_key}" "${l_rsync_source}" "${l_rsync_dest}""
caffeinate -s "$l_rsync_bin" "$l_rsync_opts" --verbose --archive --recursive --numeric-ids --human-readable --partial --progress --relative --itemize-changes --stats --rsync-path="sudo rsync" --delete-during --files-from="${l_rsync_backup_file_list}" --exclude-from="${l_rsync_exclude_file_list}" -e "ssh -q -p ${l_rsync_ssh_port} -i ${l_rsync_ssh_key}" "${l_rsync_source}" "${l_rsync_dest}"
}

Since you are quoting $rsync_opts, the entire value is passed as a single, whitespace-containing argument to rsync. In order for each option to be passed as a separate argument, you need to leave the parameter expansion unquoted:
rsync $rsync_opts
However, you can't include arguments that actually contain whitespace like this; all whitespace is treated by the shell as separating arguments. The right way to store arguments is to use an array:
rsync_opts=( --verbose --archive )
rsync "${rsync_opts[#]}"
It may not be necessary for your current use case, but it's a good idea to get into the habit of doing things the right way to avoid nasty surprises later.
For example,
local -a l_rsync_opts
l_rsync_opts=(--bwlimit=1000 --verbose --rsync-path="sudo rsync")
UPDATE: Based on your edit, you need to do the following:
backup ... "${RSYNC_OPTS[#]}" # #, not *
# Note the changes involving l_rsync_opts
function backup(){ #uses rsync to backup to server
#takes 8 args 1
#define local vars
local l_rsync_bin=$1
local l_rsync_backup_file_list=$2
local l_rsync_exclude_file_list=$3
local l_rsync_ssh_port=$4
local l_rsync_ssh_key=$5
local l_rsync_source=$6
local l_rsync_dest=$7
local l_rsync_opts=( "${#:8}" )
#local l_time
#l_time=$(date)
#caffinate stops system from sleeping
echo ""$l_rsync_bin" "${l_rsync_opts[#]}" --verbose --archive --recursive --numeric-ids --human-readable --partial --progress --relative --itemize-changes --stats --rsync-path="sudo rsync" --delete-during --files-from="${l_rsync_backup_file_list}" --exclude-from="${l_rsync_exclude_file_list}" -e "ssh -q -p ${l_rsync_ssh_port} -i ${l_rsync_ssh_key}" "${l_rsync_source}" "${l_rsync_dest}""
caffeinate -s "$l_rsync_bin" "${l_rsync_opts[#]}" --verbose --archive --recursive --numeric-ids --human-readable --partial --progress --relative --itemize-changes --stats --rsync-path="sudo rsync" --delete-during --files-from="${l_rsync_backup_file_list}" --exclude-from="${l_rsync_exclude_file_list}" -e "ssh -q -p ${l_rsync_ssh_port} -i ${l_rsync_ssh_key}" "${l_rsync_source}" "${l_rsync_dest}"
}

Related

Gitlab CI check if directory exists before pulling origin

I'm trying to deploy my flask application to AWS EC2 instance using gitlab ci runner.
.gitlab.ci.yml
stages:
- test
- deploy
test_app:
image: python:latest
stage: test
before_script:
- python -V
- pip install virtualenv
- virtualenv env
- source env/bin/activate
- pip install flask
script:
- cd flask-ci-cd
- python test.py
prod-deploy:
stage: deploy
only:
- master # Run this job only on changes for stage branch
before_script:
- mkdir -p ~/.ssh
- echo -e "$RSA_KEY" > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
script:
- bash .gitlab-deploy-prod.sh
environment:
name: deploy
.gitlab-deploy-prod.sh
#!/bin/bash
# Get servers list
set -f
# access server terminal
shell="ssh -o StrictHostKeyChecking=no ${SERVER_URL}"
git_token=$DEPLOY_TOKEN
echo "Deploy project on server ${SERVER_URL}"
if [ ${shell} -d "/flask-ci-cd" ] # check if directory exists
then
eval "${shell} cd flask-ci-cd && git clone https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd"
else
eval "${shell} git pull https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd && cd flask-ci-cd"
fi
Error: .gitlab-deploy-prod.sh: line 7: -o: command not found
How can i check if directory existing??
What i've tried.
#!/bin/bash
# Get servers list
set -f
# access server terminal
shell="ssh -o StrictHostKeyChecking=no ${SERVER_URL}"
git_token=$DEPLOY_TOKEN
eval "${shell}" # i thought gitlab would provide me with shell access
echo "Deploy project on server ${SERVER_URL}"
if [-d "/flask-ci-cd" ] # check if directory exists
then
eval "cd flask-ci-cd && git clone https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd"
else
eval "git pull https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd && cd flask-ci-cd"
fi
I've tried to log into the ssh shell before executing the scripts inside if else. But it doesn't works the way intended.
Your script has some errors.
Do not use eval. No, eval does not work that way. eval is evil
When storing a command to a variable, do not use normal variables. Use bash arrays instead to preserve "words".
Commands passed via ssh are double escaped. I would advise to prefer to use here documents, they're simpler to get the quoting right. Note the difference in expansion when the here document delimiter is quoted or not.
i thought gitlab would provide me with shell access No, without open standard input the remote shell will just terminate, as it will read EOF from input. No, it doesn't work that way.
Instead of doing many remote connection, just transfer the execution to remote side once and do all the work there.
Take your time and research how quoting and word splitting works in shell.
git_token=$DEPLOY_TOKEN No, set variables are not exported to remote shell. Either pass them manually or expand them before calling the remote side. (Or you could also use ssh -o SendEnv=git_token and configure remote ssh with AcceptEnv=git_token I think, never tried it).
Read documentation for the utilities you use.
No, git clone doesn't take branch name after url. You can specify branch with --branch or -b option. After url it takes directory name. See git clone --help. Same for git pull.
How can i check if directory existing??
Use bash arrays to store the command. Check if the directory exists just by executing the test command on the remote side.
shell=(ssh -o StrictHostKeyChecking=no "${SERVER_URL}")
if "${shell[#]}" [ -d "/flask-ci-cd" ]; then
...
In case of directory name with spaces I would go with:
if "${shell[#]}" sh <<'EOF'
[ -d "/directory with spaces" ]
EOF
then
Pass set -x to sh to see what's happening also on the remote side.
For your script, try rather to move the execution to remote side - there is little logic in making 3 separate connections. I say just
echo "Deploy project on server ${SERVER_URL}"
ssh -o StrictHostKeyChecking=no "${SERVER_URL}" bash <<EOF
if [ ! -d /flask-ci-cd ]; then
# Note: git_token is expanded on host side
git clone https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git /flask-ci-cd
fi
cd /flask-ci-cd
git pull
EOF
But instead of getting the quoting in some cases right, use declare -p and declare -f to transfer properly quoted stuff to remote side. That way you do not need case about proper quoting - it will work naturally:
echo "Deploy project on server ${SERVER_URL}"
work() {
if [ ! -d /flask-ci-cd ]; then
# Note: git_token is expanded on host side
git clone https://sbhusal123:"${git_token}"#gitlab.com/sbhusal123/flask-ci-cd.git /flask-ci-cd
fi
cd /flask-ci-cd
git pull
{
ssh -o StrictHostKeyChecking=no "${SERVER_URL}" bash <<EOF
$(declare -p git_token) # transfer variables you need
$(declare -f work) # transfer function you need
work # call the function.
EOF
Updated answer for future reads.
.gitlab-ci.yml
stages:
- test
- deploy
test_app:
image: python:latest
stage: test
before_script:
- python -V
- pip install virtualenv
- virtualenv env
- source env/bin/activate
- pip install flask
script:
- cd flask-ci-cd
- python test.py
prod-deploy:
stage: deploy
only:
- master
before_script:
- mkdir -p ~/.ssh
- echo -e "$RSA_KEY" > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
script:
- bash .gitlab-deploy-prod.sh
environment:
name: deploy
.gitlab-deploy-prod.sh
#!/bin/bash
# Get servers list
set -f
shell=(ssh -o StrictHostKeyChecking=no "${SERVER_URL}")
git_token=$DEPLOY_TOKEN
echo "Deploy project on server ${SERVER_URL}"
ssh -o StrictHostKeyChecking=no "${SERVER_URL}" bash <<EOF
if [ ! -d flask-ci-cd ]; then
echo "\n Cloning into remote repo..."
git clone https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git
# Create and activate virtualenv
echo "\n Creating virtual env"
python3 -m venv env
else
echo "Pulling remote repo origin..."
cd flask-ci-cd
git pull
cd ..
fi
# Activate virtual env
echo "\n Activating virtual env..."
source env/bin/activate
# Install packages
cd flask-ci-cd/
echo "\n Installing dependencies..."
pip install -r requirements.txt
EOF
There is a test command which is explicit about checking files and directories:
test -d "/flask-ci-cd" && eval $then_commands || eval $else_commands
Depending on the AWS instance I'd expect "test" to be available. I'd recommend putting the commands in variables. (e.g. eval $then_commands)

How to retain password for Rsync across the same script?

I have written this script:
#!/bin/bash
SSH_USER=${SSH_USER:=$USER}
for department in A B C E L M V
do
mkdir -p ./resources/${div}
rsync -Pruzh --copy-links \
${SSH_USER}#server:${department}/foo/files \
${SSH_USER}#server:${department}/foo/photos \
./resources/${department}/foo
rsync -Pruzh \
${SSH_USER}#server:${department}/bar/documents \
./resources/${department}/bar
done
It works perfect except that I have to write my password 14 times which is not really practical.
I have heard of ssh_agent but for some reasons it does not work on my WSL.
Is there any alternative that I can use to type my password only once?
If you are using openssh, then you can set up a master connection and reuse it with something like:
DEST="${SSH_USER}#server"
TMPL=/tmp/sshctl/"%L-%r#%h:%p"
mkdir -p /tmp/sshctl
if ! ssh -nNf -o ControlMaster=yes -o ControlPath="${TMPL}" "${DEST}"; then
echo "# Failed to setup SSH ControlMaster. Aborting."
exit
fi
# ...
rsync -e "ssh -o 'ControlPath=${TMPL}'" ... "${DEST}":... ...
rsync -e "ssh -o 'ControlPath=${TMPL}'" ... "${DEST}":... ...
# ...
ssh -O exit -o ControlPath="${TMPL}" "${DEST}"
Be sure to secure the socket.
Best practice would be to set up SSH key pairs for automated authentication; i.e. create an SSH key pair and copy the public key to the server where these files are located, then use the private key in the rsync command: rsync -Pruzh --copy-links -e "ssh -i /path/to/private.key" .... This is fairly simple, secure, and gets rid of the prompt.
You can also use a utility like sshpass to enter the password in the prompt, but that kind of approach is less secure.

rsync with ssh without using credentials stored in ~/.ssh/config

I have a script that transfers files. Everytime I run it It needs to connect to a different host. That's why I'm adding the host as parameter.
The script is executed as: ./transfer.sh <hostname>
#!/bin/bash -evx
SSH="ssh \
-o UseRoaming=no \
-o UserKnownHostsFile=/dev/null \
-o StrictHostKeyChecking=no \
-i ~/.ssh/privateKey.pem \
-l ec2-user \
${1}"
files=(
file1
file2
)
files="${files[#]}"
# this works
$SSH
# this does not work
rsync -avzh --stats --progress $files -e $SSH:/home/ec2-user/
# also this does not work
rsync -avzh --stats --progress $files -e $SSH ec2-user#$1:/home/ec2-user/
I can properly connect with the ssh connection stored in $SSH, but the rsync connection attempts fails because of the wrong key:
Permission denied (publickey).
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2]
What would be the correct syntax for the rsync connection?
Write set -x before the rsync line and watch how the arguments are expanded. I believe it will be wrong.
You need to enclose the ssh command with arguments (without hostname) into the quotes, otherwise the arguments will get passed to the rsync command and not to the ssh.
My solution after Jakuje pointed me in the right direction:
#!/bin/bash -evx
host=$1
SSH="ssh \
-o UseRoaming=no \
-o UserKnownHostsFile=/dev/null \
-o StrictHostKeyChecking=no \
-i ~/.ssh/privateKey.pem \
-l ec2-user"
files=(
file1
file2
)
files="${files[#]}"
# transfer all in one rsync connection
rsync -avzh --stats --progress $files -e "$SSH" $host:/home/ec2-user/
# launch setup script
$SSH $host ./setup.sh

Script for directory mirroring with inotifywait and ssh

I have a script that try to mirror a specific directory from a local server to a remote one. It looks like that:
inotifywait -mr --format '%w%f' -e close_write -e moved_to -e delete /mydir | \
while read FILECHANGE
do
if [ -f $FILECHANGE ]
then
rsync --bwlimit=4096 --progress --relative -vrae 'ssh -p 22' $FILECHANGE $REMOTEHOST:/
else
ssh -p 22 $REMOTEHOST "rm $FILECHANGE"
fi
done
In case of multiple create of files, as for example a touch command:
touch 1 2 3
The 3 files are well transfered.
But if I delete several files at once:
rm -f 1 2 3
Only the first 1 is deleted.
If I replace the ssh command by just an echo $FILECHANGE, the 3 files are well displayed in the console. So it seems the problem come from the ssh command, but I can't explain why and solve it.
Anyone as an idea?
Well, I found the issue: it seems that the ssh command was eating the output of the inotifywait command when run. So, to prevent that, I add the 0<&- redirection after the ssh, to close the stdin.
inotifywait -mr --format '%w%f' -e close_write -e moved_to -e delete /mydir | \
while read FILECHANGE
do
if [ -f $FILECHANGE ]
then
rsync --bwlimit=4096 --progress --relative -vrae 'ssh -p 22' $FILECHANGE $REMOTEHOST:/
else
ssh -p 22 $REMOTEHOST "rm $FILECHANGE" 0<&-
fi
done
Now it works.

Bash scripting rsync: rsync: link_stat (blah) failed: No such file or directory (2)

I am trying to write a simple bash script for my local (Mac OS X) machine to move files from a directory on my machine to a remote machine. This line is failing:
rsync --verbose --progress --stats --compress --rsh=ssh \
--recursive --times --perms --links --delete \
--exclude "*bak" --exclude "*~" \
/repository/* $DEV_SERVER:$REMOTE_DIR
$DEV_SERVER and $REMOTE_DIR are defined previously, and I echo them to verify they're accurate.
The error I'm getting is:
rsync: link_stat /Users/myusername/mycurrentdirectory failed: No such file or directory (2)
To note here is that rather than using the defined directory (/repository, which is in the root of the machine), it uses my working directory. What is causing this?
Check that your \ characters have no whitespace after them at the end of the line. This will cause BASH to not interpret the line wrap correctly, giving the rsync error above.
Remove the '*' from the source location, rsync knows to look in the inside of the directory if you specify the '/' in the end
like that:
rsync --verbose --progress --stats --compress --rsh=ssh --recursive --times --perms --links --delete --exclude "*bak" --exclude "*~" /repository/ $DEV_SERVER:$REMOTE_DIR
I was encountering the same error while doing some rsync work. I had the wrong character for specifying options which I must have gotten from copying and pasting the command from elsewhere:
−
rather than the correct character below:
-
The source path (here: /repository/*) does contain a wildcard (*).
There is some info in the man pages for rsync:
"Note that the expansion of wildcards on the commandline (*.c) into a list of files is handled by the shell before it runs rsync and not by rsync itself (exactly the same as all other posix-style programs)."
I.e. in case you put in bash double quotes "" around the (source path) including the wildcard symbol (*), the rsync commandwill succeed on command line interface, while inside a bash shell it will yield exactly the error described here.
rsync: link_stat "<source path>*" failed: No such file or directory (2)
=> make sure to not put the wild card(s) in quotes in an rsync source path.
This:
rsync --verbose --progress --stats --compress --rsh=ssh \
--recursive --times --perms --links --delete \
--exclude "*bak" --exclude "*~" \
/repository/* $DEV_SERVER:$REMOTE_DIR
should be this:
rsync --verbose --progress --stats --compress --rsh=ssh --recursive --times --perms --links --delete --exclude "*bak" --exclude "*~" /repository/* $DEV_SERVER:$REMOTE_DIR
Bash interprets the \ character differently than the command line, or perhaps there is an implicit non-whitespace character after it.

Resources