Bash -- Rsync variable in filename has spaces - bash

Been searching for hours and still cant figure this out :(
Anyway, im creating a script that will automatically rsync about 40 different directories to 40 other directories on another server. if you want to see the entire script you can view it here: http://pastebin.com/Nt3KKvx9
But the important bit is the for loop where i run rsync
for ((i=0; i<${#websys[#]}; i++))
do
localpath="/nusdata/staff/NUS/NUS/Systems/${kiskasys[$i]}"
remotepath="/home/www/html/nusalaska.com/html/systems/${websys[$i]}"
rsync -rlptnvz -s "$localpath" -e "ssh -p 50014" "nusak#webserver:$remotepath/"
done
The problem is that the array "kiskasys" has many directory names that have spaces in them (Example: '101 greenbrook').
I have tried making the array variables have single quotes around them, double quotes around them, escaped spaces like '\ ', and combinations of all three. I have also tried putting the $localpath in quotes, not in quotes, etc. etc.
I guess im just confused on how the -s (--protect-args) deals with the spaces and how I can get it to work in my situation.
The error output always looks something like the following:
rsync: change_dir "/nusdata/staff/NUS/NUS/101" failed: No such file or directory (2)
or
rsync: change_dir "/nusdata/staff/NUS/NUS/'101 greenbrook'" failed: No such file or directory (2)
Any help is appreciated!
Thanks
Found My Problem
In copying my code from my script to this page I accidently copied it wrong... however in copying it wrong, what i posted above was perfectly good code that works fine haha.
so what i posted above was the solution to my own problem.
The original had single quotes in the localpath variable like so:
localpath="'/nusdata/staff/NUS/NUS/Systems/${kiskasys[$i]}'"
and the single quotes was the problem. And for everyone's benefit here is an example output
echo ${kiskasys[1]}
#output would look like this:
101 greenbrook
Basically there are no special escape characters etc.

For what it's worth, I'm not able to replicate your problem. If I set localpath and remotepath to a directory with spaces:
localpath="/home/lars/tmp/so/dir1/a directory"
remotepath="/home/lars/tmp/so/dir2/a directory"
And then run your rsync command (modified slightly for my environment):
rsync -rlptvz -s "$localpath/" -e "ssh" "localhost:$remotepath/"
It Just Works. Here's the content of dir1:
dir1/a directory/file1
dir1/a directory/file3
dir1/a directory/file2
And after running rsync, dir2 looks like this:
dir2/a directory/file1
dir2/a directory/file3
dir2/a directory/file2
Can you show a specific command line that results in the errors you're seeing above? Maybe run the script with the -x flag and show exactly how localpath and remotepath are set.

Related

Rsync copies too many directories being executed via bash script

Originally I would like to sync directory (with all files and subdirectories) given in parameter in bash script.
I found this post: How can I recursively copy a directory into another and replace only the files that have not changed? which explains how to use rsync in similar case.
My bash script is quite simple and listed below:
#!/bin/bash
echo -e "Type the project to be deployed: \c "
read project
echo -e "* Deploying: $project *"
echo -e "Sync: /var/repo/released/$project"
echo -e " /var/www/released/$project"
rsync -pr /var/repo/released/$project /var/www/released/$project
As a result it copies everything within /released (there are many directories in there, let's say -projects-).
I would like to copy (sync) only project given in parameter.
Could you please advice how to do this?
When you call the script without an argument (which most likely is what you're doing since you interactively read the project name into the variable $project), the positional parameter $1 remains empty. Therefore the script will rsync the entire content of /var/repo/released/.
You need to replace $1 with $project in your script. Also, I'd recommend to put double quotes around the paths to avoid problems due to spaces in a directory name.
rsync -pr "/var/repo/released/$project" "/var/www/released/$project"

"Escaping" nightmare

I have been trying to put the following into a bash script:
scp /cygdrive/c/Program\ Files\ \(x86\)/Spiceworks/data/configurations/LA_$(date +%F)-firewall-5520 "sf-mlawler#10.21.1.212:/home/sf-mlawler/SBOX_Automation/SBOX_Dumps/08\ -\ Security/Firewalls"
...which works as expected via command line.
When I try to place it in a script, I'm obviously going to have to double escape the spaces, anything else? I'm unable to have success with the multitude of variations I've tried.
I would think:
scp /cygdrive/c/Program\\ Files\\ \(x86\)/Spiceworks/data/configurations/LA_\$(date\ +%F)-firewall-5520 "sf-mlawler#10.21.1.212:/home/sf-mlawler/SBOX_Automation/SBOX_Dumps/08\\ \\-\\ Security/Firewalls"
...would work, but it doesn't. Any suggestions, before I go grey and bald? Thanks
I wont waste the space with all variations I've tried, but I will say I have tried escaping almost everything and basically nothing and many in between with no success. When I receive a "No such file or Directory" I obviously escape until the file actually resolves, but even when I do not get a path error, the command is not successfully completing.
I do understand this is quite a specific case, but I imagine it will help others in the future: escaping spaces, in a bash script, using embedded expect (I have tested with a #!/bin/bash shebang and the embedded expect using expect -c ' .... ' as well as #!/usr/bin/expect using a simple spawn scp command with no change.
EDIT (Based on responses):
I should have mentioned I have tried quoting it...
Quoting the first (host) part gives me the error
can't read "(date +%F)": no such variable
while executing
"spawn scp "/cygdrive/c/Program Files (x86)/Spiceworks/data/configurations/LA_$(date +%F)-firewall-5520" "sf-mlawler#10.21.1.212:/home/sf-mlawler/SBOX_..."
...it is not a variable, it is a function giving me the current date in the format year-month-day
Quoting the destination section without escaping anything gives me the error
scp: ambiguous target
The same command that worked on your bash shell should work the same in your bash script. You shouldn't and cannot escape it twice.
If it doesn't work, make sure you're testing the script from the same terminal, ensure that the script contents and your command is identical, then paste the output from your terminal when you do:
$ scp /cygdrive/c/Program\ Files\ \(x86\)/Spiceworks/data/configurations/LA_$(date +%F)-firewall-5520 "sf-mlawler#10.21.1.212:/home/sf-mlawler/SBOX_Automation/SBOX_Dumps/08\ -\ Security/Firewalls"
(successful scp output here)
$ cat testscript
(testscript contents here, should be identical to above scp command)
$ bash testscript
(unsuccessful script/scp output here)
Just stick it in quotes and stop hitting yourself, as #that_other_guy said, you cannot, and shouldn't try and escape twice, wherever you got that idea, disregard it as a source of information. (my guess is it came from "thin air")
scp "/cygdrive/c/Program Files (x86)/Spiceworks/data/configurations/LA_$(date +%F)-firewall-5520" "sf-mlawler#10.21.1.212:/home/sf-mlawler/SBOX_Automation/SBOX_Dumps/08 - Security/Firewalls"
You could even give yourself some helpers:
export PROGFILESx86="/cygdrive/c/Program Files (x86)"
export SPICEWORKS_CONFIGS="${PROGFILESx86}/Spiceworks/data/configurations"
(add them to .bashrc, exec it.) and then do:
scp "${SPICEWORKS_CONFIGS}/LA_$(date +%F)-firewall-5520" "sf-mlawler#10.21.1.212:/home/sf-mlawler/SBOX_Automation/SBOX_Dumps/08 - Security/Firewalls"
I'd also be concerned the the files you are attempting to scp actually exist, wrap it up in a condition to verify they're there, it only makes sense when you're auto-generating a filename.
firewall_config="${SPICEWORKS_CONFIGS}/LA_$(date +%F)-firewall-5520"
if [ -e "${firewall_config}" ]
then
scp "${firewall_config}" "sf-mlawler#10.21.1.212:/home/sf-mlawler/SBOX_Automation/SBOX_Dumps/08 - Security/Firewalls"
else
echo "File doesn't exist: ${firewall_config}"
fi
Update:
Since you've updated your question, it's obviously the date call that's giving you the biggest problem, again, indirection saves you doing bizarre escape-char-yoga, just get the result into a var and use that.
fw_date=$(date +%F)
firewall_config="${SPICEWORKS_CONFIGS}/LA_${fw_date}-firewall-5520"

Mimic rsync command behaviour in shell script

I'm attempting to write my first short backup script. I'm using "plain" sh. When I issue the following command on the command line:
rsync /home/username/.* /home/username/backups
everything works well, as expected [IE ONLY my dotfiles and dot-directories are backed up].
But when I run a shell script with this line in it, rsync behaves strangely and ends up repeatedly and recursively backing up everything and my dotfiles. So I end up with a backup like this:
/backups/dotfiles/backups/dotfiles/backups/dotfiles/ etc etc etc.
How do I make use of the "*" in a this shell script to make rsync behave as expected?
Here is the whole script:
#!/bin/sh
#Backup dotfiles
echo 'Backing up dotfiles...'
rsync -avzu --delete --progress --exclude='.wine' --max-size='100M' /home/slowmo/.* /home/slowmo/backups/dotfiles/
#Compress into single file
echo 'Compressing dotfiles...'
tar cvzf /home/slowmo/backups/dotfiles.tar.gz /home/slowmo/backups/dotfiles/
I'm not sure if this is exactly the same, but I just ran across a similar issue.
I saw different results when trying to back up my hidden files from my home directory using both:
rsync from the shell - worked as expected, all dot files backed up when doing rsync .* /media/backup/dotfiles
rsync from a bash script. The same command as above didn't work, backed up the dot files, but also every other folder in the home directory.
On some investigation and reflection it turns out my shell (zsh) expands wildcards differently to bash, and this difference is crucial in this particular case. zsh does not expand .* to include . (this directory) and .. (parent directory), whereas bash does, so in the bash script rsync was given all hidden files, including the folder itself and the parent directory.
I've changed my script to instead call:
rsync .[^.]* /media/backup/dotfiles
which solves my problem - but do note that this pattern won't match regular files that start with .., like ..tmp
Some places to look for more information (not always in the context of rsync) include:
https://unix.stackexchange.com/a/476265/180385
https://www.cyberciti.biz/faq/linux-unix-appleosx-bsd-rsync-copy-hidden-dot-files/
https://unix.stackexchange.com/a/90075/180385
https://unix.stackexchange.com/a/685367/180385

cd(1) and variables with spaces (cygwin)

I've been having quite an unusual problem. In my .bashrc file, I have set a variable to a path name with spaces in it. I had a feeling this would cause problems, but I played around with setting an alias in a similar way and got it to work like so:
alias npp="\"/cygdrive/c/Program Files (x86)/Notepad++/notepad++.exe\""
Now, I thought I could use the same trick for my environment variable -
export PRO="\"/cygdrive/c/Program Files (x86)\""
This worked. Kind of.
[myName]
$ echo $PRO
"/cygdrive/c/Program Files (x86)"
[myName]
$ cd $PRO
bash: cd: "/cygdrive/c/Program: No such file or directory
I've tried placing an escape before the space with and without removing the double quotes, I've tried single quotes with and without the escape. I've tried using grave accents as quotes. I've tried just the escape, I've tried
export PRO=/cygdrive/c/Program\\\ Files\\\ \\\(x86\\\)
None of this has worked. The only thing that has was -
export PRO="/cygdrive/c/Program Files (x86)"
$ cd "$PRO"
Ultimately, I'm trying to find a way to make my variable work without placing quotes around it every single time I type the variable. Having run out of ideas entirely I came here hoping for someone to be able to help me.
I got around this with wildcards:
export PRO="/cygdrive/d/Program*Files/"
Due to Word Splitting if you do not quote $PRO that white space breaks your path into multiple words.
There's no way to use a variable without quoting it in your case.
Technically, I cannot answer your question, but a good workaround is to create a link to the folder you want, create a variable for the link, and then cd to that variable. It's an annoying second step and a pointless link but if its worth putting in your bash profile it might be worth the extra hassel.
Here's what I did:
ln -sf '/cygdrive/c/Users/Mic/Desktop/PENDING - Pics/' '/cygdrive/c/Users/Mic/mypics'
Then I put this in my bash_profile and now I can cd to $pic
pic=/cygdrive/c/Users/Eric/mypics/
Just enclose the program file in double quotes in this way
/cygdrive/c/"Program Files (x86)"
It works for me
I ran into a similar issue with a shell script evaluating a directory path in a variable without escaping the space in "Program Files". I got around this by running Cygwin as an administrator and creating a symlink. None of the answers here worked.
ln -s "/cygdrive/c/Program Files" /cygdrive/c/ProgramFiles

Using a filename with spaces with scp and chmod in bash

Periodically, I like to put files in the /tmp directory of my webserver to share out. What is annoying is that I must set the permissions whenever I scp the files. Following the advice from another question I've written a script which copies the file over, sets the permissions and then prints the URL:
#!/bin/bash
scp "$1" SERVER:"/var/www/tmp/$1"
ssh SERVER chmod 644 "/var/www/tmp/$1"
echo "URL is: http://SERVER/tmp/$1"
When I replace SERVER with my actual host, everything works as expected...until I execute the script with an argument including spaces. Although I suspect the solution might be to use $# I've not yet figured out how to get a spaced filename to work.
It turns out that what is needed is to escape the path which will be sent to the remote server. Bash thinks the quotes in SERVER:"/var/www/tmp/$1" are related to the $1 and removes them from the final output. If I try to run:
tmp-scp.sh Screen\ shot\ 2010-02-18\ at\ 9.38.35\ AM.png
Echoing we see it is trying to execute:
scp SERVER:/var/www/tmp/Screen shot 2010-02-18 at 9.38.35 AM.png
If instead the quotes are escaped literals then the scp command looks more like you'd expect:
scp SERVER:"/var/www/tmp/Screen shot 2010-02-18 at 9.38.35 AM.png"
With the addition of some code to truncate the path the final script becomes:
#!/bin/bash
# strip path
filename=${1##*/}
fullpath="$1"
scp "$fullpath" SERVER:\"/var/www/tmp/"$filename"\"
echo SERVER:\"/var/www/tmp/"$filename"\"
ssh SERVER chmod 644 \"/var/www/tmp/"$filename"\"
echo "URL is: http://SERVER/tmp/$filename"
The script looks right. My guess is that you need to quote the filename when you pass it into your script:
scp-chmod.sh "filename with spaces"
Or escape the spaces:
scp-chmod.sh filename\ with\ spaces
the easier way without worrying about spaces in file names, (besides quoting) is to rename your files to get rid of spaces before transferring. Or when you create the files, don't use spaces. You can make this your "best practice" whenever you name your files.

Resources