"Escaping" nightmare - bash

I have been trying to put the following into a bash script:
scp /cygdrive/c/Program\ Files\ \(x86\)/Spiceworks/data/configurations/LA_$(date +%F)-firewall-5520 "sf-mlawler#10.21.1.212:/home/sf-mlawler/SBOX_Automation/SBOX_Dumps/08\ -\ Security/Firewalls"
...which works as expected via command line.
When I try to place it in a script, I'm obviously going to have to double escape the spaces, anything else? I'm unable to have success with the multitude of variations I've tried.
I would think:
scp /cygdrive/c/Program\\ Files\\ \(x86\)/Spiceworks/data/configurations/LA_\$(date\ +%F)-firewall-5520 "sf-mlawler#10.21.1.212:/home/sf-mlawler/SBOX_Automation/SBOX_Dumps/08\\ \\-\\ Security/Firewalls"
...would work, but it doesn't. Any suggestions, before I go grey and bald? Thanks
I wont waste the space with all variations I've tried, but I will say I have tried escaping almost everything and basically nothing and many in between with no success. When I receive a "No such file or Directory" I obviously escape until the file actually resolves, but even when I do not get a path error, the command is not successfully completing.
I do understand this is quite a specific case, but I imagine it will help others in the future: escaping spaces, in a bash script, using embedded expect (I have tested with a #!/bin/bash shebang and the embedded expect using expect -c ' .... ' as well as #!/usr/bin/expect using a simple spawn scp command with no change.
EDIT (Based on responses):
I should have mentioned I have tried quoting it...
Quoting the first (host) part gives me the error
can't read "(date +%F)": no such variable
while executing
"spawn scp "/cygdrive/c/Program Files (x86)/Spiceworks/data/configurations/LA_$(date +%F)-firewall-5520" "sf-mlawler#10.21.1.212:/home/sf-mlawler/SBOX_..."
...it is not a variable, it is a function giving me the current date in the format year-month-day
Quoting the destination section without escaping anything gives me the error
scp: ambiguous target

The same command that worked on your bash shell should work the same in your bash script. You shouldn't and cannot escape it twice.
If it doesn't work, make sure you're testing the script from the same terminal, ensure that the script contents and your command is identical, then paste the output from your terminal when you do:
$ scp /cygdrive/c/Program\ Files\ \(x86\)/Spiceworks/data/configurations/LA_$(date +%F)-firewall-5520 "sf-mlawler#10.21.1.212:/home/sf-mlawler/SBOX_Automation/SBOX_Dumps/08\ -\ Security/Firewalls"
(successful scp output here)
$ cat testscript
(testscript contents here, should be identical to above scp command)
$ bash testscript
(unsuccessful script/scp output here)

Just stick it in quotes and stop hitting yourself, as #that_other_guy said, you cannot, and shouldn't try and escape twice, wherever you got that idea, disregard it as a source of information. (my guess is it came from "thin air")
scp "/cygdrive/c/Program Files (x86)/Spiceworks/data/configurations/LA_$(date +%F)-firewall-5520" "sf-mlawler#10.21.1.212:/home/sf-mlawler/SBOX_Automation/SBOX_Dumps/08 - Security/Firewalls"
You could even give yourself some helpers:
export PROGFILESx86="/cygdrive/c/Program Files (x86)"
export SPICEWORKS_CONFIGS="${PROGFILESx86}/Spiceworks/data/configurations"
(add them to .bashrc, exec it.) and then do:
scp "${SPICEWORKS_CONFIGS}/LA_$(date +%F)-firewall-5520" "sf-mlawler#10.21.1.212:/home/sf-mlawler/SBOX_Automation/SBOX_Dumps/08 - Security/Firewalls"
I'd also be concerned the the files you are attempting to scp actually exist, wrap it up in a condition to verify they're there, it only makes sense when you're auto-generating a filename.
firewall_config="${SPICEWORKS_CONFIGS}/LA_$(date +%F)-firewall-5520"
if [ -e "${firewall_config}" ]
then
scp "${firewall_config}" "sf-mlawler#10.21.1.212:/home/sf-mlawler/SBOX_Automation/SBOX_Dumps/08 - Security/Firewalls"
else
echo "File doesn't exist: ${firewall_config}"
fi
Update:
Since you've updated your question, it's obviously the date call that's giving you the biggest problem, again, indirection saves you doing bizarre escape-char-yoga, just get the result into a var and use that.
fw_date=$(date +%F)
firewall_config="${SPICEWORKS_CONFIGS}/LA_${fw_date}-firewall-5520"

Related

Cannot properly execute bash script because of a redirection in an environment where root is the owner

My script is executable and I run it as sudo. I tried many workarounds and alternatives to the ">>" operator but nothing seemed to work properly.
My script:
#! /bin/bash
if [[ -z "$1" || -z "$2" ]]; then
exit 1
else
root=$1
fileExtension=$2
fi
$(sudo find $root -regex ".*\.${fileExtension}") >> /home/mux/Desktop/AllFilesOf${fileExtension}.txt
I tried tee, sed and dd of, I also tried running it with bash -c or in sudo -i , nothing worked. Either i get an empty file or a Permission denied error.
I searched thoroughly and read many command manuals but I can't get it to work
The $() operator performs command substitution. When the overall command line is expanded, the command within the parentheses is executed, and the whole construct is replaced with the command's output. After all expansions are performed, the resulting line is executed as a command.
Consider, then, this simplified version of your command:
$(find /etc -regex ".*\.conf") >> /home/mux/Desktop/AllFilesOfconf.txt
On my system that will expand to a ginormous command of the form
/etc/rsyslog.conf /etc/pnm2ppa.conf ... /etc/updatedb.conf >> /home/mux/Desktop/AllFilesOfconf.txt
Note at this point that the redirection is separate from, and therefore independent of, the command in the command substitution. Expanding the command substitution therefore does not cause anything to be written to the target file.
But we're not done! That was just the expansion. Bash now tries to execute the result as a command. In particular, in the above example it tries to execute /etc/rsyslog.conf as a command, with all the other file names as arguments, and with output redirected as specified. But /etc/rsyslog.conf is not executable, so that will fail, producing a "permission denied" message. I'm sure you can extrapolate from there what effects different expansions would produce.
I don't think you mean to perform a command substitution at all, but rather just to run the command and redirect its output to the given file. That would simply be this:
sudo find $root -regex ".*\.${fileExtension}" >> /home/mux/Desktop/AllFilesOf${fileExtension}.txt
Update:
As #CharlesDuffy observed, the redirection in that case is performed with the permissions of the user / process running the script, just as it is in your original example. I have supposed that that is intentional and correct -- i.e. that the script is being run by user 'mux' or by another user that has access to mux's Desktop directory and to any existing file in it that the script might try to create or update. If that is not the case, and you need the redirection, too, to be privileged, then you can achieve it like so:
sudo -s <<END
find $root -regex ".*\.${fileExtension}" >> /home/mux/Desktop/AllFilesOf${fileExtension}.txt
END
That runs an interactive shell via sudo, with its input is redirected from the heredoc. The variable expansions are performed in the host shell from which sudo is executed. In this case the redirection is performed with the identity obtained via sudo, which affects access control, as well as ownership of the file if a new one is created. You could add a chown command if you don't want the output files to be owned by root.

How do I call rename successfully from a bash script on Ubuntu?

I have a bash script #!/usr/bin/env bash that is called part of a make process. This script creates a directory with the files pertinent to a realise and then tars them up. I would like to take a copy of the directory and rename some of the files to replace the version identifier with the word "latest". This will make it simple to script the acquisition of the latest file from a web-server. When I run my script, the call to rename seems to do nothing, why is that?
#!/usr/bin/env bash
DATE_NOW="$(date +'%Y%m%d')"
product_id_base="$1"
firmware_dir="${product_id_base}-full-${DATE_NOW}"
# ...rest of file ommitted to protest the innocent
# It creates and fills the ${firmware_dir} with some files that end in
# -$DATE_NOW.<extention> and I would like to rename the copies of them so that they end in
# -latest.<extention>
cp -a "./${firmware_dir}" "./${product_id_base}-full-latest"
# see what there is in pwd
cd "./${product_id_base}-full-latest"
list_output=`ls`
echo $list_output
# Things go OK until this point.
replacment="'s/${DATE_NOW}/latest/'"
rename_path=$(which rename)
echo $replacment
perl $rename_path -v $replacment *
echo $cmd
pwd
$cmd
echo "'s/-${DATE_NOW}/-latest/g'" "${product_id_base}-*"
echo $a
# check what has happened
list_output=`ls`
echo $list_output
I call the above with ./rename.sh product-id and get the expected output from ls that indicates the present working directory is the one full of files that I want renamed.
$ ./rename.sh product-id ET-PIC-v1.1.dat ET-PIC-v1.1.hex
product-id-20160321.bin product-id-20160321.dat
product-id-20160321.elf product-id-20160321.gz 's/20160321/latest/'
/home/thomasthorne/work/product-id/build/product-id-full-latest
's/-20160321/-latest/g' product-id-*
ET-PIC-v1.1.dat ET-PIC-v1.1.hex product-id-20160321.bin
product-id-20160321.dat product-id-20160321.elf product-id-20160321.gz
What I hopped to see was some renamed files. When I directly call the rename function from a terminal emulator I see the rename occur.
~/work/product-id/build/product-id-full-latest$ rename -vn
's/-20160321/-latest/g' * product-id-20160321.bin renamed as
product-id-latest.bin product-id-20160321.dat renamed as
product-id-latest.dat product-id-20160321.elf renamed as
product-id-latest.elf ...
I have tried a few variations on escaping the strings, using ` or $(), removing all the substitutions from the command line. So far nothing has worked so I must be missing something fundamental.
I have read that #!/usr/bin/env bash behaves much like #!/bin/bash so I don't think that is at play. I know that Ubuntu and Debian have different versions of the rename script to some other distributions and I am running on Ubuntu. That lead me to try calling perl /usr/bin/rename ... instead of just rename but that seems to have made no perceivable difference.
This string:
replacment="'s/${DATE_NOW}/latest/'"
will be kept exactly the same because you put it between single quotes.
Have you tried with:
replacment="s/${DATE_NOW}/latest/"
This one worked on my Ubuntu, without perl:
$ ./test_script
filename_20160321 renamed as filename_latest
filename2_20160321 renamed as filename2_latest
filename3_20160321 renamed as filename3_latest
test_script content being:
#!/bin/bash
DATE_NOW="$(date +'%Y%m%d')"
replacment="s/${DATE_NOW}/latest/"
rename -v $replacment *

Using the bash eval builtin to allow ini files to have command strings in them

I've been putting together a bash script that takes an ini file (with a format that I've been developing alongside the script) and reads through the file, performing the actions specified.
One of the functions in the ini format allows for a shell command to be passed in and run using eval. I'm running into a problem when the commands contain a variable name.
eval (or the shell in general) doesn't seem to be substituting the values correctly and most of the time it seems to replace all the variable names with blanks, breaking the command. Subshells to create a string output seem to have the same problem.
The strange part is that this worked on my development machine (Running linux mint 13), but when I moved the script to the target machine running CentOS 5.8, these issues showed up.
Some examples of code I read in from the ini file:
shellcmd $toolspath/program > /path/file
shellcmd parsedata=$( cat /path/file )
These go through a script function that strips off the leading shellcmd and then evals the string using
eval ${scmd}
Any ideas on what might be causing the weird behavior and anything I can try to resolve the problem? My ultimate goal here is to have the ability to read in a line from a file and have my script execute it and be able to correctly handle script variables from the read in command.
Using Bash 3.2.25 (CentOS 5) I tried this, and it works fine:
toolspath='/bin'
while read prefix scmd
do
if [[ $prefix == 'shellcmd' ]]
then
echo "Evaluating: <$scmd>"
eval ${scmd}
else
echo "$prefix ignored"
fi
done < ini
with:
shellcmd $toolspath/ls > /home/user1/file
shellcmd parsedata=$( cat /home/user1/file )
shellcmd echo $parsedata
I obviously had to set paths. Most likely you had to change the paths when you switched machines. Do your paths have embedded spaces?
How did you transfer the files? Did you perchance go via Windows? On a whim I did a unix2dos on the ini file and I got similar symptoms to that you describe. That's my best guess.
I found a suitable alternative for the one command that what causing this issue so I'll mark this question as solved.
From all my investigation it appears that I've discovered an obscure bug in the bash shell. The particular command I was trying to eval returned a terminal code in it's output, and because the shell was in a read loop with input redirected from a file it resulted in some strange behavior. My solution was to move the call to this particular command outside of the read loop. It still doesn't solve the root problem, which I believe to be a bug in the bash shell. Hope this will help someone else who has run into this same (obscure) issue.

Bash -- Rsync variable in filename has spaces

Been searching for hours and still cant figure this out :(
Anyway, im creating a script that will automatically rsync about 40 different directories to 40 other directories on another server. if you want to see the entire script you can view it here: http://pastebin.com/Nt3KKvx9
But the important bit is the for loop where i run rsync
for ((i=0; i<${#websys[#]}; i++))
do
localpath="/nusdata/staff/NUS/NUS/Systems/${kiskasys[$i]}"
remotepath="/home/www/html/nusalaska.com/html/systems/${websys[$i]}"
rsync -rlptnvz -s "$localpath" -e "ssh -p 50014" "nusak#webserver:$remotepath/"
done
The problem is that the array "kiskasys" has many directory names that have spaces in them (Example: '101 greenbrook').
I have tried making the array variables have single quotes around them, double quotes around them, escaped spaces like '\ ', and combinations of all three. I have also tried putting the $localpath in quotes, not in quotes, etc. etc.
I guess im just confused on how the -s (--protect-args) deals with the spaces and how I can get it to work in my situation.
The error output always looks something like the following:
rsync: change_dir "/nusdata/staff/NUS/NUS/101" failed: No such file or directory (2)
or
rsync: change_dir "/nusdata/staff/NUS/NUS/'101 greenbrook'" failed: No such file or directory (2)
Any help is appreciated!
Thanks
Found My Problem
In copying my code from my script to this page I accidently copied it wrong... however in copying it wrong, what i posted above was perfectly good code that works fine haha.
so what i posted above was the solution to my own problem.
The original had single quotes in the localpath variable like so:
localpath="'/nusdata/staff/NUS/NUS/Systems/${kiskasys[$i]}'"
and the single quotes was the problem. And for everyone's benefit here is an example output
echo ${kiskasys[1]}
#output would look like this:
101 greenbrook
Basically there are no special escape characters etc.
For what it's worth, I'm not able to replicate your problem. If I set localpath and remotepath to a directory with spaces:
localpath="/home/lars/tmp/so/dir1/a directory"
remotepath="/home/lars/tmp/so/dir2/a directory"
And then run your rsync command (modified slightly for my environment):
rsync -rlptvz -s "$localpath/" -e "ssh" "localhost:$remotepath/"
It Just Works. Here's the content of dir1:
dir1/a directory/file1
dir1/a directory/file3
dir1/a directory/file2
And after running rsync, dir2 looks like this:
dir2/a directory/file1
dir2/a directory/file3
dir2/a directory/file2
Can you show a specific command line that results in the errors you're seeing above? Maybe run the script with the -x flag and show exactly how localpath and remotepath are set.

Using a filename with spaces with scp and chmod in bash

Periodically, I like to put files in the /tmp directory of my webserver to share out. What is annoying is that I must set the permissions whenever I scp the files. Following the advice from another question I've written a script which copies the file over, sets the permissions and then prints the URL:
#!/bin/bash
scp "$1" SERVER:"/var/www/tmp/$1"
ssh SERVER chmod 644 "/var/www/tmp/$1"
echo "URL is: http://SERVER/tmp/$1"
When I replace SERVER with my actual host, everything works as expected...until I execute the script with an argument including spaces. Although I suspect the solution might be to use $# I've not yet figured out how to get a spaced filename to work.
It turns out that what is needed is to escape the path which will be sent to the remote server. Bash thinks the quotes in SERVER:"/var/www/tmp/$1" are related to the $1 and removes them from the final output. If I try to run:
tmp-scp.sh Screen\ shot\ 2010-02-18\ at\ 9.38.35\ AM.png
Echoing we see it is trying to execute:
scp SERVER:/var/www/tmp/Screen shot 2010-02-18 at 9.38.35 AM.png
If instead the quotes are escaped literals then the scp command looks more like you'd expect:
scp SERVER:"/var/www/tmp/Screen shot 2010-02-18 at 9.38.35 AM.png"
With the addition of some code to truncate the path the final script becomes:
#!/bin/bash
# strip path
filename=${1##*/}
fullpath="$1"
scp "$fullpath" SERVER:\"/var/www/tmp/"$filename"\"
echo SERVER:\"/var/www/tmp/"$filename"\"
ssh SERVER chmod 644 \"/var/www/tmp/"$filename"\"
echo "URL is: http://SERVER/tmp/$filename"
The script looks right. My guess is that you need to quote the filename when you pass it into your script:
scp-chmod.sh "filename with spaces"
Or escape the spaces:
scp-chmod.sh filename\ with\ spaces
the easier way without worrying about spaces in file names, (besides quoting) is to rename your files to get rid of spaces before transferring. Or when you create the files, don't use spaces. You can make this your "best practice" whenever you name your files.

Resources