I have a set of scripts that I use to download files via FTP and then delete them from the server.
It works as follows:
for dir in `ls /volume1/auto_downloads/sync-complete`
do
if [ "x$dir" != *"x"* ]
then
echo "DIR: $dir"
echo "Moving out of complete"
# Soft delete from server so they don't get downloaded again
ssh dan#172.19.1.15 mv -v "'/home/dan/Downloads/complete/$dir'" /home/dan/Downloads/downloaded
Now $dir could be "This is a file" which works fine.
The problem I'm having is with special characters eg:
"This is (a) file"
This is a file & stuff"
tend to error:
bash: -c: line 0: syntax error near unexpected token `('
bash: -c: line 0: `mv -v '/home/dan/Downloads/complete/This is (a) file' /home/dan/Downloads/downloaded'
I can't work out how to escape it so both the variable gets evaluated and the command gets escaped properly. I've tried various combinations of escape characters, literal quotes, normal quotes, etc
If both sides are using bash, you can escape the arguments using printf '%q ', eg:
ssh dan#172.19.1.15 "$(printf '%q ' mv -v "/home/dan/Downloads/complete/$dir" /home/dan/Downloads/downloaded)"
You need to quote the whole expression ssh user#host "command":
ssh dan#172.19.1.15 "mv -v /home/dan/Downloads/complete/$dir /home/dan/Downloads/downloaded"
I'm confused, because your code as written works for me:
> dir='foo & bar (and) baz'
> ssh host mv -v "'/home/dan/Downloads/complete/$dir'" /home/dan/Downloads/downloaded
mv: cannot stat `/home/dan/Downloads/complete/foo & bar (and) baz': No such file or directory
For debugging, use set -vx at the top of the script to see what's going on.
Will Palmer's suggestion of using printf is great but I think it makes more sense to put the literal parts in printf's format.
That way, multi-command one-liners are more intuitive to write:
ssh user#host "$(printf 'mkdir -p -- %q && cd -- "$_" && tar -zx' "$DIR")"
One can use python shlex.quote(s) to
Return a shell-escaped version of the string s
docs
Related
I am deeply confused by Bash's Heredoc construct behaviour.
Here is what I am doing:
#!/bin/bash
user="some_user"
server="some_server"
address="$user"#"$server"
printf -v user_q '%q' "$user"
function run {
ssh "$address" /bin/bash "$#"
}
run << SSHCONNECTION1
sudo dpkg-query -W -f='${Status}' nano 2>/dev/null | grep -c "ok installed" > /home/$user_q/check.txt
softwareInstalled=$(cat /home/$user_q/check.txt)
SSHCONNECTION1
What I get is
cat: /home/some_user/check.txt: No such file or directory
This is very bizarre, because the file exists if I was to connect using SSH and check the following path.
What am I doing wrong? File is not executable, just a text file.
Thank you.
If you want the cat to run remotely, rather than locally during the heredoc's evaluation, escape the $ in the $(...):
softwareInstalled=\$(cat /home/$user_q/check.txt)
Of course, this only has meaning if some other part of your remote script then refers to "$softwareInstalled" (or, since it's in an unquoted heredoc, "\$softwareInstalled").
I need to run a multi-line bash command over ssh, all possible attempt exhausted but no luck --
echo "3. All files found, creating remote directory on the server."
ssh -t $id#$host bash -c "'
if [[ -d ~/_tmp ]]; then
rm -rf ~/_tmp/*
else
mkdir ~/_tmp
fi
'" ;
echo "4. Sending files ..."
scp ${files[#]} $id#$host:~/_tmp/ ;
Here is the output --
user#linux:/tmp$ ./remotecompile
1. Please enter your id:
user
2. Please enter the names of the files that you want to compile
(Filenames *must* be space separated):
test.txt
3. All files found, creating remote directory on the server.
Password:
Unmatched '.
Unmatched '.
Connection to host.domain.com closed.
Please note, I do not want to put every 2-3 lines of bash if-then-else-fi commands into separate files.
What is the right way to do it?
Use an escaped heredoc to have its literal contents passed through. (Without the escaping, ie. using just <<EOF, shell expansions would be processed locally -- making for more interesting corner cases if you used variables inside your remotely-run code).
ssh "$id#$host" bash <<'EOF'
if [[ -d ~/_tmp ]]; then
rm -rf ~/_tmp/*
else
mkdir ~/_tmp
fi
EOF
If you want to pass arguments, doing so in an unambiguously correct manner gets more interesting (since there are two separate layers of shell parsing involved), but the printf '%q' builtin saves the day:
args=( "this is" "an array" "of things to pass" \
"this next one is a literal asterisk" '*' )
printf -v args_str '%q ' "${args[#]}"
ssh "$id#$host" bash -s "$args_str" <<'EOF'
echo "Demonstrating local argument processing:"
printf '%q\n' "$#"
echo "The asterisk is $5"
EOF
This works for me:
ssh [hostname] '
if [[ -d ~/_tmp ]]; then
rm -rf ~/_tmp
else
mkdir ~/_tmp
fi
'
Hey guys I'm trying to run this code:
#!/bin/bash
sudo /usr/local/bin/sshpass -p pwd ssh -o stricthostkeychecking=no -p 11022 admin#$1.test.com<<EOI
i=1
while read line
do
location="sudo sed -n ${i}p /Users/Shared/$1.txt"
number="sudo sed -n ${i}p /Users/Shared/$2n.txt"
my_array=("${my_array[i]}" $line)
sudo cp /Applications/TEPS\ OS\ X\ Share\ Folder/MAIN\ IMAGES\ FOLDER\ ƒ/${location}${number} /Users/Shared/FYP/$number
sudo sips -Z 256 /Users/Shared/FYP/$number /Users/Shared/FYP/$number
((i++))
done </Users/Shared/$2.txt
exit
EOI
basically it reads a text file which gives the location of certain images, and will create a thumbnail of those images, which can be downloaded later. The problem is that I need the value of $i to set the values of $location and $number, but when I set the variable within the while loop the variables are not set. I've tried setting it locally and globally with single quotes, double quotes, passing through with the sshpass, exporting it -This works as a test but $i is of course unknown- tried placing brackets, curly braces, parentheses, escaping $, at this point I have exhausted my ideas, it's probably something incredibly simple, but I could use a fresh pair of eyes, any help is greatly appreciated!
EDIT:
Thanks to Charles Duffy for helping me clean it up so this is what I have now:
#!/bin/bash
sudo /usr/local/bin/sshpass -p support ssh -o stricthostkeychecking=no -p 11022 admin#$1.noerrpole.com<<'EOI'
i=1
while read -r line
do
location=sudo sed -n ${i}p "/Users/Shared/$1.txt"
number=sudo sed -n ${i}p "/Users/Shared/$2n.txt"
my_array+=( "$line" )
sudo cp "/Applications/TEPS\ OS\ X\ Share\ Folder/MAIN\ IMAGES\ FOLDER\ ƒ/${location}${number}" "/Users/Shared/FYP/$number"
sudo sips -Z 256 "/Users/Shared/FYP/$number" "/Users/Shared/FYP/$number"
((i++))
exit
done <"/Users/Shared/$2.txt"
EOI
But now $2 isn't getting passed through to the loop here's what I get back
1:bin Photo$ bash -x thumb npco2 20131216154714
+ sudo /usr/local/bin/sshpass -p support ssh -o stricthostkeychecking=no -p 11022 admin#npco2.noerrpole.com
Pseudo-terminal will not be allocated because stdin is not a terminal.
SHPA_12-16-2013/
sed: /Users/Shared/n.txt: No such file or directory
cp: /Applications/TEPS OS X Share Folder/MAIN IMAGES FOLDER ƒ/ is a directory (not copied).
Warning: /Users/Shared/FYP/ not a valid file - skipping
Warning: /Users/Shared/FYP/ not a valid file - skipping
Error 4: no file was specified
Try 'sips --help' for help using this tool
So where $2 should equal 20131216154714 it's returning an empty string like this
sed: /Users/Shared/n.txt: No such file or directory
The correct command would be
sed: /Users/Shared/20131216154714n.txt
The rest is just failing because $2 isn't passed.
Again thanks for the help!
ssh ... <<EOI does expansion on the local end, before starting ssh. Use ssh ... <<'EOI' to do expansions on the remote end.
If you want to pass arguments, use printf '%q ' to quote them so they survive remote unescaping intact:
printf -v quoted_args '%q ' "$one" "$two"
ssh user#host "bash -s - ${quoted_args}" <<<'EOI'
...
EOI
I have larger shell script which handles different things.
It will get it's own location by the following...
BASEDIR=`dirname $0`/..
BASEDIR=`(cd "$BASEDIR"; pwd)`
then BASEDIR will be used create other variables like
REPO="$BASEDIR"/repo
But the problem is that this shell script does not work if the path contains spaces where it is currently executed.
So the question is: Does exist a good solution to solve that problem ?
Be sure to double-quote anything that may contain spaces:
BASEDIR="`dirname $0`"
BASEDIR="`(cd \"$BASEDIR\"; pwd)`"
The answer is "Quotes everywhere."
If the path you pass in has a space in it then dirname $0 will fail.
$ cat quote-test.sh
#!/bin/sh
test_dirname_noquote () {
printf 'no quotes: '
dirname $1
}
test_dirname_quote () {
printf 'quotes: '
dirname "$1"
}
test_dirname_noquote '/path/to/file with spaces/in.it'
test_dirname_quote '/path/to/file with spaces/in.it'
$ sh quote-test.sh
no quotes: usage: dirname string
quotes: /path/to/file with spaces
Also, try this fun example
#!/bin/sh
mkdir -p /tmp/foo/bar/baz
cd /tmp/foo
ln -s bar quux
cd quux
cat >>find-me.sh<<"."
#!/bin/sh
self_dir="$(dirname $0)"
base_dir="$( (cd "$self_dir/.." ; pwd -P) )"
repo="$base_dir/repo"
printf 'self: %s\n' "$self_dir"
printf 'base: %s\n' "$base_dir"
printf 'repo: %s\n' "$repo"
.
sh find-me.sh
rm -rf /tmp/foo
Result when you run it:
$ sh example.sh
self: .
base: /tmp/foo
repo: /tmp/foo/repo
Quote your full variable like this:
REPO="$BASEDIR/repo"
There is no reliable and/or portable way to do this correctly.
See How do I determine the location of my script? as to why
The best answer is the following, which is still OS dependent
BASEDIR=$(readlink -f $0)
Then you can do things like REPO="$BASEDIR"/repo , just be sure to quote your variables as you did.
Works perfectly fine for me. How are you using REPO? What specifically "doesn't work" for you?
I tested
#!/bin/sh
BASEDIR=`dirname $0`/..
BASEDIR=`(cd "$BASEDIR"; pwd)`
REPO="$BASEDIR"/repo
echo $REPO
in a ".../a b/c d" directory. It outputs ".../a b/repo", as expected.
Please give the specific error that you are receiving... A "doesn't work" bug report is the least useful bug report, and every programmer absolutely hates it.
Using spaces in directory names in unix is always an issue so if they can be avoided by using underscores, this prevents lots of strange scripting behaviour.
I'm unclear why you are setting BASEDIR to be the parent directory of the directory containing the current script (..) and then resetting it after changing into that directory
The path to the directory should still work if it has ..
e.g. /home/trevor/data/../repo
BASEDIR=`dirname $0`/..
I think if you echo out $REPO it should have the path correctly assigned because you used quotes when assigning it but if you then try to use $REPO somewhere else in the script, you will need to use double quotes around that too.
e.g.
#!/bin/ksh
BASEDIR=`dirname $0`/..
$REPO="$BASEDIR"/repo
if [ ! -d ["$REPO"]
then
echo "$REPO does not exist!"
fi
Use speech marks as below:
BASEDIR=`dirname "${0}"`/..
The script uses scp to upload a file. That works.
Now I want to log in with ssh, cd to the directory that holds the uploaded file, do an md5sum on the file. The script keeps telling me that md5sum cannot find $LOCAL_FILE. I tried escaping: \$LOCAL_FILE. Tried quoting the EOI: <<'EOI'. I'm partially understanding this, that no escaping means everything happens locally. echo pwd unescaped gives the local path. But why can I do "echo $MD5SUM > $LOCAL_FILE.md5sum", and it creates the file on the remote machine, yet "echo md5sum $LOCAL_FILE > md5sum2" does not work? And if it the local md5sum, how do I tell it to work on the remote?
scp "files/$LOCAL_FILE" "$i#$i.567.net":"$REMOTE_FILE_PATH"
ssh -T "$i#$i.567.net" <<EOI
touch I_just_logged_in
cd $REMOTE_DIRECTORY_PATH
echo `date` > I_just_changed_directories
echo `whoami` >> I_just_changed_directories
echo `pwd` >> I_just_changed_directories
echo "$MD5SUM" >> I_just_changed_directories
echo $MD5SUM > $LOCAL_FILE.md5sum
echo `md5sum $LOCAL_FILE` > md5sum2
EOI
You have to think about when $LOCAL_FILE is being interpreted. In this case, since you've used double-quotes, it's being interpreted on the sending machine. You need instead to quote the string in such a way that $LOCAL_FILE is in the command line on the receiving machine. You also need to get your "here document" correct. What you show just sends the output to touch to the ssh.
What you need will look something like
ssh -T address <'EOF'
cd $REMOTE_DIRECTORY_PATH
...
EOF
The quoting rules in bash are somewhat arcane. You might want to read up on them in Mendel Cooper's Advanced Guide to Bash Scripting.