Trying to copy a remote file to my local system using scp in bash
I've obtained the filename that i want and assigned to variable, $lastModifiedFile,
but the problem is it contains spaces in the filename.
To use this variable with scp the spaces need to be escaped with backslashes.
Is there an easy way to format this variable and insert the correct escape character where necessary i.e on spaces?
#!/bin/bash
lastModifiedFile=$(sshpass -p 'passw0rd' ssh user#server 'ls -tr /path/output*| tail -n 1')
echo "$lastModifiedFile"
sshpass -p 'passw0rd' scp user#server:"$lastModifiedFile" /root/
This is the script output ..
[user#host ~]# ./script.sh
/path/outputSat Mar 09 151905 GMT 2013.html
scp: /path/outputSat: No such file or directory
scp: Mar: No such file or directory
scp: 09: No such file or directory
scp: 151905: No such file or directory
scp: GMT: No such file or directory
scp: 2013.html: No such file or directory
I'm looking for something like below, or even a simpler solution? ..
escapedFilename=""
for letter in $lastModifiedFile
if $letter == " "
$escapedFilename += "\ "
else
$escapedFilename += $letter
With a bit of leaning toothpick syndrome:
param=user#server:${lastModifiedFile// /\\ /}
sshpass -p 'passw0rd' scp "$param" /root/
EDIT: It seems scp does not like me. I needed an additional level of variable in testing ... :)
EDIT 2: According to OP's feedback the exact solution appears to consist of using ${lastModifiedFile// /\\ \\}
I just hope there are no other characters than space that need escaping in some other filenames :)
Use single quotes around the filename passed to the remote system so that it is not subject to word splitting.
lastModifiedFile=$(sshpass -p 'passw0rd' ssh user#server 'ls -tr /path/output*| tail -n 1')
echo "$lastModifiedFile"
sshpass -p 'passw0rd' scp user#server:"'$lastModifiedFile'" /root/
or
sshpass -p 'passw0rd' scp "user#server:'$lastModifiedFile'" /root/
Just do it like this:
sshpass -p 'passw0rd' scp 'user#server:$lastModifiedFile' /root/
Here are a couple of methods that should handle almost anything (not just spaces) in the filename. First, bash's printf builtin has a %q format that adds quotes/escapes/whatever to the string:
sshpass -p 'passw0rd' scp user#server:"$(printf %q "$lastModifiedFile")" /root/
Note, however, that this quotes/escapes/etc it suitably for interpretation by bash. If the remote computer's default shell is something else, this may not work in all cases.
Option two is simpler in principle (but a bit messy in practice), and should be compatible with more remote shells. Here, I enclose the filename in single-quotes, which should work for anything other than single-quotes within the filename. For those, I substitute '\'' (which ends the single-quoted string, adds an escaped single-quote, then restarts the single-quoted string):
repl="'\''" # Have to store this in a variable to work around a bash parsing oddity
sshpass -p 'passw0rd' scp user#server:"'${lastModifiedFile//\'/$repl}'" /root/
Related
I am deeply confused by Bash's Heredoc construct behaviour.
Here is what I am doing:
#!/bin/bash
user="some_user"
server="some_server"
address="$user"#"$server"
printf -v user_q '%q' "$user"
function run {
ssh "$address" /bin/bash "$#"
}
run << SSHCONNECTION1
sudo dpkg-query -W -f='${Status}' nano 2>/dev/null | grep -c "ok installed" > /home/$user_q/check.txt
softwareInstalled=$(cat /home/$user_q/check.txt)
SSHCONNECTION1
What I get is
cat: /home/some_user/check.txt: No such file or directory
This is very bizarre, because the file exists if I was to connect using SSH and check the following path.
What am I doing wrong? File is not executable, just a text file.
Thank you.
If you want the cat to run remotely, rather than locally during the heredoc's evaluation, escape the $ in the $(...):
softwareInstalled=\$(cat /home/$user_q/check.txt)
Of course, this only has meaning if some other part of your remote script then refers to "$softwareInstalled" (or, since it's in an unquoted heredoc, "\$softwareInstalled").
I am trying to rename all files in a remote directory over SSH or SFTP. The rename should convert the file into a date extension, for example .txt into .txt.2016-05-25.
I have the following command to loop each .txt file and try to rename, but am getting an error:
ssh $user#$server "for FILENAME in $srcFolder/*.txt; do mv $FILENAME $FILENAME.$DATE; done"
The error I am getting is:
mv: missing destination file operand after `.20160525_1336'
I have also tried this over SFTP with no such luck. Any help would be appreciated!
You need to escape (or single-quote) the $ of variables in the remote shell. It's also recommended to quote variables that represent file paths:
ssh $user#$server "for FILENAME in '$srcFolder'/*.txt; do mv \"\$FILENAME\" \"\$FILENAME.$DATE\"; done"
Try this:
By using rename (perl tool):
ssh user#host /bin/sh <<<$'
rename \047use POSIX;s/$/strftime(".%F",localtime())/e\047 "'"$srcFolder\"/*.txt"
To prepare/validate your command line, replace ssh...bin/sh by cat:
cat <<<$'
rename \047use POSIX;s/$/strftime(".%F",localtime())/e\047 "'"$srcFolder\"/*.txt"
will render something like:
rename 'use POSIX;s/$/strftime(".%F",localtime())/e' "/tmp/test dir"/*.txt
And you could localy try (ensuring $srcFolder contain a path to a local test folder):
/bin/sh <<<$'
rename \047use POSIX;s/$/strftime(".%F",localtime())/e\047 "'"$srcFolder\"/*.txt"
Copy of your own syntax:
ssh $user#$server /bin/sh <<<'for FILENAME in "'"$srcFolder"'"/*.txt; do
mv "$FILENAME" "$FILENAME.'$DATE'";
done'
Again, you could locally test your inline script:
sh <<<'for FILENAME in "'"$srcFolder"'"/*.txt; do
mv "$FILENAME" "$FILENAME.'$DATE'";
done'
or preview by replacing sh by cat.
When using/sending variables over SSH, you need to be careful what is a local variable and which is a remote variable. Remote variables must be escaped; otherwise they will be interpreted locally versus remotely as you intended. Other characters also need to be escaped such as backticks. The example below should point you in the right direction:
Incorrect
user#host1:/home:> ssh user#host2 "var=`hostname`; echo \$var"
host1
Correct
user#host1:/home:> ssh user#host2 "var=\`hostname\`; echo \$var"
host2
This question already has answers here:
How do I escape spaces in path for scp copy in Linux?
(12 answers)
Closed 3 years ago.
I'm trying to mirror copy the whole directory from one cluster to another cluster right now. But it fails when there is a space in the name and I can't figure out how to solve this problem because it seems scp thinks I'm copying multiple files. I'm using a variable to flag the path that I need so it seems it would not be easily solved by adding a back slash.
This is the code that I'm using:
if ssh user#ip -i key test -d "'$current_dir'"; then
echo "Directory exists. Ready to copy $dir_name."
scp -i key -r "$current_dir/$dir_name" user#$ip:"$current_dir/$dir_name"
else
echo Directory doesn\'t exist. Making a new directory.
ssh user#$ip -i key mkdir "'$current_dir'"
scp -i key -r "$current_dir/$dir_name" user#$ip:"$current_dir/$dir_name"
fi
I have tried single quote, double quotes and single quote with double quotes, but none of them works. Can anyone help me solve it? By the way, the mkdir statement in the code works.
As detailed in this answer, you need to use two sets of quotes, because the filename is interpreted twice: once by the local computer and once by the remote computer.
scp -i key -r "$current_dir/$dir_name" user#$ip:"'$current_dir/$dir_name'"
You could escape the spaces in your directory name with sed :
current_dir = "/usr/bin/Directory With Spaces"
escaped_current_dir = `sed s/ /\\ /g $current_dir`
# escaped_current_dir = "/usr/bin/Directory\ With\ Spaces"
The string with escaped spaces will be read as one string, and one string only, which is what we want.
You need to set the field separator to a space.
OIFS=$IFS #save the original
IFS=$"\ " #set to space
current_dir="some dir with spaces"
scp_current_dir="some\\ dir\\ with\\ spaces"
dir_name="another dir with spaces"
scp_dir_name="another\\ dir\\ with\\ spaces"
if ssh user#ip -i key test -d $current_dir; then
echo "Directory exists. Ready to copy $dir_name."
scp -i key -r "$current_dir/$dir_name" user#$ip:"$scp_current_dir/$scp_dir_name"
else
echo Directory doesn\'t exist. Making a new directory.
ssh user#$ip -i key mkdir $current_dir
scp -i key -r "$current_dir/$dir_name" user#$ip:"$scp_current_dir/$scp_dir_name"
fi
IFS=$OIFS #reset it back to the original
I am writing a shell script and I need to SSH into a server, perform some actions, and then exit.
To do this, I am using code such as below:
ssh -t username#server '
cd uploads/; \
tar -xvzf torrent.tar.gz; \
'
However, I need to use a variable like so:
DIR="uploads/";
ssh -t username#server '
cd $DIR; \
tar -xvzf torrent.tar.gz; \
'
This doesn't seem to work because obviously the cd isn't being executed until the SSH connection is made, and by then there is no $DIR variable (my guess). However, is there any way I could use a variable?
Perhaps a better question is, is there a better way I could lay out my script to perform actions once the SSH connection is made? I am having to be careful, escaping apostrophes, and at one point I am actually SSHing to another server from inside an SSH connection. This is ugly code!
Edit: Just read that if I use " instead of ', the variable will work. However my question still stands about formatting?
Use double quotes to allow expansion. e.g. try this:
echo "$DIR"
vs
echo '$DIR'
and note the result.
It's worth wrapping this in a shell script thus:
#!/bin/bash -x
# ...
and the -x will output shell expansions etc. It makes life very easy for debugging.
Rather than using the above means to feed in commands, you can use a heredoc. e.g.
ssh username#host <<EOF
ls
EOF
will execute 'ls' on the remote host.
instead of single quotes (') why don't you use double quotes (") then the $DIR should be expanded
i think you have to write :
ssh username#server "( cd $DIR; tar -xvzf torrent.tar.gz; )"
this should execute after connection..
if not try with (" instead of "( and obviusly then close
I am trying to upload multiple files from one folder to a ftp site and wrote this script:
#!/bin/bash
for i in '/dir/*'
do
if [-f /dir/$i]; then
HOST='x.x.x.x'
USER='username'
PASSWD='password'
DIR=archives
File=$i
ftp -n $HOST << END_SCRIPT
quote USER $USER
quote PASS $PASSWD
ascii
put $FILE
quit
END_SCRIPT
fi
It is giving me following error when I try to execute:
username#host:~/Documents/Python$ ./script.sh
./script.sh: line 22: syntax error: unexpected end of file
I can't seem to get this to work. Any help is much appreciated.
Thanks,
Mayank
It's complaining because your for loop does not have a done marker to indicate the end of the loop. You also need more spaces in your if:
if [ -f "$i" ]; then
Recall that [ is actually a command, and it won't be recognized if it doesn't appear as such.
And... if you single quote your glob (at the for) like that, it won't be expanded. No quotes there, but double quotes when using $i. You probably also don't want to include the /dir/ part when you use $i as it's included in your glob.
If I'm not mistaken, ncftp can take wildcard arguments:
ncftpput -u username -p password x.x.x.x archives /dir/*
If you don't already have it installed, it's likely available in the standard repo for your OS.
First, the literal, fixing-your-script answer:
#!/bin/bash
# no reason to set variables that don't change inside the loop
host='x.x.x.x'
user='username'
password='password'
dir=archives
for i in /dir/*; do # no quotes if you want the wildcard to be expanded!
if [ -f "$i" ]; then # need double quotes and whitespace here!
file=$i
ftp -n "$host" <<END_SCRIPT
quote USER $user
quote PASS $password
ascii
put $file $dir/$file
quit
END_SCRIPT
fi
done
Next, the easy way:
lftp -e 'mput -a *.i' -u "$user,$password" "ftp://$host/"
(yes, lftp expands the wildcard internally, rather than expecting this to be done by the outer shell).
First of all my apologies in not making myself clear in the question. My actual task was to copy a file from local folder to a SFTP site and then move the file to an archive folder. Since the SFTP is hosted by a vendor I cannot use the key sharing (vendor limitation. Also, SCP will require password entering if used in a shell script so I have to use SSHPASS. SSHPASS is in the Ubuntu repo however for CentOS it needs to be installed from here
Current thread and How to run the sftp command with a password from Bash script? did gave me better understanding on how to write the script and I will share my solution here:
#!/bin/bash
#!/usr/bin
for i in /dir/*; do
if [ -f "$i" ]; then
file=$i
export SSHPASS=password
sshpass -e sftp -oBatchMode=no -b - user#ftp.com << !
cd foldername/foldername
put $file
bye
!
mv $file /somedir/test
fi
done
Thanks everyone for all the responses!
--Mayank