I have a list of directory names.
I want to scp into a remote machine, go into each of my directory names and copy a file back to my local computer.
I so far have:
while read line
do
scp remote_machine:/home/$line/$line.dat ./local
done < file_with_directory_names.txt
I have authorisation keys set up so that I don't have to enter the password each time - but this method does login to the remote machine for every file it transfers. I imagine that there is a much better way than this.
You can specify multiple files in a single scp argument by separating them with spaces; you just need to make sure it's one argument to scp itself. This should work in your case:
scp "remote_machine:$(
sed 's,.*,/home/&/&.dat,' file_with_directory_names.txt | xargs)" ./local
The sed command sticks the /home/ prefix and name.dat suffix on each line; the xargs outputs all the resulting pathnames on a single line separated by spaces. Plug that all into the source argument after the remote_machine: part, all inside double quotes so it's still a single argument to scp, and you're good to go.
Related
I need to archive files with sys time on remote server but name of the file contains "SPACE" and special character. So below commands are not working.
FileName="BBB ABC#textfile.xml"
ts=`date +"%m%d%Y%H%M%S"`
ssh remoteid#remoteserver "'mv /upload/hotfolders/in/"$FileName"
/upload/hotfolders/Archive/${FileName}_${ts}'"
But above command is failing with below error.
bash: mv /upload/hotfolders/in/BBB ABC#textfile.xml /upload/hotfolders/Archive/BBB ABC#textfile.xml_01282019050200: No
such file or directory
In the original provided code:
ssh remoteid#remoteserver 'cd /upload/hotfolders/; mv "$FileName"
/upload/hotfolders/Archive/"${FileName}_${ts}"'
the outermost ' are used on the local filesystem to keep all the commands as a single argument to ssh. However, this means that $FileName, etc are not expanded locally! Instead, the unexpanded strings are passed verbatim to the remoteserver, where a shell is started to run the command. $FileName, etc, are then expanded there. Because they are not defined there (probably), the expansion fails to produce anything useful.
In the amended version:
ssh remoteid#remoteserver "'mv /upload/hotfolders/in/"$FileName"
/upload/hotfolders/Archive/${FileName}_${ts}'"
there is a different problem. Here, the two sets of outermost " allow the local system to expand the variables (although it may not be obvious that the first $FileName is not actually inside "). However, as the command that is passed is now wrapped in ', the remote server will treat the entire string as a single word.
If we assume that FileName and ts will not contain shell-special characters (such as ') then the fix is to wrap the command sequence in " (so that it expands locally), and only wrap the variables in ' (so that the remote server treats the now-expanded strings as single words):
ssh remoteid#remoteserver "cd /upload/hotfolders/; mv '$FileName'
/upload/hotfolders/Archive/'${Filename}_${ts}'"
I have a bash script to retrieve files from ftp.
Now the files have one part a date string in the filename, but also undefined numbers that changes on every file. I want to download the files based on the date.
This is my code. I only need to do the wildcard trick, the ftp script is allready work.
filename=$(echo $TIMESTAMP'0***vel.radar.h5')
The stars are 3 digits with different numbers that i can't estimate, so i would use the wildcard for them.
Thank you
It sounds like you want to handle multiple files, but your script can only handle one file at a time. Furthermore, because you specified FTP, it sounds like the files are on the FTP server, in which case local filename expansion will not help.
You probably want to use the ftp client's mget command to download multiple files matching a pattern on the remote side. You also want to include $TIMESTAMP as part of the pattern. I'd suggest something like this:
ftp remote-hostname <<EOF
cd path/to/log/files
prompt
mget ${TIMESTAMP}0???vel.radar.h5
bye
EOF
This uses a here-document (<<EOF to EOF on a line by itself) to supply input text to the ftp commmand. It will expand the variable $TIMESTAMP so it becomes part of the mget command, e.g. if $TIMESTAMP was 12345, the ftp command will be told mget 123450???vel.radar.h5.
With the ? characters you include not only numbers, but any character. So if you want to include only numbers, you can replace the ??? characters with [0-9][0-9][0-9].
Example:
If you have the following files:
$ ls
0123vel.h5 0333vel.h5 033vel.h5 0pecvel.h5
with this ls you show the correct files:
$ ls 0[0-9][0-9][0-9]vel.radar.h5
0123vel.h5 0333vel.h5
I'm trying to create a bash script which will sync a directory specified as a command line parameter to a remote server (also specified by a parameter). At the moment, I'm using eval, which solves a parameter expansion problem, but for some reason causes rsync not to preserve ownership on the remote files (apart from being Evil, I know). Running the rsync command with all the same flags and parameters from the command prompt works fine.
I tried using $() as an alternative, but I got into a real mess with variable expansion and protecting the bits that need protecting for the remote rsync path (which needs both quotes and backslashes for paths with spaces).
So - I guess 2 questions - is there a reason that eval is preventing rsync from preserving ownership (the bash script is being run as root on the source machine, and sshing to the remote machine as root too - just for now)? And is there a way of getting $() to work in this scenario? The (trimmed) code is below:
#!/bin/bash
RSYNC_CMD="/usr/bin/rsync"
RSYNC_FLAGS="-az --rsh=\"/usr/bin/ssh -i \${DST_KEY}\"" # Protect ${DST_KEY} until it is assigned later
SRC=${1} # Normally this is sense checked and processed to be a canonical path
# Logic for setting DST based on command line parameter snipped for clarity - just directly assign for testing
DST='root#some.server.com:'
DST_KEY='/path/to/sshKey.rsa'
TARG=${DST}${SRC//' '/'\ '} # Escape whitespace for target system
eval ${RSYNC_CMD} ${RSYNC_FLAGS} \"${SRC}\" \"${TARG}\" # Put quotes round the paths - even though ${TARG} is already escaped
# All synced OK - but ownership not preserved despite -a flag
I've tried changing RSYNC_CMD to sudo /usr/bin/rsync, and also adding --rsync-path="sudo /usr/bin/rsync to RSYNC_FLAGS, but neither made any difference. I just can't see what I'm missing...
The correct way to do this is to use an array. -a should already imply -o.
RSYNC_CMD="/usr/bin/rsync"
DST='root#some.server.com:'
DST_KEY='/path/to/sshKey.rsa'
RSYNC_FLAGS=(-az --rsh="/usr/bin/ssh -i ${DST_KEY}")
SRC=${1}
TARG="${DST}$SRC"
${RSYNC_CMD} "${RSYNC_FLAGS[#]}" "${SRC}" "${TARG}"
Using RSYNC_RSH instead of --rsh, you can export the variable before you set its value. This at least lets you put the export in the same area where you set the rest of the flags. Then you can defer completing its value until after you have the correct identity file.
RSYNC_CMD="/usr/bin/rsync"
export RSYNC_RSH="/usr/bin/ssh -i %s" # Use a placeholder for now; set it later
RSYNC_FLAGS=( -a -z )
# Later...
DST='root#some.server.com:'
DST_KEY='/path/to/sshKey.rsa'
RSYNC_RSH=$( printf "$RSYNC_RSH" "$DST_KEY" )
SRC=${1}
TARG="${DST}$SRC"
${RSYNC_CMD} "${RSYNC_FLAGS[#]}" "${SRC}" "${TARG}"
I'm trying to scp a backup tgz file from one server to another every night. The backup script uses the following $date var just fine but when I modify it slightly for scp it breaks:
#!/bin/sh
date=`date +%Y-%m-%d`
rbfile=`/backups/$date_00h00.tgz`
scp $rbfile user#myserverip:
But the script dies with the error:
/backups/.tgz: No such file or directory
On a side note, I really should switch to rsync for better remote backups - the tgz files are at 3.5GB now. Any recommended tutorials?
when using $date_00h00 you tell bash to use the variable named date_00h00, because letters, numbers and _ characters are allowed as variables names.
Enclose the variable name in {} and it will correct the problem :
rbfile=`/backups/${date}_00h00.tgz`
Periodically, I like to put files in the /tmp directory of my webserver to share out. What is annoying is that I must set the permissions whenever I scp the files. Following the advice from another question I've written a script which copies the file over, sets the permissions and then prints the URL:
#!/bin/bash
scp "$1" SERVER:"/var/www/tmp/$1"
ssh SERVER chmod 644 "/var/www/tmp/$1"
echo "URL is: http://SERVER/tmp/$1"
When I replace SERVER with my actual host, everything works as expected...until I execute the script with an argument including spaces. Although I suspect the solution might be to use $# I've not yet figured out how to get a spaced filename to work.
It turns out that what is needed is to escape the path which will be sent to the remote server. Bash thinks the quotes in SERVER:"/var/www/tmp/$1" are related to the $1 and removes them from the final output. If I try to run:
tmp-scp.sh Screen\ shot\ 2010-02-18\ at\ 9.38.35\ AM.png
Echoing we see it is trying to execute:
scp SERVER:/var/www/tmp/Screen shot 2010-02-18 at 9.38.35 AM.png
If instead the quotes are escaped literals then the scp command looks more like you'd expect:
scp SERVER:"/var/www/tmp/Screen shot 2010-02-18 at 9.38.35 AM.png"
With the addition of some code to truncate the path the final script becomes:
#!/bin/bash
# strip path
filename=${1##*/}
fullpath="$1"
scp "$fullpath" SERVER:\"/var/www/tmp/"$filename"\"
echo SERVER:\"/var/www/tmp/"$filename"\"
ssh SERVER chmod 644 \"/var/www/tmp/"$filename"\"
echo "URL is: http://SERVER/tmp/$filename"
The script looks right. My guess is that you need to quote the filename when you pass it into your script:
scp-chmod.sh "filename with spaces"
Or escape the spaces:
scp-chmod.sh filename\ with\ spaces
the easier way without worrying about spaces in file names, (besides quoting) is to rename your files to get rid of spaces before transferring. Or when you create the files, don't use spaces. You can make this your "best practice" whenever you name your files.