I am writing a bash script to copy a file from a remote server, to my local machine. I need to check to see if the file is available, so I can take an alternative action if it is not there.
I know how to test if a local file exists, however, using scp complicates things a bit. Common sense tells me that one way would be to try to scp the file anyhow, and check the return code from the scp command. Is this the correct way to go about it?
If yes, how do I test the return code from the scp invocation?
using ssh + some shell code embedded in the cmd line; use this method when you need to take a decision before the file transfer will fail;
ssh remote-host 'sh -c "if [ -f ~/myfile ] ; then gzip -c ~/myfile ; fi" ' | gzip -dc > /tmp/pkparse.py
if you want to transfer directories you may want to "tar"-it first
if you want to use scp you can check the return code like this:
if scp remote-host:~/myfile ./ >&/dev/null ; then echo "transfer OK" ; else echo "transfer failed" ; fi
it really depends on when its important for you to know if the file is there or not; before the transfer starts (use ssh+sh) or after its finished.
well, since you can use scp you can try using ssh to list and see if the file is their or not before proceeding.
Related
I have created one script to copy the local files to the remote folder. The script is working fine outside of if condition. But when I enclosed inside the if condition the put command is not working. It logged into the remote server using SFTP protocol and when exist it's showing the error:
put command not found
See what is happening after executing the script:
Connected to 10.42.255.209.
sftp> bye
sftp.sh: line 23: put: command not found
Please find the below script.
echo -e;
echo -e "This script is used to copy the files";
sleep 2;
localpath=/home/localpath/sftp
remotepath=/home/destination/sftp/
if [ -d $localpath ]
then
echo -e "Source Path found"
echo -e "Reading source path"
echo -e "Uploading the files"
sleep 2;
sftp username#10.42.255.209
put $localpath/* $remotepath
else
In a simple case such as this, you could use scp instad of sftp and specify the files to copy on the command line:
scp $localpath/* username#10.42.255.209:/$remotepath/
But if you would rather want to issue sftp commands, then sftp can read commands from its stdin, so you can do:
echo "put $localpath/* $remotepath" | sftp username#10.42.255.209
Or you can use a here document to pass data as stdin to sftp, which might be easier if you want to run several sftp commands:
sftp username#10.42.255.209 << EOF
put $localpath/fileA $remotepath/
put $localpath/fileB $remotepath/
EOF
Finally, you could place the sftp commands in a separate file, say sftp_commands.txt , and have sftp execute those commands using its -b flag:
sftp -b ./sftp_commands.txt username#10.42.255.209
I got the result using this format
HOST='xyz.abc.com'
USER='xyzasd'
REMOTEPATH='/var/www/data-csv/'
file_name='/tmp/sample.csv'
sftp $USER#$HOST <<EOF
cd /var/www/data-csv/
put $file_name
EOF
It will ask for password if the user have a password. Otherwise this code works fine.
This code worked for me
for reference read https://help.oclc.org/Librarian_Toolbox/Exchange_files_with_OCLC/Upload_files_with_SFTP/40SFTP_commands?sl=en
uploadFileToMFT(){
sftp -P ${PORT_NO} ${HOST_NAME}#${HOST_ID} <<EOF
cd /mdm_dev05
put ${EXPORT_OUTPUT}'/'${ID}'/'${F_NAME}
quit
EOF
}
I'm implementing agent script bash to pull files from the remote server with SFTP service.
The script must:
connect SFTP
file listing
cycling on files found
get every file and copy agent side
after that files copied must be deleted
The script is followed:
#!/bin/bash
SFTP_CONNECTION="sftp -oIdentityFile=/home/account_xxx/.ssh/service_ssh user#host"
DEST_DATA=/tmp/test/data/
# GET list file by ls command ###############
$SFTP_CONNECTION
$LIST_FILES_DATA_OSM1 = $("ls fromvan/test/data/test_1")
echo $LIST_FILES_DATA_OSM1
for file in "${LIST_FILES_DATA_OSM1[#]}"
do
$SFTP_CONNECTION get $file $DEST_DATA
$SFTP_CONNECTION rm $file
done
I tried the script but it seems that the connection and command execution (ls) are distinct on thread separated.
How can I provide command sequential as described above ?
Screenshoot:
Invalid find command
SSH it seem not available
RSYNC result to take the files is the followed:
Thanks
First of all, I would recommend the following syntax changes:
#!/bin/bash
sftp_connection() {
sftp -oIdentityFile=/home/account_xxx/.ssh/service_ssh user#host "$#";
}
Dest_Data=/tmp/test/data/
# GET list file by ls command ###############
sftp_connection
List_Files_D_OSM1=$("ls fromvan/test/data/test_1")
echo "$LIST_FILES_DATA_OSM1"
for file in "${LIST_FILES_DATA_OSM1[#]}"
do
sftp_connection get "$file" $Dest_Data
sftp_connection rm "$file"
done
Quoting $file and $List_Files_D_OSM1 to prevent globbing and word splitting.
Assignments can't start with a $, otherwise bash will try to execute List_Files_D_OSM1 and will complain with a command not found
No white spaces in assignments like List_Files_D_OSM1 = $("ls fromvan/test/data/test_1")
You can use ShellCheck to catch this kind of errors.
Having said that, it is in general not a good idea to use ls in such way.
What you can use instead is something like find. For example:
find . -type d -exec echo '{}' \;
Use a different client. lftp supports sftp as a transport, and has a subcommand for mirroring which will do the work of listing the remote directory and iterating over files for you.
Assuming your ~/.ssh/config contains an entry like:
Host myhost
IdentityFile /home/account_xxx/.ssh/service_ssh
...you can run:
lftp -e 'mirror -R fromvan/test/data/test_1 /tmp/test/data' sftp://user#myhost
I've been trying to zip a file and then upload it by FTP using a bash script, but it's uploading a corrupt zip file. I've had a look around and I'm trying to use lsof | grep to confirm the file is complete but I'm not really sure what I'm doing.
So I've got
cd /var/test/tobezipped
zip -r test.zip *
FOLDER="/var/test/tobezipped"
ZIPS=$(ls $FOLDER)
for F in $zips ; do
while [ -n "$(lsof | grep $F" ] ; do
sleep 1
done
ftp -n <<EOF
open myserver
user user pass
put test.zip
EOF
done
and test.zip is corrupt at the time it's being uploaded, so on the other server it's not readable but on the server it's zipped on it's all good by the time I check it.
Any kind of advice is appreciated, I'm pretty new to this sort of thing and tried to search around heaps to find a solution, not too sure I'm going in the right direction. Thanks in advance.
From the man page:
put local-file [remote-file]
Store a local file on the remote machine. If remote-file is left unspecified, the local file name is used after
processing according to
any ntrans or nmap settings in naming the remote file. File transfer uses the current settings for type, format, mode,
and structure.
I guess the problem is that you are not transfering files in BINARY mode.
Try this:
ftp -n <<EOF
open myserver
user user pass
binary
put test.zip
EOF
Try this.
PID=$(pgrep zip)
while [[ ( -d /proc/$PID) ) && ( -z `grep zombie /proc/$PID/status` ) ]]; do
sleep 1
done
hi i have some file path like
/ifshk5/BC_IP/PROJECT/T1
1073/T11073_RICljiR/split/AG19_235/120225_I872_FCC0HN2ACXX_L8_RICljiRSYHSD2-1-IP
AAPEK-17_1.fq.gz
i need copy files from one ftp server to other. and also need to create directory if it not exist in server.
i login the sever which contains those file then run this code
#! /bin/bash
while read myline
do
for i in $myline
do
if [ -f $i ]
then
location=$(echo "$i" | awk -F "/" '{ print "", $6, $7, $8 }' OFS="/")
#location shows /T11073_RICekkR/Fq/AS59_59304
location="/opt/CLiMB/Storage3/ftp/ftp_climb/100033"$location
echo $location
ssh tam#192.168.174.43 mkdir -p $location
scp -r $i tam#192.168.174.43:$location
fi
done
done < /ifshk5/BC_IP/PROJECT/T11073/T11073_all_3254.fq.list
it has some problem, 1. it can't work always shows permission denied, please try again.
but when i direct type
ssh tam#192.168.174.43 mkdir -p /sample/xxxx
it can work, and the new dir location is right it shows like
/opt/CLiMB/Storage3/ftp/ftp_climb/100033/T11073_RICekkR/Fq/AS59_59304
I don't see where the "permission denied" error might come from; run the script with bash -x to see the command which causes the error. Maybe it's not what you expect.
Also try rsync instead of inventing the wheel again:
rsync --dirs $i tam#192.168.171.34:$b
--dirs will create the necessary folders on the remote side (and it will give you good error messages when something fails).
It might even be possible to do everything with a single call to rsync if you have the same folder structure on both sides:
rsync -avP /ifshk5/BC_IP/PROJECT/T11073/ tam#192.168.171.34:/opt/CLiMB/Storage3/ftp/ftp_climb/100033/
Note the / after the paths! Don't omit them.
rsync will figure out which files need to be transferred and copy only those. If you want to transfer only a subset, use --include-from
I'm writing a cron to backup some stuffs on a server.
Basically I'm sending specific files form a local directory using scp.
I'm using a public key to avoid authentication.
For reusability I'm passing the local directory and the server url by arguments to my bash script.
How I set my parameters:
#!/bin/bash
DIR="$1"
URL="$2"
FILES="$DIR*.ext"
My problem is about formatting the url.
Without formatting
How I send files to the server:
#!/bin/bash
for F in $FILEs
do
scp $F $URL;
if ssh $URL stat $(basename "$F")
then
rm $F
else
echo "Fails to copy $F to $URL"
fi
done
If I try to copy at user's home on the server I do:
$ ~/backup /path/to/local/folder/ user#server.com:
If I try to copy at a specific directory on the server I do:
$ ~/backup /path/to/local/folder/ user#server.com:/path/to/remote/folder/
In all cases it gives me the well known error (and my custom echo):
ssh: Could not resolve hostname user#server.com: nodename nor [...]
Can't upload /path/to/local/folder/file.ext to user#server.com
And it works anyway (the file is copied). But that's not a solution, cause as scp fails (seems to), the file is never deleted.
With formatting
I tried sending files using this method:
#!/bin/bash
for F in $FILES
do
scp $F "$URL:"
done
I no longer get an error, and it works for copying at user's home directory then deleting the local file:
$ ~/backup /path/to/local/folder/ user#server.com
But, of course, sending to a specific directory don't work at all.
Finally
So I think that my first method is more appropriate, but how can I get rid of that error?
Your mistake is that you can scp to user#server.com: but not ssh to it : you need to remove the trailing : character (and possible path after it). You can do it easily like this with bash parameter expansion :
ssh "${URL%:*}" stat "$(basename "$F")"
RECOMMENDATIONS
"USE MORE QUOTES!" They are vital. Also, learn the difference between ' and " and `. See http://mywiki.wooledge.org/Quotes and http://wiki.bash-hackers.org/syntax/words
if you have spaces in filenames, your code will breaks things up. Better use while IFS= read -r line; do #stuff with $line; done < file.txt
See bash parameter expansion