I am trying to bash script a daily file transfer between 2 machines. The script runs on the destination machine and pulls a file from the source machine. Occasionally the source machine will not have a file ready, this is acceptable.
I would like the script to exit 0 on successful transfer, and when there is no file available to be transferred. I would like the script to exit non 0 on any other failure condition (connectivity, etc).
I tried the following 2 approaches, I found with SCP the return code is always 1 no matter what the actual error, so its hard for the script to differentiate between my acceptable error condition, and others.
The sftp method seems to always return 0 no matter what takes place during the command. Any suggestions?
scpGet(){
echo "Attempting File Transfer"
scp -P $REMOTEPORT $REMOTEHOST:$REMOTEPATH $LOCALPATH
echo $?
}
sftpGet(){
cd $LOCALPATH
sftp -P $REMOTEPORT $REMOTEHOST << EOF
get $REMOTEPATH
quit
EOF
echo $?
}
I haven't validated this, so please check that it actually does what you want -
but you are apparently running scp with no password, so you can probably execute arbitrary code remotely to test for the existence of the file. Just be careful.
scpGet() {
echo "Attempting File Transfer"
if scp -P $REMOTEPORT $REMOTEHOST:$REMOTEPATH $LOCALPATH
then echo "$( ls -l $LOCALPATH) - successfully retrieved"
elif ssh -P $REMOTEPORT ls -l $REMOTEHOST:$REMOTEPATH
then echo "$REMOTEHOST:$REMOTEPATH exists, but I can't retrieve it!" >&2
exit $oopsieCode
elif (( 2 == $rc )) # ls failed to find the file - verify this code
then echo "File not ready. Ignoring."
else : handle errors other than "not found"
fi
}
Related
I'm calling curl on bash to copy a file from a mounted SD card with the option to resume the copy later if the device gets unmounted. I receive the same status exit code 0 when I interrupt the copy by unmounting the volume and when the file gets actually copied. Any suggestions how to catch the case where the file has not been copied?
I'm copying only one file at a time.
This is the command:
curl -C - -O file:///mnt/sdcard/DCIM/100/0044.MP4
I came to a solution which is not as clear as I want, but still working. I'm executing the command 2 times one after another, so when the first command returns 0 upon unmount, the second now tries to copy the file and return error code 37 because of the unreachable source. If the second command returns 0 I consider the file copied.
Following your concept you could have a script like this:
#!/bin/bash
# Copies files persistently.
#
# Usage: pc <filepath> [<filepath2>] ...
#
function pc {
local FILE
for FILE; do
echo "Copying $FILE."
until curl -C - -O "file://${FILE}" && curl -C - -O "file://${FILE}"; do
if [[ -e $FILE ]]; then
echo "File $FILE can't be copied."
break
else
echo "Waiting for $FILE."
until
sleep 5
[[ -e $FILE ]]
do
continue
done
fi
done
done
}
pc "$#"
You could also just embed the function to a bash startup script if you like.
Let's say I have a very simple script which creates a link in a certain directory, and kills the script if it fails.
ln -s "/opt/myapp" "${1}/link" || exit 1;
Right now it just quits if it runs into errors. I want to change it so only if it runs into permission errors when creating the link, it will execute the following lines instead of exiting:
echo "The target directory requires root privileges to access."
sudo ln -s "/opt/myapp" "${1}/myapp" || exit 1;
I don't want to prompt the users to run as root unless they absolutely have to.
ln seems to retun exit code 1 on failure regardless of whether it was a problem with permissions or any other errors such as a directory not existing, so I can't use that to detect which problem it ran into.
And if I instead store search through the output of ln for the string "Permission denied", I'm assuming it will fail on non-english operating systems.
I don't know of any ways to categorize ln exit reasons, or at least any documentation about specific exit codes you could test with $?, but you can test for relevant permissions with the standard test or [ command:
SOURCEFILE="/opt/myapp"
DESTDIR="${1}"
DESTTARGET="${DESTDIR}/myapp"
if [ ! -d "$DESTDIR" -o ! -e "$SOURCEFILE" ]; then
echo "Source file does not exist or destination directory does not exist." >&2
elif [ ! -r "$SOURCEFILE" -o ! -w "$DESTDIR" ]; then
echo "Source file is not readable or destination directory is not writable." >&2
# Run sudo command here
else
# Should work, run command here
fi
I'm trying to write a shell script that grabs a set of parameters from a text file and then performs SFTP based on those parameters. Basically, I'm taking a daily webstats log and moving it to a central location.
The issue I'm having is that the SFTP fails based on the way I am assigning variables. I have debugged and found that the while loop works correctly by echoing out the loop of variables. The error I get is that the connection is closed.
#!/bin/sh
source /home/ntadmin/webstats/bin/webstats.profile
source /home/ntadmin/webstats/bin/webstats.blogs.profile
DATE=`date +%m%d%Y`
SOURCE_FILE="`echo $WS_BC_SOURCE_FILE | sed -e 's/mmddyyyy/'$DATE'/'`"
IFS=","
while read WS_BLOG_NAME WS_BLOG_SOURCE_VAR WS_BLOG_DEST_VAR WS_BC_SERVER1;
do
#Step 1 SFTP
cd $PERL_DIR
if $PERL_DIR/sftp.pl $WS_BC_SERVER1 $WS_BC_ID $WS_BC_PW $WS_BLOG_SOURCE_VAR/$SOURCE_FILE $WS_BLOG_DEST_VAR/$SOURCE_FILE
then
echo 'SFTP complete'
else
echo 'SFTP failed!'
exit 1
fi
#Step 2 - Check that ftp was successful (that the files exist)
if [ -e $WS_BLOG_DEST_VAR/$SOURCE_FILE ]
then
echo "FTP of $WS_BLOG_SOURCE_VAR/$SOURCE_FILE from $WS_BC_SERVER1 was successful"
else
echo "FTP of $WS_BLOG_SOURCE_VAR/$SOURCE_FILE from $WS_BC_SERVER1 was not successful!"
exit 1
fi
done < blogs_array.txt
exit 0
There is not enough information to determine what was wrong, but here is a debugging method.
Try replace the actual sftp command in perl script with a debug script like this, you should be able to locate the problem quickly.
#!/usr/bin/perl
print "arguments passed to $0\n";
$i=0;
while (defined $ARGV[$i]) {
print "arg ".($i+1)." is <$ARGV[$i++]>\n"
}
how to find a file exists in particular dir through SSH
for example :
host1 and dir /home/tree/TEST
Host2:- ssh host1 - find the TEST file exists or not using bash
ssh will return the exit code of the command you ask it to execute:
if ssh host1 stat /home/tree/TEST \> /dev/null 2\>\&1
then
echo File exists
else
echo Not found
fi
You'll need to have key authentication setup of course, so you avoid the password prompt.
This is what I ended up doing after reading and trying out the stuff here:
FileExists=`ssh host "test -e /home/tree/TEST && echo 1 || echo 0"`
if [ ${FileExists} = 0 ]
#do something because the file doesn't exist
fi
More info about test: http://linux.die.net/man/1/test
An extension to Erik's accepted answer.
Here is my bash script for waiting on an external process to upload a file. This will block current script execution indefinitely until the file exists.
Requires key-based SSH access although this could be easily modified to a curl version for checks over HTTP.
This is useful for uploads via external systems that use temporary file names:
rsync
transmission (torrent)
Script below:
#!/bin/bash
set -vx
#AUTH="user#server"
AUTH="${1}"
#FILE="/tmp/test.txt"
FILE="${2}"
while (sleep 60); do
if ssh ${AUTH} stat "${FILE}" > /dev/null 2>&1; then
echo "File found";
exit 0;
fi;
done;
No need for echo. Can't get much simpler than this :)
ssh host "test -e /path/to/file"
if [ $? -eq 0 ]; then
# your file exists
fi
I have a shell script that I'm working on with this line of code that does a loop through local files (.gz) and does an scp. I want to test for a failed scp if possible. I am doing a loop so I can echo each file name to a log so I can keep track of it.
Can someone show me how to check for failed scp? or better yet, a good code example to do this? Thanks for your help.
for gzfile in $LOCALDMPDIR/*.gz
do
/usr/bin/scp -P 2222 -i $KEYFILE $gzfile foobar#$1:$TGTDIR
echo "$gzfile is done. " 2>&1
done
Use $? to access the return value of the last command. Check the man page for scp to verify, but I think a return value of zero means success. A non-zero value means some kind of failure.
use :
if [ $? -eq 0 ];
then
echo "OK"</br>
else
echo "NOK"</br>
fi
there're blank after "[" and before "]".
don't surround $? and 0 with quotes
You can check the varaible $? to see the return code of scp. If it returns non-zero then an error occurred.
You could also try to capture the error to a log:
for gzfile in $LOCALDMPDIR/*.gz
do
/usr/bin/scp -P 2222 -i $KEYFILE $gzfile foobar#$1:$TGTDIR 2>>/var/log/scperror.log \
&& echo "$gzfile is done." \
|| echo "scp error: $gzfile"
done
For the simpleminded like me out there who spent longer than normal messing with formatting errors:
scp "fromHere" hostname:"toThere"
if [ "$?" -eq "0" ];
then
echo "SUCCESS"
else
echo "FAIL"
fi