Currently i have a shell script which will sftp all the .txt files in the remote path to local and delete all the .txt files from the remote path.
But i need to change the logic such that i need to delete the file which successfully fetched from the remote path not all the file.
Current code:
sftp user#server << END_SCRIPT
cd /test
mget *.txt
rm *.txt
quit
END_SCRIPT
You can do it in two steps:
# get the files
sftp user#server << END_SCRIPT
cd /test
mget *.txt
quit
END_SCRIPT
# remove the files
{
echo "cd /test"
# assuming that all files in the current dir are fetched through sftp
for file in *.txt; printf 'rm "%s"\n' "$file"; fi
quit
} | sftp user#server
Related
I want to create a shell script that will rename all .txt files from a specific directory in remote server by using SFTP (will download the files first then rename in remote server). Please check the attempt below:
sftp user#host <<EOF
cd $remoteDir
get *.txt
ls *.txt | awk '{printf "rename %s %s.done\n",$0,$0 ;}'
exit
EOF
From the statement ls *.txt | awk '{printf "rename %s %s.done\n",$0,$0 ;}' it will generate and print out a list of rename command, my question is, how to run these command generated from awk printf?
You are trying to rename files on the server but you only know what commands to run after you have downloaded the files.
The simple option would be to run two sftp sessions. The first downloads the files. Then you generate the rename commands. Then you run a second sftp session.
However it is possible to do both in one session:
#!/bin/bash
(
# clean up from any previous run
rmdir -f syncpoint
# echo commands are fed into the sftp session
# be careful with quoting to avoid local shell expansion
echo 'cd remoteDir'
echo 'get *.txt'
echo '!mkdir syncpoint'
# wait for sftp to create the syncpoint folder
while [ ! -d syncpoint ]; do sleep 5; done
# the files have been downloaded
# now we can generate the rename commands
for f in *.txt; do
# #Q is a bash (v4.4+) way to quote special characters
echo "rename ${f#Q} ${f#Q}.done"
# if not available, single-quoting may be enough
#echo "rename '$f' '$f'.done"
done
# clean up
rmdir syncpoint
) | sftp user#host
Hello Newbie please use this
sftp user#host <<EOF
cd $remoteDir
ls *.txt | awk '{printf "mv %s %s.done\n",$0,$0 ;}' | sh
exit
EOF
I am trying to ftp a set of files to a client server. I tried automating it using ftp since we dont have SSH key setup yet. I have stored the destination host (Name, UN, PW) in .netrc file. When the script is executed, it opens up the ftp terminal and stops. How is it possible to continue the script post opening the ftp terminal ? Can you guys help ?
#!/bin/ksh
cd /scripts
. ./getYesterdayDate.sh
today=${YESTERDAY_DT}
today=`echo $today | sed 's/-//g'``
echo "Today is $today"
#Copying file from Source server
/usr/local/bin/scp UN#HOST:/backup/abc*.$today /scripts
#Renaming the files to custom names in the intermidatory server
cd /scripts
rm abc.data
touch abc.data
cat abc.data.$today >> abc.data
chmod a+r *
mv abc.START.$today abc.START
mv abc.END.$today abc.END
#FTPing to the DESTINATIONSERVER
/usr/bin/ftp DESTINATIONSERVERenter code here
cd /test
put abc.START
put abc.END
put abc.data
#In the above statements, when ftp is successful, instead of taking "cd /test" as input, terminal just stays at "ftp>"
echo "Upload for $today" | mailx -s "Successfull" name#domain.com
Thanks
If the .netrc is correctly setup and the ftp command isn't prompting for credentials, then you can upload the files with:
echo put abc.START /test/abc.START | ftp DESTINATIONSERVER
echo put abc.END /test/abc.START | ftp DESTINATIONSERVER
echo put abc.data /test/abc.START | ftp DESTINATIONSERVER
I want to put in for-loop in the ftp script. My code is as follows.
ftp -n ftp.server.com <<EOF
user name passwd
bin
prompt
for DATA in d f g l m n o p q; do
mkdir /directory/$DATA
cd /directory/$DATA
mput *.$DATA
done
hash
bye
EOF
It's not working and error is
Interactive mode off.
We only support non-print format, sorry.
?Invalid command
?Invalid command
?Invalid command
Create directory operation failed.
local: mput: No such file or directory
(local-files) ?Invalid command
Hash mark printing on (1024 bytes/hash mark).
Of course I can put this ftp script in for loop, inversely. Is there anything good advice?
You cannot use bash loop inside ftp prompt. But you can generate the stdin for ftp via a bash loop.
{
echo username passwd
echo bin
echo prompt
for DATA in d f g l m n o p q; do
echo mkdir /directory/$DATA
echo cd /directory/$DATA
echo mput *.$DATA
done
echo hash
echo bye
} | ftp -n ftp.server.com
My solution to upload all files to remote server which are created/modified today.
#!/bin/bash
HOST='hostname'
USER='username'
PASSWD='password'
# Local directory where the files are stored.
cd "/local/directory/from where to upload files/"
# To get all the files added today only.
TODAYSFILES=`find -maxdepth 1 -type f -mtime -1`
# remote server directory to upload backup
REMOTEDIR="/directory on remote ftp computer/"
for FILENAME in ${TODAYSFILES[#]}; do
ftp -n -v $HOST << EOT
ascii
user $USER $PASSWD
prompt
cd $REMOTEDIR
put $FILENAME
bye
EOT
done
I want to create a directory if it doesn't exists after login to sftp server.
test.sh
sftp name#example.com << EOF
mkdir test
put test.xml
bye
EOF
Now i call test.sh and upload different files each time to test folder. When running this
mkdir test
First time it works and second time it throws Couldn't create directory: Failure error?
How to create a directory if doesn't exists and if exists don't create directory in sftp.
man 1 sftp (from openssh-client package):
-b batchfile
Batch mode reads a series of commands from an input
batchfile instead of stdin. Since it lacks user
interaction it should be used in conjunction with
non-interactive authentication. A batchfile of ‘-’
may be used to indicate standard input. sftp will
abort if any of the following commands fail: get,
put, reget, reput, rename, ln, rm, mkdir, chdir, ls,
lchdir, chmod, chown, chgrp, lpwd, df, symlink, and
lmkdir. Termination on error can be suppressed on a
command by command basis by prefixing the command
with a ‘-’ character (for example, -rm /tmp/blah*).
So:
{
echo -mkdir dir1
echo -mkdir dir1/dir2
echo -mkdir dir1/dir2/dir3
} | sftp -b - $user#$host
I understand this thread is old and has been marked as answered but the answer did not work in my case. The second page on google for a search regarding "sftp checking for directory" so here is an update that would have saved me a few hours.
Using an EOT you cannot capture the error code resulting from the directory not being found. The work around I found was to create a file containing instructions for the call and then capture the result of that automated call.
The example below using sshpass but my script also uses this same method authenticating with sshkeys.
Create the file containing the instructions:
echo "cd $RemoteDir" > check4directory
cat check4directory; echo "bye" >> check4directory
Set permissions:
chmod +x check4directory
Then make the connection using the batch feature:
export SSHPAA=$remote_pass
sshpass -e sftp -v -oBatchMode=no -b check4directory $remote_user#$remote_addy
Lastly check for the error code:
if [ $? -ge "1" ] ; then
echo -e "The remote directory was not found or the connection failed."
fi
At this point you can exit 1 or initiate some other action. Note that if the SFTP connection fails for another reason like password or the address is incorrect the error will trip the action.
Another variant is to split the SFTP session into two.
First SFTP session simply issues the MKDIR command.
Second SFTP session can then assume existence of the directory and put the files.
You can use the SSH access of your account to first verify if the directory exists at all (using the "test" command). If it returns exit code 0, the dir exists, otherwise it doesn't. You can act on that accordingly.
# Both the command and the name of your directory are "test"
# To avoid confusion, I just put the directory in a separate variable
YOURDIR="test"
# Check if the folder exists remotely
ssh name#example.com "test -d $YOURDIR"
if [ $? -ne 0 ]; then
# Directory does not exist
sftp name#example.com << EOF
mkdir test
put test.xml
bye
EOF
else
# Directory already exists
sftp name#example.com << EOF
put test.xml
bye
EOF
fi
Try this to ignore errors if directory already exists.
# Turn OFF error
set +e
# Create remote dirs
sftp -P 22 -o StrictHostKeyChecking=no -oIdentityFile=key.pem -v $user#$host <<EOF
mkdir <remote_path> # create remote directory
bye
EOF
# Turn ON error
set -e
# Do upload to SFTP
sftp -P 22 -o StrictHostKeyChecking=no -oIdentityFile=key.pem -v $user#$host <<EOF
cd <remote_path> # remote_path
put <local_file_path> # local_path
quit
EOF
I'm writing a bash script to send files from a linux server to a remote Windows FTP server.
I would like to check using FTP if the folder where the file will be stored exists before attempting to create it.
Please note that I cannot use SSH nor SCP and I cannot install new scripts on the linux server. Also, for performance issues, I would prefer if checking and creating the folders is done using only one FTP connection.
Here's the function to send the file:
sendFile() {
ftp -n $FTP_HOST <<! >> ${LOCAL_LOG}
quote USER ${FTP_USER}
quote PASS ${FTP_PASS}
binary
$(ftp_mkdir_loop "$FTP_PATH")
put ${FILE_PATH} ${FTP_PATH}/${FILENAME}
bye
!
}
And here's what ftp_mkdir_loop looks like:
ftp_mkdir_loop() {
local r
local a
r="$#"
while [[ "$r" != "$a" ]]; do
a=${r%%/*}
echo "mkdir $a"
echo "cd $a"
r=${r#*/}
done
}
The ftp_mkdir_loop function helps in creating all the folders in $FTP_PATH (Since I cannot do mkdir -p $FTP_PATH through FTP).
Overall my script works but is not "clean"; this is what I'm getting in my log file after the execution of the script (yes, $FTP_PATH is composed of 5 existing directories):
(directory-name) Cannot create a file when that file already exists.
Cannot create a file when that file already exists.
Cannot create a file when that file already exists.
Cannot create a file when that file already exists.
Cannot create a file when that file already exists.
To solve this, do as follows:
To ensure that you only use one FTP connection, you create the input (FTP commands) as an output of a shell script
E.g.
$ cat a.sh
cd /home/test1
mkdir /home/test1/test2
$ ./a.sh | ftp $Your_login_and_server > /your/log 2>&1
To allow the FTP to test if a directory exists, you use the fact that "DIR" command has an option to write to file
# ...continuing a.sh
# In a loop, $CURRENT_DIR is the next subdirectory to check-or-create
echo "DIR $CURRENT_DIR $local_output_file"
sleep 5 # to leave time for the file to be created
if (! -s $local_output_file)
then
echo "mkdir $CURRENT_DIR"
endif
Please note that "-s" test is not necessarily correct - I don't have acccess to ftp now and don't know what the exact output of running DIR on non-existing directory will be - cold be empty file, could be a specific error. If error, you can grep the error text in $local_output_file
Now, wrap the step #2 into a loop over your individual subdirectories in a.sh
#!/bin/bash
FTP_HOST=prep.ai.mit.edu
FTP_USER=anonymous
FTP_PASS=foobar#example.com
DIRECTORY=/foo # /foo does not exist, /pub exists
LOCAL_LOG=/tmp/foo.log
ERROR="Failed to change directory"
ftp -n $FTP_HOST << EOF | tee -a ${LOCAL_LOG} | grep -q "${ERROR}"
quote USER ${FTP_USER}
quote pass ${FTP_PASS}
cd ${DIRECTORY}
EOF
if [[ "${PIPESTATUS[2]}" -eq 1 ]]; then
echo ${DIRECTORY} exists
else
echo ${DIRECTORY} does not exist
fi
Output:
/foo does not exist
If you want to suppress only the messages in ${LOCAL_LOG}:
ftp -n $FTP_HOST <<! | grep -v "Cannot create a file" >> ${LOCAL_LOG}