I want to create a shell script that will rename all .txt files from a specific directory in remote server by using SFTP (will download the files first then rename in remote server). Please check the attempt below:
sftp user#host <<EOF
cd $remoteDir
get *.txt
ls *.txt | awk '{printf "rename %s %s.done\n",$0,$0 ;}'
exit
EOF
From the statement ls *.txt | awk '{printf "rename %s %s.done\n",$0,$0 ;}' it will generate and print out a list of rename command, my question is, how to run these command generated from awk printf?
You are trying to rename files on the server but you only know what commands to run after you have downloaded the files.
The simple option would be to run two sftp sessions. The first downloads the files. Then you generate the rename commands. Then you run a second sftp session.
However it is possible to do both in one session:
#!/bin/bash
(
# clean up from any previous run
rmdir -f syncpoint
# echo commands are fed into the sftp session
# be careful with quoting to avoid local shell expansion
echo 'cd remoteDir'
echo 'get *.txt'
echo '!mkdir syncpoint'
# wait for sftp to create the syncpoint folder
while [ ! -d syncpoint ]; do sleep 5; done
# the files have been downloaded
# now we can generate the rename commands
for f in *.txt; do
# #Q is a bash (v4.4+) way to quote special characters
echo "rename ${f#Q} ${f#Q}.done"
# if not available, single-quoting may be enough
#echo "rename '$f' '$f'.done"
done
# clean up
rmdir syncpoint
) | sftp user#host
Hello Newbie please use this
sftp user#host <<EOF
cd $remoteDir
ls *.txt | awk '{printf "mv %s %s.done\n",$0,$0 ;}' | sh
exit
EOF
Related
My aim is to create a shell script such that it logins and filter the list of files available and select a file to get. Here I need to run commands like in bash.
My sample code is:
sshpass -p password sftp user#10.10.10.10 <<EOF
cd /home/
var=$(ls -rt)
echo $var
echo "select a folder"
read folder
cd $folder
filen=&(ls -rt)
echo $filen
echo "select a file"
read name
get $name
bye
EOF
The above approach will not work. Remember that the 'here document' (<<EOF ... EOF) is evaluate as input to the sftp session. Prompts will be displayed, and user input will be requested BEFORE any output (ls in this case) will be available from sftp.
Consider using lftp, which has more flexible construct. In particular, it will let you use variables, create command dynamically, etc.
lftp sftp://user#host <<EOF
cd /home
ls
echo "Select Folder"
shell 'read folder ; echo "cd $folder" >> temp-cmd'
source temp-cmd
ls
echo "Select Folder"
shell 'read file ; echo "get $file" >> temp-cmd'
source temp-cmd
EOF
In theory, you can create similar constructs with pipes and sftp (may be a co-process ?), but this is much harder.
Of course, the other alternative is to create different sftp sessions for listing, but this will be expensive/inefficient.
After some research and experimentation, found a way to create batch/interactive sessions with sftp. Posting as separate answer, as I still believe the easier way to go is with lftp (see other answer). Might be used on system without lftp
The initial exec create FD#3 - pointing to the original stdout - probably user terminal. Anything send to stdout will be executed by the sftp in the pipeline.
The pipe is required to allow both process to run concurrently. Using here doc will result in sequential execution. The sleep statement are required to allow SFTP to complete data retrieval from remote host.
exec 3>&1
(
echo "cd /home/"
echo "ls"
sleep 3 # Allow time for sftp
echo "select a folder" >&3
read folder
echo "cd $folder"
echo "ls"
sleep 3 # Allow time for sftp
echo "select a file" >&3
read name
echo "get $name"
echo "bye"
) | sshpass -p password sftp user#10.10.10.10
I would suggest you to create a file with pattern of the files you want downloaded and then you can get files downloaded in one single line:
sftp_connection_string <<< $"ls -lrt"|grep -v '^sftp'|grep -f pattern_file|awk '{print $9}'|sed -e 's/^/get -P /g'|sftp_connection_string
if there are multiple definite folders to be looked into, then:
**Script version**
for fldr in folder1 folder2 folder3;do
sftp_connection_string <<< $"ls -lrt ${fldr}/"|grep -v '^sftp'|grep -f pattern_file|awk '{print $9}'|sed -e "s/^/get -P ${fldr}/g"|sftp_connection_string
done
One-liner
for fldr in folder1 folder2 folder3;do sftp_connection_string <<< $"ls -lrt ${fldr}/"|grep -v '^sftp'|grep -f pattern_file|awk '{print $9}'|sed -e "s/^/get -P ${fldr}\//g"|sftp_connection_string;done
let me know if it works.
I'm connected to a remote machine via SSH as part of a bash script. After navigating to the directory, I run ls which confirms matching files are found. However, I then try to loop through the files and run other commands on them, and the variable is now empty.
Code:
echo "DOING STUFF!"
cd /mnt/slowdata/ls8_processing
ls
for f in *.tar.gz
do
echo $f
done
Output:
DOING STUFF!
LC080330242019031901T1-SC20190606111327.tar.gz
LC080330242019042001T1-SC20190606111203.tar.gz
LC080330242019052201T1-SC20190606111130.tar.gz
LC080330252019030301T2-SC20190606111021.tar.gz
LC080330252019031901T1-SC20190606120750.tar.gz
LC080340232019031001T1-SC20190606111056.tar.gz
LC080340232019041101T1-SC20190606111215.tar.gz
LC080340242019031001T1-SC20190606111201.tar.gz
LC080340242019041101T1-SC20190606111250.tar.gz
LC080340242019052901T1-SC20190606111331.tar.gz
As can be seen via the output, the $f is picking something up, as there are the correct number of blank lines. However I wish to untar each file which I cannot do.
TIA.
You have to remove special meaning of $ to pass it to the remote host as '$' else the variable will be expanded before you send the command to the remote host.
Keep in mind the for cycle will run regardless of whether the cd was successful.
ssh server1 << EOF
cd /mnt/slowdata/ls8_processing
ls
for f in *.tar.gz
do
echo \$f
done
EOF
My example show the difference:
script.sh
#!/bin/bash
f=123
ssh -i .ssh/keyauth.pem root#server1 << EOF
for f in ./*.log
do
echo "\$f"
echo "$f"
done
EOF
Output
[edvin#server2 ~]$ ./script.sh
./sepap-install.log
123
./sepfl-upgrade.log
123
./sep-install.log
123
./sepjlu-install.log
123
./sepui-install.log
123
I want to automate a SFTP process to transfer the last file created in local server and send it to remote server.
In local server I have "/Source/Path/" I have files named like below:
Logfile_2019-04-24
Logfile_2019-04-24_old.txt
This is my current script:
dyear=`date +'%Y' -d "1 day ago"`
dmonth=`date +'%b' -d "1 day ago"`
ddate=`date +%Y-%m-%d -d "1 day ago"`
HOST='192.168.X.X'
USER='user'
PASSWD='password'
localpath='/Source/Path/'$dyear'/'$dmonth'/'*$ddate*'.txt'
remotepath='/Destination/Path/'$dyear'/'$dmonth'/'
echo $localpath
echo $remotepath
export SSHPASS=$PASSWD
sshpass -e sftp $USER#$HOST << EOF
put '$localpath' '$remotepath'
EOF
When I do echo $localpath it prints the correct file but in the script I get this error:
Connecting to 192.168.X.X...
sftp> put '/Source/Path/2019/Apr/*2019-04-24*' '/Destination/Path/2019/Apr/'
stat /Source/Path/2019/Apr/*2019-04-24*: No such file or directory
How would be the correct regex in this pasrt *$ddate*'.txt' in followingline:
localpath='/Source/Path/'$dyear'/'$dmonth'/'*$ddate*'.txt'
in order to transfer the file "Logfile_2019-04-24_old.txt"?
Thanks in advance
Replace
put '$localpath' '$remotepath'
with
put "$(echo $localpath)" '$remotepath'
to force wildcard (*) replacement in your here-doc.
This does not work if your wildcard is replaced by multiple files.
I don't think you need a regex for this problem. You can get the latest file created in the directory by the following shell command and assign it to your localpath variable.
ls -t directoryPath | head -n1
latestfile=`ls -t /Source/Path/$dyear/$dmonth | head -n1`
localpath='/Source/Path/'$dyear'/'$dmonth'/'$latestfile''
remotepath='/Destination/Path/'$dyear'/'$dmonth'/'
If you are able to get the filename, source and destination directories properly, you can directly use scp to copy the file to remote server:
sshpass -p $PASSWD scp $localpath $USER#$HOST:$remotepath
I'm writing a bash script to send files from a linux server to a remote Windows FTP server.
I would like to check using FTP if the folder where the file will be stored exists before attempting to create it.
Please note that I cannot use SSH nor SCP and I cannot install new scripts on the linux server. Also, for performance issues, I would prefer if checking and creating the folders is done using only one FTP connection.
Here's the function to send the file:
sendFile() {
ftp -n $FTP_HOST <<! >> ${LOCAL_LOG}
quote USER ${FTP_USER}
quote PASS ${FTP_PASS}
binary
$(ftp_mkdir_loop "$FTP_PATH")
put ${FILE_PATH} ${FTP_PATH}/${FILENAME}
bye
!
}
And here's what ftp_mkdir_loop looks like:
ftp_mkdir_loop() {
local r
local a
r="$#"
while [[ "$r" != "$a" ]]; do
a=${r%%/*}
echo "mkdir $a"
echo "cd $a"
r=${r#*/}
done
}
The ftp_mkdir_loop function helps in creating all the folders in $FTP_PATH (Since I cannot do mkdir -p $FTP_PATH through FTP).
Overall my script works but is not "clean"; this is what I'm getting in my log file after the execution of the script (yes, $FTP_PATH is composed of 5 existing directories):
(directory-name) Cannot create a file when that file already exists.
Cannot create a file when that file already exists.
Cannot create a file when that file already exists.
Cannot create a file when that file already exists.
Cannot create a file when that file already exists.
To solve this, do as follows:
To ensure that you only use one FTP connection, you create the input (FTP commands) as an output of a shell script
E.g.
$ cat a.sh
cd /home/test1
mkdir /home/test1/test2
$ ./a.sh | ftp $Your_login_and_server > /your/log 2>&1
To allow the FTP to test if a directory exists, you use the fact that "DIR" command has an option to write to file
# ...continuing a.sh
# In a loop, $CURRENT_DIR is the next subdirectory to check-or-create
echo "DIR $CURRENT_DIR $local_output_file"
sleep 5 # to leave time for the file to be created
if (! -s $local_output_file)
then
echo "mkdir $CURRENT_DIR"
endif
Please note that "-s" test is not necessarily correct - I don't have acccess to ftp now and don't know what the exact output of running DIR on non-existing directory will be - cold be empty file, could be a specific error. If error, you can grep the error text in $local_output_file
Now, wrap the step #2 into a loop over your individual subdirectories in a.sh
#!/bin/bash
FTP_HOST=prep.ai.mit.edu
FTP_USER=anonymous
FTP_PASS=foobar#example.com
DIRECTORY=/foo # /foo does not exist, /pub exists
LOCAL_LOG=/tmp/foo.log
ERROR="Failed to change directory"
ftp -n $FTP_HOST << EOF | tee -a ${LOCAL_LOG} | grep -q "${ERROR}"
quote USER ${FTP_USER}
quote pass ${FTP_PASS}
cd ${DIRECTORY}
EOF
if [[ "${PIPESTATUS[2]}" -eq 1 ]]; then
echo ${DIRECTORY} exists
else
echo ${DIRECTORY} does not exist
fi
Output:
/foo does not exist
If you want to suppress only the messages in ${LOCAL_LOG}:
ftp -n $FTP_HOST <<! | grep -v "Cannot create a file" >> ${LOCAL_LOG}
I have file containing a list of files separated by end of lines
$ cat file_list
file1
file2
file3
I want to copy this list of files with FTP
How can I do that ? Do I have to write a script ?
You can turn your list of files into list of ftp commands easily enough:
(echo open hostname.host;
echo user username;
cat filelist | awk '{ print "put " $1; }';
echo bye) > script.ftp
Then you can just run:
ftp -s script.ftp
Or possibly (with other versions of ftp)
ftp -n < script.ftp
Something along these lines - the somecommand depends on what you want to do - I don't get that from your question, sorry.
#!/bin/bash
# Iterate through lines in file
for line in `cat file.txt`;do
#your ftp command here do something
somecommand $line
done
edit: If you really want to persue this route for multiple files (you shouldn't!), you can use the following command in place of somecommand $line:
ncftpput -m -u username -p password ftp.server.com /remote/folder $line
ncftpput propably also takes an arbitrary number of files to upload in one go, but I havn't checked it. Notice that this approach will connect and disconnect for every single file!
Thanks for the very helpful example of how to feed a list of files to ftp. This worked beautifully for me.
After creating my ftp script in Linux (CentOs 5.5), I ran the script with:
ftp –n < ../script.ftp
My script (with names changed to protect the innocent) starts with:
open <ftpsite>
user <userid> <passwd>
cd <remote directory>
bin
prompt
get <file1>
get <file2>
And ends with:
get <filen-1>
get <filen>
bye