I wrote a bash script rm_remote_file.sh that ssh to a set of remote machines and delete a file. I used & at the end of each function call to run these command in parallel, the script looks like this:
#!/bin/bash
rm_remote_file() {
echo "Removing file on node $1:"
ssh $1 'rm ~/test_file'
}
for node in node1.com node2.com node3.com; do
rm_remote_file $node &
done
When test_file exists on each node - the rm command succeeds - the output of this script is:
Removing file on node node1.com:
Removing file on node node2.com:
Removing file on node node3.com:
I prefer having each hostname printed out. However, if the test_file does not exist on each node - the rm command fails - the output of this script is:
rm: cannot remove ‘~/test_file’: No such file or directory
rm: cannot remove ‘~/test_file’: No such file or directory
rm: cannot remove ‘~/test_file’: No such file or directory
So the printing of the nodes' host names are suppressed by this error message. I think this behavior has something to do with the I/O redirection and using things such as 2>&1 can solve the issue. But I would like to know why ssh command error message would suppress the echo command.
Note that this only happens with ssh command, the following script, which just removes some local files, would outputs both "Removing file" and the "No such file or directory".
#!/bin/bash
rm_file() {
echo "Removing file..."
rm ./$1
}
for file in test1 test2 test3 test4 test5; do
rm_file $file &
done
Add 2>&1 to ssh command, to that message and error stay in order:
ssh $1 'rm ~/test_file' 2>&1
Solution after discussion:
rm_remote_file() {
echo "Removing file on node $1: $( ssh $1 'rm ~/test_file' 2>&1 )"
}
Related
I'm connected to a remote machine via SSH as part of a bash script. After navigating to the directory, I run ls which confirms matching files are found. However, I then try to loop through the files and run other commands on them, and the variable is now empty.
Code:
echo "DOING STUFF!"
cd /mnt/slowdata/ls8_processing
ls
for f in *.tar.gz
do
echo $f
done
Output:
DOING STUFF!
LC080330242019031901T1-SC20190606111327.tar.gz
LC080330242019042001T1-SC20190606111203.tar.gz
LC080330242019052201T1-SC20190606111130.tar.gz
LC080330252019030301T2-SC20190606111021.tar.gz
LC080330252019031901T1-SC20190606120750.tar.gz
LC080340232019031001T1-SC20190606111056.tar.gz
LC080340232019041101T1-SC20190606111215.tar.gz
LC080340242019031001T1-SC20190606111201.tar.gz
LC080340242019041101T1-SC20190606111250.tar.gz
LC080340242019052901T1-SC20190606111331.tar.gz
As can be seen via the output, the $f is picking something up, as there are the correct number of blank lines. However I wish to untar each file which I cannot do.
TIA.
You have to remove special meaning of $ to pass it to the remote host as '$' else the variable will be expanded before you send the command to the remote host.
Keep in mind the for cycle will run regardless of whether the cd was successful.
ssh server1 << EOF
cd /mnt/slowdata/ls8_processing
ls
for f in *.tar.gz
do
echo \$f
done
EOF
My example show the difference:
script.sh
#!/bin/bash
f=123
ssh -i .ssh/keyauth.pem root#server1 << EOF
for f in ./*.log
do
echo "\$f"
echo "$f"
done
EOF
Output
[edvin#server2 ~]$ ./script.sh
./sepap-install.log
123
./sepfl-upgrade.log
123
./sep-install.log
123
./sepjlu-install.log
123
./sepui-install.log
123
I'm trying to run the following command:
ssh -A -t -i ~/.ssh/DevKP.pem -o StrictHostKeyChecking=no root#MyServer "for file in \`ls /root/spark/work/ \`; do echo 'file - ' $file; done"
The output is:
file -
file -
Connection to MyServer closed.
When I ran the command on the remote server itself:
for file in `ls /root/spark/work/ `; do echo 'file - ' $file; done
I get the output:
file - test1.txt
file - test2.txt
How do I get ti to work on the local server? it seems that it gets the right files (because there were two sysouts)
anyone has any idea?
thanks
You need to escape the $ in $file to make sure the remote shell interprets it instead of your local. You should also simplify the ls /root/.. to for file in /root/../*:
ssh root#MyServer "for file in /root/spark/work/* ; do echo 'file - ' \$file; done"
I want to create a directory if it doesn't exists after login to sftp server.
test.sh
sftp name#example.com << EOF
mkdir test
put test.xml
bye
EOF
Now i call test.sh and upload different files each time to test folder. When running this
mkdir test
First time it works and second time it throws Couldn't create directory: Failure error?
How to create a directory if doesn't exists and if exists don't create directory in sftp.
man 1 sftp (from openssh-client package):
-b batchfile
Batch mode reads a series of commands from an input
batchfile instead of stdin. Since it lacks user
interaction it should be used in conjunction with
non-interactive authentication. A batchfile of ‘-’
may be used to indicate standard input. sftp will
abort if any of the following commands fail: get,
put, reget, reput, rename, ln, rm, mkdir, chdir, ls,
lchdir, chmod, chown, chgrp, lpwd, df, symlink, and
lmkdir. Termination on error can be suppressed on a
command by command basis by prefixing the command
with a ‘-’ character (for example, -rm /tmp/blah*).
So:
{
echo -mkdir dir1
echo -mkdir dir1/dir2
echo -mkdir dir1/dir2/dir3
} | sftp -b - $user#$host
I understand this thread is old and has been marked as answered but the answer did not work in my case. The second page on google for a search regarding "sftp checking for directory" so here is an update that would have saved me a few hours.
Using an EOT you cannot capture the error code resulting from the directory not being found. The work around I found was to create a file containing instructions for the call and then capture the result of that automated call.
The example below using sshpass but my script also uses this same method authenticating with sshkeys.
Create the file containing the instructions:
echo "cd $RemoteDir" > check4directory
cat check4directory; echo "bye" >> check4directory
Set permissions:
chmod +x check4directory
Then make the connection using the batch feature:
export SSHPAA=$remote_pass
sshpass -e sftp -v -oBatchMode=no -b check4directory $remote_user#$remote_addy
Lastly check for the error code:
if [ $? -ge "1" ] ; then
echo -e "The remote directory was not found or the connection failed."
fi
At this point you can exit 1 or initiate some other action. Note that if the SFTP connection fails for another reason like password or the address is incorrect the error will trip the action.
Another variant is to split the SFTP session into two.
First SFTP session simply issues the MKDIR command.
Second SFTP session can then assume existence of the directory and put the files.
You can use the SSH access of your account to first verify if the directory exists at all (using the "test" command). If it returns exit code 0, the dir exists, otherwise it doesn't. You can act on that accordingly.
# Both the command and the name of your directory are "test"
# To avoid confusion, I just put the directory in a separate variable
YOURDIR="test"
# Check if the folder exists remotely
ssh name#example.com "test -d $YOURDIR"
if [ $? -ne 0 ]; then
# Directory does not exist
sftp name#example.com << EOF
mkdir test
put test.xml
bye
EOF
else
# Directory already exists
sftp name#example.com << EOF
put test.xml
bye
EOF
fi
Try this to ignore errors if directory already exists.
# Turn OFF error
set +e
# Create remote dirs
sftp -P 22 -o StrictHostKeyChecking=no -oIdentityFile=key.pem -v $user#$host <<EOF
mkdir <remote_path> # create remote directory
bye
EOF
# Turn ON error
set -e
# Do upload to SFTP
sftp -P 22 -o StrictHostKeyChecking=no -oIdentityFile=key.pem -v $user#$host <<EOF
cd <remote_path> # remote_path
put <local_file_path> # local_path
quit
EOF
I'm writing a bash script to send files from a linux server to a remote Windows FTP server.
I would like to check using FTP if the folder where the file will be stored exists before attempting to create it.
Please note that I cannot use SSH nor SCP and I cannot install new scripts on the linux server. Also, for performance issues, I would prefer if checking and creating the folders is done using only one FTP connection.
Here's the function to send the file:
sendFile() {
ftp -n $FTP_HOST <<! >> ${LOCAL_LOG}
quote USER ${FTP_USER}
quote PASS ${FTP_PASS}
binary
$(ftp_mkdir_loop "$FTP_PATH")
put ${FILE_PATH} ${FTP_PATH}/${FILENAME}
bye
!
}
And here's what ftp_mkdir_loop looks like:
ftp_mkdir_loop() {
local r
local a
r="$#"
while [[ "$r" != "$a" ]]; do
a=${r%%/*}
echo "mkdir $a"
echo "cd $a"
r=${r#*/}
done
}
The ftp_mkdir_loop function helps in creating all the folders in $FTP_PATH (Since I cannot do mkdir -p $FTP_PATH through FTP).
Overall my script works but is not "clean"; this is what I'm getting in my log file after the execution of the script (yes, $FTP_PATH is composed of 5 existing directories):
(directory-name) Cannot create a file when that file already exists.
Cannot create a file when that file already exists.
Cannot create a file when that file already exists.
Cannot create a file when that file already exists.
Cannot create a file when that file already exists.
To solve this, do as follows:
To ensure that you only use one FTP connection, you create the input (FTP commands) as an output of a shell script
E.g.
$ cat a.sh
cd /home/test1
mkdir /home/test1/test2
$ ./a.sh | ftp $Your_login_and_server > /your/log 2>&1
To allow the FTP to test if a directory exists, you use the fact that "DIR" command has an option to write to file
# ...continuing a.sh
# In a loop, $CURRENT_DIR is the next subdirectory to check-or-create
echo "DIR $CURRENT_DIR $local_output_file"
sleep 5 # to leave time for the file to be created
if (! -s $local_output_file)
then
echo "mkdir $CURRENT_DIR"
endif
Please note that "-s" test is not necessarily correct - I don't have acccess to ftp now and don't know what the exact output of running DIR on non-existing directory will be - cold be empty file, could be a specific error. If error, you can grep the error text in $local_output_file
Now, wrap the step #2 into a loop over your individual subdirectories in a.sh
#!/bin/bash
FTP_HOST=prep.ai.mit.edu
FTP_USER=anonymous
FTP_PASS=foobar#example.com
DIRECTORY=/foo # /foo does not exist, /pub exists
LOCAL_LOG=/tmp/foo.log
ERROR="Failed to change directory"
ftp -n $FTP_HOST << EOF | tee -a ${LOCAL_LOG} | grep -q "${ERROR}"
quote USER ${FTP_USER}
quote pass ${FTP_PASS}
cd ${DIRECTORY}
EOF
if [[ "${PIPESTATUS[2]}" -eq 1 ]]; then
echo ${DIRECTORY} exists
else
echo ${DIRECTORY} does not exist
fi
Output:
/foo does not exist
If you want to suppress only the messages in ${LOCAL_LOG}:
ftp -n $FTP_HOST <<! | grep -v "Cannot create a file" >> ${LOCAL_LOG}
Consider the following simple rsync script I am tryint to slap up:
#!/bin/bash
PROJECT="$1"
USER=stef
LOCAL_DIR="~/drupal-files/"
REMOTE_HOST="hostname.com"
REMOTE_PROJECTS_PATH=""
# Should not have anything to change below
PROJECT_LIST="proj1 proj2 proj3 quit"
echo "/nSelect project you wish to rsync\n\n"
select PROJECT in $PROJECT_LIST
do
if [ "$PROJECT" = "quit" ]; then
echo
echo "Quitting $0"
echo
exit
fi
echo "Rsynching $PROJECT from $REMOTE_HOST into" $LOCAL_DIR$PROJECT
rsync -avzrvP $USER#$REMOTE_HOST:/var/projects/$PROJECT/ $LOCAL_DIR$PROJECT
done
echo "Rsync complete."
exit;
The variable $LOCALDIR$PROJECT set in the rsync command always includes the scripts path, :
OUTPUT:
Rsynching casa from hostname.com.com into ~/drupal-files/casa
opening connection using: ssh -l stef hostname.com rsync --server --sender -vvlogDtprz e.iLsf . /var/groupe_tva/casa/
receiving incremental file list
rsync: mkdir "/home/stef/bin/~/drupal-files/proj1" failed: No such file or directory (2)
rsync error: error in file IO (code 11) at main.c(605) [Receiver=3.0.9]
The line with mkdir should not have /home/stef/bin, why is bash adding the script's running dir on the variable?
Thanks
LOCAL_DIR="~/drupal-files/"
The string is in quotes so there's pathname expansion, and the variable will contain the literal string.
Remove the quotes.
$ x="~/test"; echo $x
~/test
$ x=~/test; echo $x
/home/user/test