i have question
i hope here get help
this is my makefile
#echo '************ Start ************'
rsync -av -e "ssh -p 22" --delete --exclude=.git $(shell pwd) $(remote_host_10):$(www_dir)
#make create_version -f $(make_file) n=$(n)
#echo '************ Complete ************'
well, i try to do if rsync retrun successful then do make create_version -f $(make_file) n=$(n)
so
i must check rsync results
but i don't know how to do Q_Q
Related
I've seen in posts like rsync - create all missing parent directories? :
rsync -aq --rsync-path='mkdir -p /tmp/imaginary/ && rsync' file user#remote:/tmp/imaginary/
I thought - great, let me try that:
$ rsync -aP --remove-source-files --rsync-path="mkdir -p /home/pi/ARCHIVE/2020/01/24 && rsync" a1.test a1.json a1.pdf /home/pi/ARCHIVE/2020/01/24/
sending incremental file list
rsync: mkdir "/home/pi/ARCHIVE/2020/01/24" failed: No such file or directory (2)
rsync error: error in file IO (code 11) at main.c(675) [Receiver=3.1.2]
Well, sure mkdir "/home/pi/ARCHIVE/2020/01/24" failed - but I did NOT issue mkdir, i issued mkdir -p!
So why did rsync ignore this? Is there any other setting I should set for it? Or maybe rsync-path can be used STRICTLY for ssh connections (which is not the case here)?
--rsync-path only applies to remote machines:
--rsync-path=PROGRAM specify the rsync to run on remote machine
This is because you don't need to invoke a second instance of rsync when you are doing a local copy. The copy will simply be done by the process you ran.
Since it's technically you who are invoking rsync on the target machine, it's you who should be adding mkdir .. && in front of rsync:
mkdir -p /home/pi/ARCHIVE/2020/01/24 &&
rsync -aP --remove-source-files a1.test a1.json a1.pdf /home/pi/ARCHIVE/2020/01/24/
I have written this script:
#!/bin/bash
SSH_USER=${SSH_USER:=$USER}
for department in A B C E L M V
do
mkdir -p ./resources/${div}
rsync -Pruzh --copy-links \
${SSH_USER}#server:${department}/foo/files \
${SSH_USER}#server:${department}/foo/photos \
./resources/${department}/foo
rsync -Pruzh \
${SSH_USER}#server:${department}/bar/documents \
./resources/${department}/bar
done
It works perfect except that I have to write my password 14 times which is not really practical.
I have heard of ssh_agent but for some reasons it does not work on my WSL.
Is there any alternative that I can use to type my password only once?
If you are using openssh, then you can set up a master connection and reuse it with something like:
DEST="${SSH_USER}#server"
TMPL=/tmp/sshctl/"%L-%r#%h:%p"
mkdir -p /tmp/sshctl
if ! ssh -nNf -o ControlMaster=yes -o ControlPath="${TMPL}" "${DEST}"; then
echo "# Failed to setup SSH ControlMaster. Aborting."
exit
fi
# ...
rsync -e "ssh -o 'ControlPath=${TMPL}'" ... "${DEST}":... ...
rsync -e "ssh -o 'ControlPath=${TMPL}'" ... "${DEST}":... ...
# ...
ssh -O exit -o ControlPath="${TMPL}" "${DEST}"
Be sure to secure the socket.
Best practice would be to set up SSH key pairs for automated authentication; i.e. create an SSH key pair and copy the public key to the server where these files are located, then use the private key in the rsync command: rsync -Pruzh --copy-links -e "ssh -i /path/to/private.key" .... This is fairly simple, secure, and gets rid of the prompt.
You can also use a utility like sshpass to enter the password in the prompt, but that kind of approach is less secure.
I'm trying to use SFTP to copy some files from one server to another, this task should run everyweek. The script I use :
HOST='sftp://my.server.com'
USER='user1'
PASSWORD='passwd'
DIR=$HOSTNAME
REMOTE_DIR='/home/remote'
LOCAL_DIR='/home/local'
# LFTP via SFTP connexion
lftp -u "$USER","$PASSWORD" $HOST <<EOF
# changing directory
cd "$REMOTE_DIR"
$(if [ ! -d "$DIR" ]; then
mkdir $DIR
fi)
put -O "$REMOTE_DIR"/$DIR "$LOCAL_DIR"/uploaded.txt
EOF
My issue is that put is executed without taking in consideration the result of if statment.
PS : The error message I got is the following :
put: Access failed: No such file (/home/backups/myhost/upload.txt)
LFTP has no if statement!
What you are doing here?
lftp -u "$USER","$PASSWORD" $HOST <<EOF
cd "$REMOTE_DIR"
$(if [ ! -d "$DIR" ]; then
mkdir $DIR
fi)
put -O "$REMOTE_DIR"/$DIR "$LOCAL_DIR"/uploaded.txt
EOF
You call a sub command in a here document. The sub command is executed locally before lftp is started and its output is pasted in the here document, which gets passed to lftp. This works just, because mkdir has no output. You do not call mkdir on the ftp server. You call the mkdir of your local shell. Effectively it is the same as if you put the if statement before the lftp execution.
if [ ! -d "$DIR" ]; then
mkdir $DIR
fi
lftp -u "$USER","$PASSWORD" $HOST <<EOF
cd "$REMOTE_DIR"
put -O "$REMOTE_DIR"/$DIR "$LOCAL_DIR"/uploaded.txt
EOF
What you are trying to do, does not work. You have to think about a different solution.
Right now I have no FTP server to test it, but it might be possible to use the -f option of LFTP's mkdir. I assume that it may work like the -f option of the Unix rm command. Try this:
lftp -u "$USER","$PASSWORD" $HOST <<EOF
cd "$REMOTE_DIR"
mkdir -f "$DIR"
put -O "$REMOTE_DIR"/$DIR "$LOCAL_DIR"/uploaded.txt
EOF
Update: It works as supposed. The creation of a directory, which exist already, throws no error, if you use the option -f:
lftp anonymous#localhost:/pub> mkdir -f dir
mkdir ok, `dir' created
lftp anonymous#localhost:/pub> mkdir -f dir
lftp anonymous#localhost:/pub> ls
drwx------ 2 116 122 4096 Aug 10 12:04 dir
Maybe you lftp client is outdated. I tested it with Debian 9.
I have a script that transfers files. Everytime I run it It needs to connect to a different host. That's why I'm adding the host as parameter.
The script is executed as: ./transfer.sh <hostname>
#!/bin/bash -evx
SSH="ssh \
-o UseRoaming=no \
-o UserKnownHostsFile=/dev/null \
-o StrictHostKeyChecking=no \
-i ~/.ssh/privateKey.pem \
-l ec2-user \
${1}"
files=(
file1
file2
)
files="${files[#]}"
# this works
$SSH
# this does not work
rsync -avzh --stats --progress $files -e $SSH:/home/ec2-user/
# also this does not work
rsync -avzh --stats --progress $files -e $SSH ec2-user#$1:/home/ec2-user/
I can properly connect with the ssh connection stored in $SSH, but the rsync connection attempts fails because of the wrong key:
Permission denied (publickey).
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2]
What would be the correct syntax for the rsync connection?
Write set -x before the rsync line and watch how the arguments are expanded. I believe it will be wrong.
You need to enclose the ssh command with arguments (without hostname) into the quotes, otherwise the arguments will get passed to the rsync command and not to the ssh.
My solution after Jakuje pointed me in the right direction:
#!/bin/bash -evx
host=$1
SSH="ssh \
-o UseRoaming=no \
-o UserKnownHostsFile=/dev/null \
-o StrictHostKeyChecking=no \
-i ~/.ssh/privateKey.pem \
-l ec2-user"
files=(
file1
file2
)
files="${files[#]}"
# transfer all in one rsync connection
rsync -avzh --stats --progress $files -e "$SSH" $host:/home/ec2-user/
# launch setup script
$SSH $host ./setup.sh
I have a script that try to mirror a specific directory from a local server to a remote one. It looks like that:
inotifywait -mr --format '%w%f' -e close_write -e moved_to -e delete /mydir | \
while read FILECHANGE
do
if [ -f $FILECHANGE ]
then
rsync --bwlimit=4096 --progress --relative -vrae 'ssh -p 22' $FILECHANGE $REMOTEHOST:/
else
ssh -p 22 $REMOTEHOST "rm $FILECHANGE"
fi
done
In case of multiple create of files, as for example a touch command:
touch 1 2 3
The 3 files are well transfered.
But if I delete several files at once:
rm -f 1 2 3
Only the first 1 is deleted.
If I replace the ssh command by just an echo $FILECHANGE, the 3 files are well displayed in the console. So it seems the problem come from the ssh command, but I can't explain why and solve it.
Anyone as an idea?
Well, I found the issue: it seems that the ssh command was eating the output of the inotifywait command when run. So, to prevent that, I add the 0<&- redirection after the ssh, to close the stdin.
inotifywait -mr --format '%w%f' -e close_write -e moved_to -e delete /mydir | \
while read FILECHANGE
do
if [ -f $FILECHANGE ]
then
rsync --bwlimit=4096 --progress --relative -vrae 'ssh -p 22' $FILECHANGE $REMOTEHOST:/
else
ssh -p 22 $REMOTEHOST "rm $FILECHANGE" 0<&-
fi
done
Now it works.