I try to do a simple rsync command and to exclude directories that have a flagged file in it. The flagged flagged.txt is a text file contains only *. This command below run fine in terminal:
rsync -avP --delete --filter="dir-merge,n- flagged.txt" \
"/mnt/c/Documents/somefolder" "/mnt/j/"
However when I put in in bash script, the flagged is ignored.
Tried
rsync -avP --delete --filter='dir-merge,n- flagged.txt' \
"/mnt/c/Documents/somefolder" "/mnt/j/"
and
filter=(--filter="dir-merge,n- flagged.txt")
rsync -avP --delete "${filter[#]}" "/mnt/c/Documents/somefolder" "/mnt/j/"
and
filter=(--filter='dir-merge,n- flagged.txt')
rsync -avP --delete "${filter[#]}" "/mnt/c/Documents/somefolder" "/mnt/j/"
All failed. Any idea? Thanks in advance!
Related
I have this commandline in Cygwin's terminal on Windows to rsync files from Win to Linux:
/usr/bin/rsync -avh --progress --no-whole-file --partial --no-owner --no-group --no-perms --omit-dir-times --log-file=/home/Stefan-Admin/transfer_d.txt /cygdrive/d/* root#172.16.1.224:/mnt/disc_1/drive_d/. > backup_to_d_drive_log.txt
This works perfectly if run by hand in the Cygwin terminal to copy the disc to a remote Ubuntu machine.
However, placing the above in a script in Cygwin terminal, fails...
E. g.
set -e
set -x
timestart=$(date)
/usr/bin/rsync -avh --progress --no-whole-file --partial --no-owner --no-group --no-perms --omit-dir-times --log-file=/home/Stefan-Admin/transfer_d.txt /cygdrive/d/* root#172.16.1.224:/mnt/disc_1/drive_d/. > backup_to_d_drive_log.txt
in a file with rwx permissions in Cygwin e. g. called, say
do_copy.sh
causes Cygwin rsync to report it cannot find /cygdrive/d ?
This is what is shown if the script file is run via "bash -f do_copy.sh" in the Cygwin terminal:
+ /usr/bin/rsync -avh --progress --no-whole-file --partial --no-owner --no-group --no-perms --omit-dir-times --log-file=/home/Stefan-Admin/transfer_d.txt '/cygdrive/d/*' root#172.16.1.224:/mnt/disc_1/drive_d/.
rsync: [sender] link_stat "/cygdrive/d/*" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1330) [sender=3.2.4dev]
I note that the rsync source specification, written as
/cygdrive/d/*
when run in bash in the Cygwin terminal, suddenly becomes
'cygdrive/d/*'
if run from a .sh script via "bash -f do_copy.sh" in the Cygwin terminal.
/cygdrive/d does exist:
$ df -mh
Filesystem Size Used Avail Use% Mounted on
D: 1.6T 606G 954G 39% /cygdrive/d
$
as proven by the fact that if the above rsync command is ran "straight" in Cygwin terminal, it DOES work...
... but if you run the exact same command in a .sh file via
$ bash -f do_copy.sh
in the Cygwin terminal, rsync suddenly cannot find /cygdrive/d anywhere.
Why is that?
How can it be fixed so that rsync can be run out of a bash script on the Cygwin terminal?
Thanks!
Stefan
Ok, the solution to this issue with running rsync inside a Cygwin BASH script, when rsync cannot find
/cygdrive/d/*
is to change the rsync statement in the BASH script file to reference as rsync source
/cygdrive/d/.
e. g. the correct, working rsync commandline to rsync from Windows to Linux from a Cygwin bash script is
/usr/bin/rsync -avh --progress --no-whole-file --partial --no-owner --no-group --no-perms --omit-dir-times --log-file=/home/Stefan-Admin/transfer_d.txt /cygdrive/d/. root#172.16.1.224:/mnt/disc_1/drive_d/. > backup_to_d_drive_log.txt
After removing the wildcard * from my commandline, I can now rsync with no problems from my Windows machine to my Linux machine from a Cygwin bash -script-.
If executing the command by hand in the Cygwin terminal, using
/usr/bin/rsync -avh --progress --no-whole-file --partial --no-owner --no-group --no-perms --omit-dir-times --log-file=/home/Stefan-Admin/transfer_d.txt /cygdrive/d/* root#172.16.1.224:/mnt/disc_1/drive_d/. > backup_to_d_drive_log.txt
still works, but the "*" -must- be replaced with "." after the "/cygdrive/d/" string if running rsync from inside a Cygwin BASH script.
Hope this helps someone.
Stefan
I've seen in posts like rsync - create all missing parent directories? :
rsync -aq --rsync-path='mkdir -p /tmp/imaginary/ && rsync' file user#remote:/tmp/imaginary/
I thought - great, let me try that:
$ rsync -aP --remove-source-files --rsync-path="mkdir -p /home/pi/ARCHIVE/2020/01/24 && rsync" a1.test a1.json a1.pdf /home/pi/ARCHIVE/2020/01/24/
sending incremental file list
rsync: mkdir "/home/pi/ARCHIVE/2020/01/24" failed: No such file or directory (2)
rsync error: error in file IO (code 11) at main.c(675) [Receiver=3.1.2]
Well, sure mkdir "/home/pi/ARCHIVE/2020/01/24" failed - but I did NOT issue mkdir, i issued mkdir -p!
So why did rsync ignore this? Is there any other setting I should set for it? Or maybe rsync-path can be used STRICTLY for ssh connections (which is not the case here)?
--rsync-path only applies to remote machines:
--rsync-path=PROGRAM specify the rsync to run on remote machine
This is because you don't need to invoke a second instance of rsync when you are doing a local copy. The copy will simply be done by the process you ran.
Since it's technically you who are invoking rsync on the target machine, it's you who should be adding mkdir .. && in front of rsync:
mkdir -p /home/pi/ARCHIVE/2020/01/24 &&
rsync -aP --remove-source-files a1.test a1.json a1.pdf /home/pi/ARCHIVE/2020/01/24/
I am using fswatch and rsync to automatically sync directories between Mac and ubuntu servers.
The shell is like
/Users/david/Documents/code/shell/fswatch_rsync.sh:
#!/bin/bash
sudo fswatch -o /Users/david/Documents/code/python/ | xargs -n1 /Users/david/Documents/code/shell/rsync_shell.sh
/Users/david/Documents/code/shell/rsync_shell.sh:
sudo rsync -vzrp --delete /Users/david/Documents/code/python/ --password-file=/etc/rsyncd.pass -e "ssh -i /Users/david/.ssh/id_rsa -p 22" 192.168.0.109::python
However, when I run bash /Users/david/Documents/code/shell/fswatch_rsync.sh in terminal and change the files in /Users/david/Documents/code/python/, it responses:
xargs: /Users/david/Documents/code/shell/rsync_shell.sh: No such file or directory
in the terminal.
I have a bash script that’s using rsync to backup some files from my local desktop to a remote machine on my LAN.
I have the main script with some customisable variables in a separate .sh file to make for easy maintenance, deployment and git management.
So I have this dir structure
sync-backup-to-cp.sh
config/settings.sh
And the following code to include the config/settings.sh into the main sync-backup-to-cp.sh
#! /bin/bash
#load variables file
source /Users/enwhat/Dropbox/Flex/Scripts/mac/rysnc-backup-to-cp/config/settings.sh
However the imported variables aren’t behaving as expected. If I have any space in any of the variables it throws an error about the variables being invalid. It seems that bash is interpreting this oddly.
Ie. rsync_opts="--verbose --archive” will cause the script to break and run an error such as “invalid numeric arguments or unknown arguments supplied”. Where as rsync_opts="--verbose” runs perfectly.
To help illustrate the script I've taken some snippets of the code showing the flow with help so far
from: config / settings.sh
RSYNC_OPTS=( --bwlimit=1000 --verbose )
from my main script, there's a function call where these variables are passed in.
backup "$RSYNC_BIN" "$BACKUP_FILE_LIST" "$EXCLUDE_FILE_LIST" "$SSH_PORT" "$SSH_KEY" "$SOURCE" "$DESTINATION" "$RSYNC_OPTS[*]"
then the full function
function backup(){ #uses rsync to backup to server
#takes 8 args 1
#define local vars
local l_rsync_bin=$1
local l_rsync_backup_file_list=$2
local l_rsync_exclude_file_list=$3
local l_rsync_ssh_port=$4
local l_rsync_ssh_key=$5
local l_rsync_source=$6
local l_rsync_dest=$7
local l_rsync_opts=$8
#local l_time
#l_time=$(date)
#caffinate stops system from sleeping
echo ""$l_rsync_bin" "$l_rsync_opts" --verbose --archive --recursive --numeric-ids --human-readable --partial --progress --relative --itemize-changes --stats --rsync-path="sudo rsync" --delete-during --files-from="${l_rsync_backup_file_list}" --exclude-from="${l_rsync_exclude_file_list}" -e "ssh -q -p ${l_rsync_ssh_port} -i ${l_rsync_ssh_key}" "${l_rsync_source}" "${l_rsync_dest}""
caffeinate -s "$l_rsync_bin" "$l_rsync_opts" --verbose --archive --recursive --numeric-ids --human-readable --partial --progress --relative --itemize-changes --stats --rsync-path="sudo rsync" --delete-during --files-from="${l_rsync_backup_file_list}" --exclude-from="${l_rsync_exclude_file_list}" -e "ssh -q -p ${l_rsync_ssh_port} -i ${l_rsync_ssh_key}" "${l_rsync_source}" "${l_rsync_dest}"
}
Since you are quoting $rsync_opts, the entire value is passed as a single, whitespace-containing argument to rsync. In order for each option to be passed as a separate argument, you need to leave the parameter expansion unquoted:
rsync $rsync_opts
However, you can't include arguments that actually contain whitespace like this; all whitespace is treated by the shell as separating arguments. The right way to store arguments is to use an array:
rsync_opts=( --verbose --archive )
rsync "${rsync_opts[#]}"
It may not be necessary for your current use case, but it's a good idea to get into the habit of doing things the right way to avoid nasty surprises later.
For example,
local -a l_rsync_opts
l_rsync_opts=(--bwlimit=1000 --verbose --rsync-path="sudo rsync")
UPDATE: Based on your edit, you need to do the following:
backup ... "${RSYNC_OPTS[#]}" # #, not *
# Note the changes involving l_rsync_opts
function backup(){ #uses rsync to backup to server
#takes 8 args 1
#define local vars
local l_rsync_bin=$1
local l_rsync_backup_file_list=$2
local l_rsync_exclude_file_list=$3
local l_rsync_ssh_port=$4
local l_rsync_ssh_key=$5
local l_rsync_source=$6
local l_rsync_dest=$7
local l_rsync_opts=( "${#:8}" )
#local l_time
#l_time=$(date)
#caffinate stops system from sleeping
echo ""$l_rsync_bin" "${l_rsync_opts[#]}" --verbose --archive --recursive --numeric-ids --human-readable --partial --progress --relative --itemize-changes --stats --rsync-path="sudo rsync" --delete-during --files-from="${l_rsync_backup_file_list}" --exclude-from="${l_rsync_exclude_file_list}" -e "ssh -q -p ${l_rsync_ssh_port} -i ${l_rsync_ssh_key}" "${l_rsync_source}" "${l_rsync_dest}""
caffeinate -s "$l_rsync_bin" "${l_rsync_opts[#]}" --verbose --archive --recursive --numeric-ids --human-readable --partial --progress --relative --itemize-changes --stats --rsync-path="sudo rsync" --delete-during --files-from="${l_rsync_backup_file_list}" --exclude-from="${l_rsync_exclude_file_list}" -e "ssh -q -p ${l_rsync_ssh_port} -i ${l_rsync_ssh_key}" "${l_rsync_source}" "${l_rsync_dest}"
}
I am trying to write a simple bash script for my local (Mac OS X) machine to move files from a directory on my machine to a remote machine. This line is failing:
rsync --verbose --progress --stats --compress --rsh=ssh \
--recursive --times --perms --links --delete \
--exclude "*bak" --exclude "*~" \
/repository/* $DEV_SERVER:$REMOTE_DIR
$DEV_SERVER and $REMOTE_DIR are defined previously, and I echo them to verify they're accurate.
The error I'm getting is:
rsync: link_stat /Users/myusername/mycurrentdirectory failed: No such file or directory (2)
To note here is that rather than using the defined directory (/repository, which is in the root of the machine), it uses my working directory. What is causing this?
Check that your \ characters have no whitespace after them at the end of the line. This will cause BASH to not interpret the line wrap correctly, giving the rsync error above.
Remove the '*' from the source location, rsync knows to look in the inside of the directory if you specify the '/' in the end
like that:
rsync --verbose --progress --stats --compress --rsh=ssh --recursive --times --perms --links --delete --exclude "*bak" --exclude "*~" /repository/ $DEV_SERVER:$REMOTE_DIR
I was encountering the same error while doing some rsync work. I had the wrong character for specifying options which I must have gotten from copying and pasting the command from elsewhere:
−
rather than the correct character below:
-
The source path (here: /repository/*) does contain a wildcard (*).
There is some info in the man pages for rsync:
"Note that the expansion of wildcards on the commandline (*.c) into a list of files is handled by the shell before it runs rsync and not by rsync itself (exactly the same as all other posix-style programs)."
I.e. in case you put in bash double quotes "" around the (source path) including the wildcard symbol (*), the rsync commandwill succeed on command line interface, while inside a bash shell it will yield exactly the error described here.
rsync: link_stat "<source path>*" failed: No such file or directory (2)
=> make sure to not put the wild card(s) in quotes in an rsync source path.
This:
rsync --verbose --progress --stats --compress --rsh=ssh \
--recursive --times --perms --links --delete \
--exclude "*bak" --exclude "*~" \
/repository/* $DEV_SERVER:$REMOTE_DIR
should be this:
rsync --verbose --progress --stats --compress --rsh=ssh --recursive --times --perms --links --delete --exclude "*bak" --exclude "*~" /repository/* $DEV_SERVER:$REMOTE_DIR
Bash interprets the \ character differently than the command line, or perhaps there is an implicit non-whitespace character after it.