I cant get my rsync script to work if source or destination variable have a space in the folder name
#!/bin/bash
HOST="root#192.168.1.100"
source="/path/My Files/"
dest="/path/My Files2/"
rsync -avhP --delete "$HOST":"$source" "$dest"
the error i get is
receiving incremental file list
rsync: [sender] link_stat "/path/My" failed: No such file or directory (2)
rsync: [sender] change_dir "/root/Files" failed: No such file or directory (2)
Any help how to write this would be great
Add option -s with current rsync version.
From man rsync:
-s, --protect-args: This option sends all filenames and most options to the remote rsync without allowing the remote shell to interpret them. This means that spaces are not split
in names, and any non-wildcard special characters are not translated (such as ~, $, ;, &, etc.). Wildcards are expanded on the remote host by rsync (instead of
the shell doing it).
I'm trying to download a single directory from a remote ftp server. I'm using this command
wget -nH -r -N -l inf --ask-password ftp://ftp.server.com/some/remote/dir/xyz -P dirName
I'd like the remote xyz directory to be copied and called dirName. There is a local directory called dirName, but its contents are dirName/some/remote/dir/xyz, which is not what I wanted.
After a careful reading of the man page I found the --cut-dirs option which cuts parent directories from the local storage. That does what I need it to do!
Answers to this question do not seem to work for me to parallelize rsync from my local machine to an AWS instance.
I have done a dry run and obtained a filelist with filepaths for each file to be transferred, split this filelist into 59 smaller filelists residing in the split directory. I then created a new text file with the name of each filelist, split_list.txt. I'm trying to open up this filelist of filelists to pipe into 59 parallel rsync runs:
cat split_list.txt | parallel --will-cite -j 59 rsync --relative --human-readable --ignore-existing --files-from={} -rave "ssh -i key.pem" /Volumes/BRIENNAKH\ 4/xml/ [REMOTE-HOST]:~/arxiv/xml
I thought that by doing cat split_list.txt I would be passing to the second argument the path of each filelist in the split directory, and that the --files-from={} argument would be receiving each filelist.
It doesn't run and instead returns
Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others.
<http://rsync.samba.org/>
Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,
inplace, IPv6, 64-bit system inums, 64-bit internal inums
rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you
are welcome to redistribute it under certain conditions. See the GNU
General Public Licence for details.
rsync is a file transfer program capable of efficient remote update
via a fast differencing algorithm.
Usage: rsync [OPTION]... SRC [SRC]... DEST
or rsync [OPTION]... SRC [SRC]... [USER#]HOST:DEST
or rsync [OPTION]... SRC [SRC]... [USER#]HOST::DEST
or rsync [OPTION]... SRC [SRC]... rsync://[USER#]HOST[:PORT]/DEST
or rsync [OPTION]... [USER#]HOST:SRC [DEST]
or rsync [OPTION]... [USER#]HOST::SRC [DEST]
or rsync [OPTION]... rsync://[USER#]HOST[:PORT]/SRC [DEST]
The ':' usages connect via remote shell, while '::' & 'rsync://' usages connect
to an rsync daemon, and require SRC or DEST to start with a module name.
Options
-v, --verbose increase verbosity
-q, --quiet suppress non-error messages
--no-motd suppress daemon-mode MOTD (see manpage caveat)
-c, --checksum skip based on checksum, not mod-time & size
-a, --archive archive mode; same as -rlptgoD (no -H)
--no-OPTION turn off an implied OPTION (e.g. --no-D)
-r, --recursive recurse into directories
-R, --relative use relative path names
--no-implied-dirs don't send implied dirs with --relative
-b, --backup make backups (see --suffix & --backup-dir)
--backup-dir=DIR make backups into hierarchy based in DIR
--suffix=SUFFIX set backup suffix (default ~ w/o --backup-dir)
-u, --update skip files that are newer on the receiver
--inplace update destination files in-place (SEE MAN PAGE)
--append append data onto shorter files
-d, --dirs transfer directories without recursing
-l, --links copy symlinks as symlinks
-L, --copy-links transform symlink into referent file/dir
--copy-unsafe-links only "unsafe" symlinks are transformed
--safe-links ignore symlinks that point outside the source tree
-k, --copy-dirlinks transform symlink to a dir into referent dir
-K, --keep-dirlinks treat symlinked dir on receiver as dir
-H, --hard-links preserve hard links
-p, --perms preserve permissions
--executability preserve the file's executability
--chmod=CHMOD affect file and/or directory permissions
-o, --owner preserve owner (super-user only)
-g, --group preserve group
--devices preserve device files (super-user only)
--specials preserve special files
-D same as --devices --specials
-t, --times preserve times
-O, --omit-dir-times omit directories when preserving times
--super receiver attempts super-user activities
-S, --sparse handle sparse files efficiently
-n, --dry-run show what would have been transferred
-W, --whole-file copy files whole (without rsync algorithm)
-x, --one-file-system don't cross filesystem boundaries
-B, --block-size=SIZE force a fixed checksum block-size
-e, --rsh=COMMAND specify the remote shell to use
--rsync-path=PROGRAM specify the rsync to run on the remote machine
--existing skip creating new files on receiver
--ignore-existing skip updating files that already exist on receiver
--remove-source-files sender removes synchronized files (non-dirs)
--del an alias for --delete-during
--delete delete extraneous files from destination dirs
--delete-before receiver deletes before transfer (default)
--delete-during receiver deletes during transfer, not before
--delete-after receiver deletes after transfer, not before
--delete-excluded also delete excluded files from destination dirs
--ignore-errors delete even if there are I/O errors
--force force deletion of directories even if not empty
--max-delete=NUM don't delete more than NUM files
--max-size=SIZE don't transfer any file larger than SIZE
--min-size=SIZE don't transfer any file smaller than SIZE
--partial keep partially transferred files
--partial-dir=DIR put a partially transferred file into DIR
--delay-updates put all updated files into place at transfer's end
-m, --prune-empty-dirs prune empty directory chains from the file-list
--numeric-ids don't map uid/gid values by user/group name
--timeout=TIME set I/O timeout in seconds
-I, --ignore-times don't skip files that match in size and mod-time
--size-only skip files that match in size
--modify-window=NUM compare mod-times with reduced accuracy
-T, --temp-dir=DIR create temporary files in directory DIR
-y, --fuzzy find similar file for basis if no dest file
--compare-dest=DIR also compare destination files relative to DIR
--copy-dest=DIR ... and include copies of unchanged files
--link-dest=DIR hardlink to files in DIR when unchanged
-z, --compress compress file data during the transfer
--compress-level=NUM explicitly set compression level
-C, --cvs-exclude auto-ignore files the same way CVS does
-f, --filter=RULE add a file-filtering RULE
-F same as --filter='dir-merge /.rsync-filter'
repeated: --filter='- .rsync-filter'
--exclude=PATTERN exclude files matching PATTERN
--exclude-from=FILE read exclude patterns from FILE
--include=PATTERN don't exclude files matching PATTERN
--include-from=FILE read include patterns from FILE
--files-from=FILE read list of source-file names from FILE
-0, --from0 all *-from/filter files are delimited by 0s
--address=ADDRESS bind address for outgoing socket to daemon
--port=PORT specify double-colon alternate port number
--sockopts=OPTIONS specify custom TCP options
--blocking-io use blocking I/O for the remote shell
--stats give some file-transfer stats
-8, --8-bit-output leave high-bit chars unescaped in output
-h, --human-readable output numbers in a human-readable format
--progress show progress during transfer
-P same as --partial --progress
-i, --itemize-changes output a change-summary for all updates
--out-format=FORMAT output updates using the specified FORMAT
--log-file=FILE log what we're doing to the specified FILE
--log-file-format=FMT log updates using the specified FMT
--password-file=FILE read password from FILE
--list-only list the files instead of copying them
--bwlimit=KBPS limit I/O bandwidth; KBytes per second
--write-batch=FILE write a batched update to FILE
--only-write-batch=FILE like --write-batch but w/o updating destination
--read-batch=FILE read a batched update from FILE
--protocol=NUM force an older protocol version to be used
-E, --extended-attributes copy extended attributes
--cache disable fcntl(F_NOCACHE)
-4, --ipv4 prefer IPv4
-6, --ipv6 prefer IPv6
--version print version number
(-h) --help show this help (-h works with no other options)
Use "rsync --daemon --help" to see the daemon-mode command-line options.
Please see the rsync(1) and rsyncd.conf(5) man pages for full documentation.
See http://rsync.samba.org/ for updates, bug reports, and answers
rsync error: syntax or usage error (code 1) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-51/rsync/options.c(1436) [client=2.6.9]
rsync version 2.6.9 protocol version 29
How should I be formatting my command so that I can do what I intended to do?
I can confirm that I'm encountering the same "bug" when trying to do a very simple case of rsync with the "--files-from" switch. Any attempt to use it results in a dump of the help message, although it doesn't complain about being an invalid option. Research on the net suggests that there is or was a bug in the "--relative" option , which is implied by "--files-from".
My work around has been to replace "--files-from=list-file" with "cat list-file" (using back quotes) in the command line.
I'm using LUbunto 18.04LTS and the rsync version is version 3.1.2 protocol version 31 from rsync.samba.org. That's what's packaged with Lubuntu (Ubuntu).
Who knows what's really going on?
[UPDATE] Well, after some further digging I discovered that the --files-from=list.txt must be used in conjuction with the source directory. I naively thought specifying the list would be relative to the current directory. It is not. It is relative to the source directory.
So the following should work:
% rsync -a -r -v --files-from=/someSrc/list.txt /someSrc [remote:]/someDest
and it does in simple cases. Test your complex case by testing a simple subset first.
I am using a NAS to backup my file server. The NAS exports /share/Backup via NFS, which is mounted on the fileserver as /mount/qnap. I want to keep track which files are rsynced but exclude the Backup-Dir, which contains many small files.Therefore I am running two instances of rsync, one with -v and another one without. The following command works as it should, after executing it the directory structure on /mount/qnap is identical to /mount/btrfs-raid.
rsync --delete -av --exclude Backup /mnt/btrfs-raid/ /mnt/qnap/
Rsyncing the Backup folder with the command
rsync --delete -av /mnt/btrfs-raid/Backup /mnt/qnap/Backup
produces the following directory structure on the NAS:
/mnt/qnap/Backup/Backup/..Subdirectories
To get the result I want I have to delete the last "Backup" from the target directory path:
rsync --delete -av /mnt/btrfs-raid/Backup /mnt/qnap/
Why does the second example not work like the first one?
Thanks
Stefan
Trailing slashes in paths are important for rsync. See the documentation.
rsync -avz foo:src/bar /data/tmp
This would recursively transfer all files from the directory src/bar on the machine foo into the /data/tmp/bar directory on the local machine. The files are transferred in "archive" mode, which ensures that symbolic links, devices, attributes, permissions, ownerships, etc. are preserved in the transfer. Additionally, compression will be used to reduce the size of data portions of the transfer.
rsync -avz foo:src/bar/ /data/tmp
A trailing slash on the source changes this behavior to avoid creating an additional directory level at the destination. You can think of a trailing / on a source as meaning "copy the contents of this directory" as opposed to "copy the directory by name", but in both cases the attributes of the containing directory are transferred to the containing directory on the destination. In other words, each of the following commands copies the files in the same way, including their setting of the attributes of /dest/foo:
I'm loging to a remote server which has dir structure as follows-
cd /exp/gc/data/tmp/
ABCD
GED
TROOP
These directories in turn may or may not have subdirectories.My requirement is to copy files from remote server to hadoop(final destination) one at a time keeping the directory structure same as source.
eg:
source:
cd /exp/gc/data/tmp/ABCD
now ABCD has 3 files and two subdirectories.So my target location(hadoop) should also have same directory structure i.e. ABCD must have 3 files and two subdir.
Using rsync iam able to replicate the directory structure but directories with spaces or special character in the name are not getting copied.
Code snippet:
result = $(ssh username#hostname "find /path/ -type f")
for file in $result
do
rsync -arsv username#hostname /path-source/ /target-path-tmp/
hadoop fs -copyFromLocal /target-path-tmp/ /hadoop -location/
rm -rf /target-path-tmp/*
I think what you are looking for is rsync, try it as follow:
rsync -avz foo:src/bar /data/tmp
This would recursively transfer all files from the directory src/bar on the machine foo into the /data/tmp/bar directory on the local machine. The files are transferred in archive permissions, ownerships, etc. are preserved in the transfer. Additionally, compression will be used to reduce the size of data portions of the transfer.
example