I'm trying to set up rsync to run backups from my Unraid shares to external drives mounted to the Unraid server.
I have different folders in different shares that I want backed up to different folders on different external drives.
I.e:
/share1/folder1/ should be synced to /externalDisk1/folder1/
/share1/folder2/ should be synced to /externalDisk2/folder1/
etc.
I have eight of these jobs, all together.
The idea is to run these as a cron job every night, and so I wrote them all down in a bash script.
Running the rsync lines one by one, or even as "job1; job2.." in the command line works great, however, calling them from "backup.sh" returns errors.
The script (with only one rsync line as an example):
#!/bin/sh
rsync -rtvh --progress /mnt/user/share1/folder1/ /mnt/disks/externalDrive1/Folder1/
returns
sending incremental file list
rsync: [Receiver] mkdir "/mnt/disks/externalDisk1/Folder1/\#015" failed: No such file or directory (2)
rsync error: error in file IO (code 11) at main.c(784) [Receiver=3.2.3]
rsync: [sender] write error: Broken pipe (32)
rsync error: error in file IO (code 11) at io.c(823) [sender=3.2.3]
Any suggestions on what's going on and how to fix this?
Running the script through the CA User Scripts plug-in makes it run perfectly. It's a work-around, if nothing else :)
I'm a beginner with rsync.
I have a file "filelist.txt" with some files with full path :
/tmp/folder1/file.txt
/tmp/folder2/file.txt
/tmp/folder2/file.txt
I want to copy this files from the server A to serveur B, and create directories if it's needed.
This file can evolve several time a day, so I don't want to handle dir creation manually on the other server before transfert files.
So I used :
cat filelist.txt | xargs -I {} rsync -r {} admin#riw-appcmd.i-wel.fr:{}
But I have for each line :
rsync: change_dir#3 "/tmp/folder1" failed: No such file or directory (2)
rsync error: errors selecting input/output files, dirs (code 3) at main.c(632) [receiver=3.0.3]
rsync: connection unexpectedly closed (8 bytes received so far) [sender]
What I'm doing wrong ?
You can use the --relative option. For example:
rsync --files-from filelist.txt -R -av / user#host:/
According to rsync manual:
-R, --relative
Use relative paths. This means that the full path names specified on the command line are sent to the server rather than just the last parts of the filenames. This is particularly useful when you want to send several different directories at the same time. For example, if you used this command:
rsync -av /foo/bar/baz.c remote:/tmp/
... this would create a file named baz.c in /tmp/ on the remote machine. If instead you used
rsync -avR /foo/bar/baz.c remote:/tmp/
then a file named /tmp/foo/bar/baz.c would be created on the remote machine, preserving its full path. These extra path elements are called "implied directories" (i.e. the "foo" and the "foo/bar" directories in the above example).
[...]
I'm trying to set up a workflow to copy my iCloud script projects to my user library. Sadly apple decided to add quite a few unfriendly characters in the file path.
rsync -aE –delete ~/Library/Mobile\ Documents/com\~apple\~ScriptEditor2/Documents/Script\ Libraries/ ~/Library/Script\ Libraries/
I keep getting. The following error:
rsync: link_stat "/Users/{{my_username}}/\#342\#200\#223delete" failed: No such file or directory (2)
rsync error: some files could not be transferred (code 23) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-54/rsync/main.c(996) [sender=2.6.9]
The file is truncated so I feel it's a parse error, but I can't get it to work. I've tried a few different ways of writing the path, but they all are saying the path does not exist.
TL;DR: Your delete option is wrong. It should be --delete with 2 ASCII "minus" characters.
I suspect you may have a character encoding problem. Rich text editors and word processors often combine -- to an "emdash" character. The error message shows it is trying to access a file with some non-ASCII characters followed by "delete". This shows that rsync is treating the delete part as a file path instead of a command flag.
Answers to this question do not seem to work for me to parallelize rsync from my local machine to an AWS instance.
I have done a dry run and obtained a filelist with filepaths for each file to be transferred, split this filelist into 59 smaller filelists residing in the split directory. I then created a new text file with the name of each filelist, split_list.txt. I'm trying to open up this filelist of filelists to pipe into 59 parallel rsync runs:
cat split_list.txt | parallel --will-cite -j 59 rsync --relative --human-readable --ignore-existing --files-from={} -rave "ssh -i key.pem" /Volumes/BRIENNAKH\ 4/xml/ [REMOTE-HOST]:~/arxiv/xml
I thought that by doing cat split_list.txt I would be passing to the second argument the path of each filelist in the split directory, and that the --files-from={} argument would be receiving each filelist.
It doesn't run and instead returns
Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others.
<http://rsync.samba.org/>
Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,
inplace, IPv6, 64-bit system inums, 64-bit internal inums
rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you
are welcome to redistribute it under certain conditions. See the GNU
General Public Licence for details.
rsync is a file transfer program capable of efficient remote update
via a fast differencing algorithm.
Usage: rsync [OPTION]... SRC [SRC]... DEST
or rsync [OPTION]... SRC [SRC]... [USER#]HOST:DEST
or rsync [OPTION]... SRC [SRC]... [USER#]HOST::DEST
or rsync [OPTION]... SRC [SRC]... rsync://[USER#]HOST[:PORT]/DEST
or rsync [OPTION]... [USER#]HOST:SRC [DEST]
or rsync [OPTION]... [USER#]HOST::SRC [DEST]
or rsync [OPTION]... rsync://[USER#]HOST[:PORT]/SRC [DEST]
The ':' usages connect via remote shell, while '::' & 'rsync://' usages connect
to an rsync daemon, and require SRC or DEST to start with a module name.
Options
-v, --verbose increase verbosity
-q, --quiet suppress non-error messages
--no-motd suppress daemon-mode MOTD (see manpage caveat)
-c, --checksum skip based on checksum, not mod-time & size
-a, --archive archive mode; same as -rlptgoD (no -H)
--no-OPTION turn off an implied OPTION (e.g. --no-D)
-r, --recursive recurse into directories
-R, --relative use relative path names
--no-implied-dirs don't send implied dirs with --relative
-b, --backup make backups (see --suffix & --backup-dir)
--backup-dir=DIR make backups into hierarchy based in DIR
--suffix=SUFFIX set backup suffix (default ~ w/o --backup-dir)
-u, --update skip files that are newer on the receiver
--inplace update destination files in-place (SEE MAN PAGE)
--append append data onto shorter files
-d, --dirs transfer directories without recursing
-l, --links copy symlinks as symlinks
-L, --copy-links transform symlink into referent file/dir
--copy-unsafe-links only "unsafe" symlinks are transformed
--safe-links ignore symlinks that point outside the source tree
-k, --copy-dirlinks transform symlink to a dir into referent dir
-K, --keep-dirlinks treat symlinked dir on receiver as dir
-H, --hard-links preserve hard links
-p, --perms preserve permissions
--executability preserve the file's executability
--chmod=CHMOD affect file and/or directory permissions
-o, --owner preserve owner (super-user only)
-g, --group preserve group
--devices preserve device files (super-user only)
--specials preserve special files
-D same as --devices --specials
-t, --times preserve times
-O, --omit-dir-times omit directories when preserving times
--super receiver attempts super-user activities
-S, --sparse handle sparse files efficiently
-n, --dry-run show what would have been transferred
-W, --whole-file copy files whole (without rsync algorithm)
-x, --one-file-system don't cross filesystem boundaries
-B, --block-size=SIZE force a fixed checksum block-size
-e, --rsh=COMMAND specify the remote shell to use
--rsync-path=PROGRAM specify the rsync to run on the remote machine
--existing skip creating new files on receiver
--ignore-existing skip updating files that already exist on receiver
--remove-source-files sender removes synchronized files (non-dirs)
--del an alias for --delete-during
--delete delete extraneous files from destination dirs
--delete-before receiver deletes before transfer (default)
--delete-during receiver deletes during transfer, not before
--delete-after receiver deletes after transfer, not before
--delete-excluded also delete excluded files from destination dirs
--ignore-errors delete even if there are I/O errors
--force force deletion of directories even if not empty
--max-delete=NUM don't delete more than NUM files
--max-size=SIZE don't transfer any file larger than SIZE
--min-size=SIZE don't transfer any file smaller than SIZE
--partial keep partially transferred files
--partial-dir=DIR put a partially transferred file into DIR
--delay-updates put all updated files into place at transfer's end
-m, --prune-empty-dirs prune empty directory chains from the file-list
--numeric-ids don't map uid/gid values by user/group name
--timeout=TIME set I/O timeout in seconds
-I, --ignore-times don't skip files that match in size and mod-time
--size-only skip files that match in size
--modify-window=NUM compare mod-times with reduced accuracy
-T, --temp-dir=DIR create temporary files in directory DIR
-y, --fuzzy find similar file for basis if no dest file
--compare-dest=DIR also compare destination files relative to DIR
--copy-dest=DIR ... and include copies of unchanged files
--link-dest=DIR hardlink to files in DIR when unchanged
-z, --compress compress file data during the transfer
--compress-level=NUM explicitly set compression level
-C, --cvs-exclude auto-ignore files the same way CVS does
-f, --filter=RULE add a file-filtering RULE
-F same as --filter='dir-merge /.rsync-filter'
repeated: --filter='- .rsync-filter'
--exclude=PATTERN exclude files matching PATTERN
--exclude-from=FILE read exclude patterns from FILE
--include=PATTERN don't exclude files matching PATTERN
--include-from=FILE read include patterns from FILE
--files-from=FILE read list of source-file names from FILE
-0, --from0 all *-from/filter files are delimited by 0s
--address=ADDRESS bind address for outgoing socket to daemon
--port=PORT specify double-colon alternate port number
--sockopts=OPTIONS specify custom TCP options
--blocking-io use blocking I/O for the remote shell
--stats give some file-transfer stats
-8, --8-bit-output leave high-bit chars unescaped in output
-h, --human-readable output numbers in a human-readable format
--progress show progress during transfer
-P same as --partial --progress
-i, --itemize-changes output a change-summary for all updates
--out-format=FORMAT output updates using the specified FORMAT
--log-file=FILE log what we're doing to the specified FILE
--log-file-format=FMT log updates using the specified FMT
--password-file=FILE read password from FILE
--list-only list the files instead of copying them
--bwlimit=KBPS limit I/O bandwidth; KBytes per second
--write-batch=FILE write a batched update to FILE
--only-write-batch=FILE like --write-batch but w/o updating destination
--read-batch=FILE read a batched update from FILE
--protocol=NUM force an older protocol version to be used
-E, --extended-attributes copy extended attributes
--cache disable fcntl(F_NOCACHE)
-4, --ipv4 prefer IPv4
-6, --ipv6 prefer IPv6
--version print version number
(-h) --help show this help (-h works with no other options)
Use "rsync --daemon --help" to see the daemon-mode command-line options.
Please see the rsync(1) and rsyncd.conf(5) man pages for full documentation.
See http://rsync.samba.org/ for updates, bug reports, and answers
rsync error: syntax or usage error (code 1) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-51/rsync/options.c(1436) [client=2.6.9]
rsync version 2.6.9 protocol version 29
How should I be formatting my command so that I can do what I intended to do?
I can confirm that I'm encountering the same "bug" when trying to do a very simple case of rsync with the "--files-from" switch. Any attempt to use it results in a dump of the help message, although it doesn't complain about being an invalid option. Research on the net suggests that there is or was a bug in the "--relative" option , which is implied by "--files-from".
My work around has been to replace "--files-from=list-file" with "cat list-file" (using back quotes) in the command line.
I'm using LUbunto 18.04LTS and the rsync version is version 3.1.2 protocol version 31 from rsync.samba.org. That's what's packaged with Lubuntu (Ubuntu).
Who knows what's really going on?
[UPDATE] Well, after some further digging I discovered that the --files-from=list.txt must be used in conjuction with the source directory. I naively thought specifying the list would be relative to the current directory. It is not. It is relative to the source directory.
So the following should work:
% rsync -a -r -v --files-from=/someSrc/list.txt /someSrc [remote:]/someDest
and it does in simple cases. Test your complex case by testing a simple subset first.
This question already has answers here:
When to wrap quotes around a shell variable?
(5 answers)
Closed 4 years ago.
I have ~18gb worth of photos/videos on my internal that I'm trying to move to my external via Rsync. I initially tried:
sudo rsync -avh --progress --delete /Non-system files/Photo.photoslibrary/
/#2/Text_photos/
But no dice, "No such file" error. I thought that maybe .photoslibrary had to be in a folder so I moved it to "photos" folder in Non-system files and subsequently tried:
sudo rsync -avh --progress --delete /Non-system files/photos/
Photo.photoslibrary/ /#2/Text_photos/
I'm still getting the same error of:
rsync: link_stat "/Non-system" failed: No such file or directory (2)
rsync: link_stat "/Users/jcb_carrillo/files/photos/Photos" failed: No such file
or directory (2)
So two things I've noticed but don't know why it's reacting the way it is.
1. It's not reading the space in my partition "Non-system files" it only reads "Non-system".
And 2. Once the first error of "No such file or directory" happens it for some reason moves to "/Users/jcb_carrillo/files/photos/Photos" which is in an entirely other partition that as you can see in my command, was never specified to go there.
I think there's something wrong with my syntax. Any idea?
Looks like you might need to escape the blank space for the directory name "Non-system file"
sudo rsync -avh --progress --delete /Non-system\ files/Photo.photoslibrary/
/#2/Text_photos/