I'm using rsync on Windows 7 (in particular, cwrsync). I'm using a simple command as such:
rsync -r --perms --delete /cygdrive/c/Users/Michael/Documents/Personal/ /cygdrive/c/Users/Michael/Documents/Personal_Backup/
The recursive copy works fine, except if I was to (right-click/Properties/Security tab) on any folder created by rsync on the destination; I get the following pop-up message:
The permissions on {folderName} are incorrectly ordered, which may cause
some entries to be ineffective.
I also tried the --acls option but get the following error:
recv_acl_access: value out of range: ff rsync error: error in rsync
protocol data stream (code 12) at acls.c(690) [Receiver=3.0. rsync:
connection unexpectedly closed (9 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at
io.c(610) [sender=3.0.8]
In any case, I just want to use rsync correctly so that viewing the Security permissions in Windows won't throw an error.
Michael,
This solution suggests that you should not be using --perms, but using --chmod=ugo=rwX instead.
Good luck!
Dotan
One word, Robocopy.
I had exactly the same isues with borked permissions while using cwRsync, tried numerous things but none seemed to work so I gave up eventually.
This is default Windows tool and has similar (for your purpose, the same) feature set.
I discovered it last night and ditched rsync completely. It's built for unix-like's so some sort of bummer is expected on Windows.
This got me started:
http://www.sevenforums.com/tutorials/187346-robocopy-create-backup-script.html
Here's the little backup script I made for myself to mirror my partitions to external drive.
Don't look back for rsync any more.
I use the now-deprecated cacls to add myself back in after the copy occurs.
rsync -avASPC sourceDir/* destDir
cacls destDir /t /e /r doej
cacls destDir /t /e /g doej:f
Where sourceDir is the source directory and destDir is the destination directory and doej is the username. It would probably be better to use icacls, but I haven't learned it yet.
I also tried robocopy, but I did not have the permissions I needed to make that work, it seems.
Flags used for rsync
-a, --archive archive mode; equals -rlptgoD (no -H,-A,-X)
-r, --recursive recurse into directories
-l, --links copy symlinks as symlinks
-p, --perms preserve permissions
-t, --times preserve modification times
-g, --group preserve group
-o, --owner preserve owner (super-user only)
-D same as --devices --specials
--devices preserve device files (super-user only)
--specials preserve special files
-v, --verbose increase verbosity
-S, --sparse handle sparse files efficiently
-A, --acls preserve ACLs (implies -p, which is also implied by -a)
-P same as --partial --progress
--progress show progress during transfer
--partial keep partially transferred files
-C, --cvs-exclude auto-ignore files in the same way CVS does
Flags used from CACLS
/T Changes ACLs of specified files in
/E Edit ACL instead of replacing it.
/R user Revoke specified user's access rights (only valid with /E).
/P user:perm Replace specified user's access rights.
Perm can be: ...
F Full control
Related
Answers to this question do not seem to work for me to parallelize rsync from my local machine to an AWS instance.
I have done a dry run and obtained a filelist with filepaths for each file to be transferred, split this filelist into 59 smaller filelists residing in the split directory. I then created a new text file with the name of each filelist, split_list.txt. I'm trying to open up this filelist of filelists to pipe into 59 parallel rsync runs:
cat split_list.txt | parallel --will-cite -j 59 rsync --relative --human-readable --ignore-existing --files-from={} -rave "ssh -i key.pem" /Volumes/BRIENNAKH\ 4/xml/ [REMOTE-HOST]:~/arxiv/xml
I thought that by doing cat split_list.txt I would be passing to the second argument the path of each filelist in the split directory, and that the --files-from={} argument would be receiving each filelist.
It doesn't run and instead returns
Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others.
<http://rsync.samba.org/>
Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,
inplace, IPv6, 64-bit system inums, 64-bit internal inums
rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you
are welcome to redistribute it under certain conditions. See the GNU
General Public Licence for details.
rsync is a file transfer program capable of efficient remote update
via a fast differencing algorithm.
Usage: rsync [OPTION]... SRC [SRC]... DEST
or rsync [OPTION]... SRC [SRC]... [USER#]HOST:DEST
or rsync [OPTION]... SRC [SRC]... [USER#]HOST::DEST
or rsync [OPTION]... SRC [SRC]... rsync://[USER#]HOST[:PORT]/DEST
or rsync [OPTION]... [USER#]HOST:SRC [DEST]
or rsync [OPTION]... [USER#]HOST::SRC [DEST]
or rsync [OPTION]... rsync://[USER#]HOST[:PORT]/SRC [DEST]
The ':' usages connect via remote shell, while '::' & 'rsync://' usages connect
to an rsync daemon, and require SRC or DEST to start with a module name.
Options
-v, --verbose increase verbosity
-q, --quiet suppress non-error messages
--no-motd suppress daemon-mode MOTD (see manpage caveat)
-c, --checksum skip based on checksum, not mod-time & size
-a, --archive archive mode; same as -rlptgoD (no -H)
--no-OPTION turn off an implied OPTION (e.g. --no-D)
-r, --recursive recurse into directories
-R, --relative use relative path names
--no-implied-dirs don't send implied dirs with --relative
-b, --backup make backups (see --suffix & --backup-dir)
--backup-dir=DIR make backups into hierarchy based in DIR
--suffix=SUFFIX set backup suffix (default ~ w/o --backup-dir)
-u, --update skip files that are newer on the receiver
--inplace update destination files in-place (SEE MAN PAGE)
--append append data onto shorter files
-d, --dirs transfer directories without recursing
-l, --links copy symlinks as symlinks
-L, --copy-links transform symlink into referent file/dir
--copy-unsafe-links only "unsafe" symlinks are transformed
--safe-links ignore symlinks that point outside the source tree
-k, --copy-dirlinks transform symlink to a dir into referent dir
-K, --keep-dirlinks treat symlinked dir on receiver as dir
-H, --hard-links preserve hard links
-p, --perms preserve permissions
--executability preserve the file's executability
--chmod=CHMOD affect file and/or directory permissions
-o, --owner preserve owner (super-user only)
-g, --group preserve group
--devices preserve device files (super-user only)
--specials preserve special files
-D same as --devices --specials
-t, --times preserve times
-O, --omit-dir-times omit directories when preserving times
--super receiver attempts super-user activities
-S, --sparse handle sparse files efficiently
-n, --dry-run show what would have been transferred
-W, --whole-file copy files whole (without rsync algorithm)
-x, --one-file-system don't cross filesystem boundaries
-B, --block-size=SIZE force a fixed checksum block-size
-e, --rsh=COMMAND specify the remote shell to use
--rsync-path=PROGRAM specify the rsync to run on the remote machine
--existing skip creating new files on receiver
--ignore-existing skip updating files that already exist on receiver
--remove-source-files sender removes synchronized files (non-dirs)
--del an alias for --delete-during
--delete delete extraneous files from destination dirs
--delete-before receiver deletes before transfer (default)
--delete-during receiver deletes during transfer, not before
--delete-after receiver deletes after transfer, not before
--delete-excluded also delete excluded files from destination dirs
--ignore-errors delete even if there are I/O errors
--force force deletion of directories even if not empty
--max-delete=NUM don't delete more than NUM files
--max-size=SIZE don't transfer any file larger than SIZE
--min-size=SIZE don't transfer any file smaller than SIZE
--partial keep partially transferred files
--partial-dir=DIR put a partially transferred file into DIR
--delay-updates put all updated files into place at transfer's end
-m, --prune-empty-dirs prune empty directory chains from the file-list
--numeric-ids don't map uid/gid values by user/group name
--timeout=TIME set I/O timeout in seconds
-I, --ignore-times don't skip files that match in size and mod-time
--size-only skip files that match in size
--modify-window=NUM compare mod-times with reduced accuracy
-T, --temp-dir=DIR create temporary files in directory DIR
-y, --fuzzy find similar file for basis if no dest file
--compare-dest=DIR also compare destination files relative to DIR
--copy-dest=DIR ... and include copies of unchanged files
--link-dest=DIR hardlink to files in DIR when unchanged
-z, --compress compress file data during the transfer
--compress-level=NUM explicitly set compression level
-C, --cvs-exclude auto-ignore files the same way CVS does
-f, --filter=RULE add a file-filtering RULE
-F same as --filter='dir-merge /.rsync-filter'
repeated: --filter='- .rsync-filter'
--exclude=PATTERN exclude files matching PATTERN
--exclude-from=FILE read exclude patterns from FILE
--include=PATTERN don't exclude files matching PATTERN
--include-from=FILE read include patterns from FILE
--files-from=FILE read list of source-file names from FILE
-0, --from0 all *-from/filter files are delimited by 0s
--address=ADDRESS bind address for outgoing socket to daemon
--port=PORT specify double-colon alternate port number
--sockopts=OPTIONS specify custom TCP options
--blocking-io use blocking I/O for the remote shell
--stats give some file-transfer stats
-8, --8-bit-output leave high-bit chars unescaped in output
-h, --human-readable output numbers in a human-readable format
--progress show progress during transfer
-P same as --partial --progress
-i, --itemize-changes output a change-summary for all updates
--out-format=FORMAT output updates using the specified FORMAT
--log-file=FILE log what we're doing to the specified FILE
--log-file-format=FMT log updates using the specified FMT
--password-file=FILE read password from FILE
--list-only list the files instead of copying them
--bwlimit=KBPS limit I/O bandwidth; KBytes per second
--write-batch=FILE write a batched update to FILE
--only-write-batch=FILE like --write-batch but w/o updating destination
--read-batch=FILE read a batched update from FILE
--protocol=NUM force an older protocol version to be used
-E, --extended-attributes copy extended attributes
--cache disable fcntl(F_NOCACHE)
-4, --ipv4 prefer IPv4
-6, --ipv6 prefer IPv6
--version print version number
(-h) --help show this help (-h works with no other options)
Use "rsync --daemon --help" to see the daemon-mode command-line options.
Please see the rsync(1) and rsyncd.conf(5) man pages for full documentation.
See http://rsync.samba.org/ for updates, bug reports, and answers
rsync error: syntax or usage error (code 1) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-51/rsync/options.c(1436) [client=2.6.9]
rsync version 2.6.9 protocol version 29
How should I be formatting my command so that I can do what I intended to do?
I can confirm that I'm encountering the same "bug" when trying to do a very simple case of rsync with the "--files-from" switch. Any attempt to use it results in a dump of the help message, although it doesn't complain about being an invalid option. Research on the net suggests that there is or was a bug in the "--relative" option , which is implied by "--files-from".
My work around has been to replace "--files-from=list-file" with "cat list-file" (using back quotes) in the command line.
I'm using LUbunto 18.04LTS and the rsync version is version 3.1.2 protocol version 31 from rsync.samba.org. That's what's packaged with Lubuntu (Ubuntu).
Who knows what's really going on?
[UPDATE] Well, after some further digging I discovered that the --files-from=list.txt must be used in conjuction with the source directory. I naively thought specifying the list would be relative to the current directory. It is not. It is relative to the source directory.
So the following should work:
% rsync -a -r -v --files-from=/someSrc/list.txt /someSrc [remote:]/someDest
and it does in simple cases. Test your complex case by testing a simple subset first.
I tried to back up data from my macbook to an external hard drive - formatted with exFat (bacause of the Windows and Linux/Mac compatibility).
With Automator I will create a little Program, to backup my data easily. It works fine on the local drive and from local drive to an SD-Card. But it do not work from local drive to an external hard drive. What's wrong?
SOURCE=/Users/username/Pictures/test
TARGET=/Volumes/Backup/
LOG=~/Documents/AutomatorLogs/BackupSync.log
rsync -aze "ssh" --delete --exclude=".*" "$SOURCE" "$TARGET" > "$LOG"
I got this Error:
rsync: recv_generator: mkdir "/Volumes/Backup/test" failed: Permission
denied (13)
I know this is older but I just ran into this and I wanted to make sure this info was included. I know the OP is a little different, but I'm using a macbook and ran into the error I describe so I don't know how even with changing the disk name it worked.
rsync can't use -a when the target is an exfat drive. It will make the directories and seem to be backing up but no files are actually created. You need to use:
rsync -rltDv [SRC] [DESTINATION]
where:
-v, --verbose increase verbosity
-r, --recursive recurse into directories
-l, --links copy symlinks as symlinks
--devices preserve device files (super-user only)
--specials preserve special files
-D same as --devices --specials
-t, --times preserve times
The reason is because rsync doesn't handle permissions on exfat. You will see an rsync error (in syslog or if you control-c out):
chgrp [file] failed: Function not implemented (38)
It looks like the user that you're running the command as doesn't have permission to make a new directory in the /Volumes/Backup/ directory.
To solve this, you will probably need to change the permissions on that directory so that your script will be able to write to it and create the new directory it uses to make the backup.
Here are some links about permissions:
http://linuxcommand.org/lts0070.php
http://www.perlfect.com/articles/chmod.shtml
I think, I've got it:
It is related to the name of the external hard disk.
With the name "Backup" for my external hard drive, it does not work.
If I changed the name to anything else, it works.
(I tested some other exFat formatted external hard drives with other names and it worked. So I changed the name of this external drive to anything else and now it works. Crazy...)
I am trying to sync a local path to another drive using Git Bash in Windows 7. I downloaded cwRsync and copied the files over to my Git bash bin directory. I can run rsync but the following command gives me problems
rsync -av /c/00-dev/05-batch /f/RSYNC/
The source and destination cannot both be remote.
rsync error: syntax or usage error (code 1) at main.c(1135) [receiver=3.0.6]
Both the paths are correct but it won't sync.
Workaround: cd to /c and rsync using relative paths.
So this works:
cd /c
touch temp/test.txt
rsync temp/test.txt temp/test2.txt
while this doesn't, even if you cd to /c:
$ rsync /c/temp/test.txt /c/temp/test2.txt
The source and destination cannot both be remote.
Tested with the Git Bash that came with git 2.9.0.windows.1.
This works using cwRsync from Git Bash.
export MSYS_NO_PATHCONV=1
rsync -av /cygdrive/c/00-dev/05-batch /cygdrive/f/RSYNC/
cwRsync.exe needs /cygdrive/<drive-letter> for absolute pathing, so the simple rsync -av /cygdrive/c/00-dev/05-batch /cygdrive/f/RSYNC/ works from the cmd.
However, Git Bash converts paths like .../c/... to weird things like ...C:\.... The line: export MSYS_NO_PATHCONV=1 prevents that conversion. (See more: How to stop MinGW and MSYS from mangling path names given at the command line).
Use unset MSYS_NO_PATHCONV in the script to restore standard Git Bash behavior.
In Windows I tried cwRsync to sync data between folders located on Z:(network disk) and D:(usb stick) excluding a subfolder 'Data':
rsync -r -v --size-only --dry-run --exclude=Data --chmod=ugo=rwX /cygdrive/z/"Scan folder"/"Sub folder" /cygdrive/d/"Scan folder"/"Sub folder"
-r means recursively, -v verbose output, --dry-run is just for testing (data doesn't transfer), --exclude excludes subfolder or file(s), --chmod=ugo=rwX sets proper permissions (read more about permissions here with comments)
For cyrillic names on Windows use 'chcp 65001'.
So I'm currently trying to write a few small scripts that allow me to manage my iTunes library of which I have clones on multiple OS X machines.
The basic idea is that I have a NAS holding a copy of the library that is used as an intermediate "master copy" since the machines holding the actually used copies aren't available all the time. If I want to update my old copy on machine B with the newer version from machine A, I'd then update the NAS copy based on machine A's current state, then update machine B from the updated NAS copy possibly at a later time.
The script files are located on the NAS within the same folder that also houses the iTunes directory. Since I'm mounting the NAS as a volume via AFP, I simply open a Finder window with the directory containing the scripts and drag'n'drop the script I want to use to a Terminal window for easy execution.
This is my attempt at the "update NAS from local copy" script:
rsync -avz --compress-level 1 --exclude 'Mobile Applications/*.ipa' --delete --delay-updates -n "$(echo $HOME | sed 's/ /\\ /g')/Music/iTunes" "$(dirname $0 | sed 's/ /\\ /g')"
(-n option of course only for testing the script)
Since there will be spaces in the paths I supply rsync with, I already figured out that I'd need to escape those somehow. I also know that the standard way to do that on OS X is to prepend all the spaces with a backslash, at least when manually typing paths in Terminal. But the code above still won't work – rsync complains that it cannot change to the directory I supplied, although the path it spits out in the error message seems to be perfectly fine and can be cd'd to, if you remove the double quotes around it first:
[...]
building file list ... rsync: change_dir "/Volumes/Macintosh\ HD/Users/Julian/Music" failed: No such file or directory (2)
done
[...]
If I remove the surrounding double quotes in the script itself, rsync seems to not honor the escaping backslashes at all and still treat the space following the backslash as a path separator:
[...]
building file list ... rsync: link_stat "/Volumes/Macintosh\" failed: No such file or directory (2)
rsync: change_dir "/Volumes/Macintosh HD/Users/Julian/HD/Users/Julian/Music" failed: No such file or directory (2)
done
[...]
And no, I can't work around the issue by shortening /Volumes/Macintosh\ HD/Users/Julian/Music to /Users/Julian/Music since this machine has multiple HDDs and / is not the same disk/partition as /Volumes/Macintosh\ HD. So I need to find a solution for this specific problem.
I'm seriously lost now.
Can anyone please explain to me what I need to change in order to have rsync recognize the paths correctly?
After messing around quite a bit more and finding this question, I managed to develop a working solution:
localpath=$HOME/Music/iTunes
remotepath=$(dirname $0)/
rsync -avz \
--compress-level 1 \
--exclude 'Mobile Applications/*.ipa' \
--delete \
--delay-updates \
-n \
"$localpath" \
"$remotepath"
I'm trying to push changes to my server through ssh on windows (cygwin) using rsync.
The command I am using is:
rsync -rvz -e ssh /cygdrive/c/myfolder/ rsyncuser#192.168.1.110:/srv/www/prj112/myfolder/
/srv/www/prj112/myfolder/ is owned by rsyncuser. My problem is that eventhough with rsync the sub directories are create as they publish, each directory is assigned default permission of d--------- so rsync fails to copy any files inside it.
How do I fix this?
The option to ignore NTFS permissions has changed in Cygwin version 1.7. This might be what's causing the problem.
Try adding the 'noacl' flag to your Cygwin mounts in C:\cygwin\etc\fstab, for example:
none /cygdrive cygdrive user,noacl,posix=0 0 0
You can pass custom permissions via rsync using the 'chmod' option:
rsync -rvz --chmod=ugo=rwX -e ssh source destination
Your problem stems from the fact that the Unix permissions on that directory really are 0. All of the access information is stored in separate ACLs, which rsync does not copy. Thus, it sets the permissions on the remote copy to 0, and, obviously, is unable to write to that directory afterwards.
You can run
chmod -R 775
on that directory, which should fix your rsync problem.
After a look at the manpage I can tell that the chmod param is available in rsync since version ~2.6.8. But you have to use --chmod=ugo=rwX in combination with rsync -av
You should also try this command:
rsync -av <SOURCE_DIR> rsyncuser#192.168.1.110:/srv/www/prj112/myfolder
It would work on Linux at least. And note that rsync does not need to mention ssh--at least on Linux.
But if all fails and just to give an option you may take a look at this ready packed-up tool cwRsync
if you deploy a site from windows (for ex. octopress use rsync) it's possible set permission to 775 adding multiple chmod command:
rsync -avz --chmod=ug=rwx --chmod=o=rx -e ssh
To rsync from Windows to Unix/Linux you should provide a command like
SET BACKUP_SERVER=my.backup.server
SET SSH_USER=theUnixUserName
SET STORAGEPATH=/home/%SSH_USER%/Backup/
SET STORAGEURI=%BACKUP_SERVER%:%STORAGEPATH%
SET SSH_ID=/cygdrive/c/Users/theWindowsUserName/Documents/keyfiles/id_dsa
SET EXCLUDEFILE=backup_excludes.txt
SET BACKUPLOGFILE=/cygdrive/c/Users/theWindowsUserName/Backuplogs/backup-%DATE%-%TIME::=-%.log
The ssh command then is
SET BACKUP=rsync -azvu --chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r --rsh="ssh -l %SSH_USER% -i '%SSH_ID%'" --exclude-from=%EXCLUDEFILE% --delete --delete-excluded --log-file="%BACKUPLOGFILE%"
with backup_excludes.txt containing lines of ignored elements like
.git
.svn
.o
\Debug
\Release
Then you would use this in a script with
%BACKUP% /cygdrive/c/mySensibleData %STORAGEURI%
%BACKUP% /cygdrive/c/myOtherSensibleData %STORAGEURI%
%BACKUP% /cygdrive/c/myOtherSensibleData2 %STORAGEURI%
and so on. This will backup your directories mySensibleData, myOtherSensibleData and myOtherSensibleData2 with the permissions 755 for directories and 644 for files. You also get backup logs in your %BACKUPLOGFILE% for each backup.
Cygwin rsync will report permission denied when some process has the target file open. Download and run Process Explorer and find out if anything else is locking the file or simply try renaming the file and see if you get the Windows error about some other process having the file open.
Also, you can try to create a (global) environment variable CYGWIN and set its value to nontsec