rsync include rules refuse to work - include

I have researched this pretty thoroughly. Everything that supposedly "works" doesn't for my case, so I thought I would ask here.
I have a directory tree that looks like:
root/
CK1E/
3DConformers/
OtherStuff/
PDE4B/
3DConformers/
OtherStuff/
...
I am trying to use an rsync command to retrieve the "3DConformers" subdirectories from each of these projects.
I am issuing a command of the form:
rsync -avz user#host:/path/to/root/ . --exclude "*" --include-from rules -v -v
I couldn't get general rules to work so I decided to use specific ones, focusing on the "CK1E" subdirectory. I read conflicting reports as to whether leading or trailing slashes were desired/required by rsync, so I took the shotgun approach. My current rules file looks like this:
/CK1E
/CK1E/
/CK1E/3DConformers
/CK1E/3DConformers/
/CK1E/3DConformers/*
CK1E
CK1E/
CK1E/3DConformers
CK1E/3DConformers/
CK1E/3DConformers/*
I get the following output from rsync. Note that the "CK1E" directory is still getting ignored!
receiving incremental file list
[sender] add_rule(- *)
[sender] add_rule(+ /CK1E)
[sender] add_rule(+ /CK1E/)
[sender] add_rule(+ /CK1E/3DConformers)
[sender] add_rule(+ /CK1E/3DConformers/)
[sender] add_rule(+ /CK1E/3DConformers/*)
[sender] add_rule(+ CK1E)
[sender] add_rule(+ CK1E/)
[sender] add_rule(+ CK1E/3DConformers)
[sender] add_rule(+ CK1E/3DConformers/)
[sender] add_rule(+ CK1E/3DConformers/*)
server_sender starting pid=2451269
[sender] make_file(.,*,0)
recv_file_name(.)
...
[sender] hiding directory CK1E because of pattern *
...
received 1 names
recv_file_list done
send_file_list done
get_local_name count=1 .
generator starting pid=27883
delta-transmission enabled
recv_generator(.,0)
recv_generator(.,1)
generate_files phase=1
recv_files(1) starting
send_files starting
send_files(0, /path/to/root/.)
recv_files(.)
send_files phase=1
recv_files phase=1
generate_files phase=2
send_files phase=2
recv_files phase=2
send files finished
total: matches=0 hash_hits=0 false_alarms=0 data=0
recv_files finished
generate_files phase=3
generate_files finished
sent 212 bytes received 1578 bytes 275.38 bytes/sec
total size is 0 speedup is 0.00
_exit_cleanup(code=0, file=main.c, line=1505): about to call exit(0)
Help?!

The ordering of include/exclude rules is important: the first include/exclude rule that matches the file is the one that matters.
Your first rule is --exclude="*", so you're excluding everything. You need to reorder your rules so that your include rule matches before your catch-all exclude rule.
This command should work:
rsync -avz user#host:/path/to/root/ . \
--include "/CK1E/" \
--include "/CK1E/3DConformers/" \
--include "/CK1E/3DConformers/*" \
--exclude "*"
You need to add the intermediate directories, as well as the files, because, if you don't mention them explicitly, the catch-all exclude rule will match them, and if the direct is excluded then so are the contents.
(The \ are not part of the command; they just indicate line continuation; your shell will do the right thing when you cut-and-paste.)

Related

Using Rsync with Makefile

I am trying to use a makefile to sync some of my directories that are on my pc with my thumb drive. For this purpose, I am using multiple commands in a single target.
The makefile looks somewhat like this
pend:
rsync -avhzPu /foo/ /bar/
rsync -avhzPu /bob/ /alice/
But, every time I am doing make -f .sync pend only the first command runs and gives some output along with some error
sent 45.78K bytes received 3.32K bytes 98.20K bytes/sec
total size is 4.42G speedup is 89,981.37
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1196) [sender=3.1.2]
.sync:14: recipe for target 'pend' failed
make: *** [pend] Error 23
But when I am running the commands separately it works. I am not sure what wrong I am doing here.
From the output...
sent 45.78K bytes received 3.32K bytes 98.20K bytes/sec
total size is 4.42G speedup is 89,981.37
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1196) [sender=3.1.2]
.sync:14: recipe for target 'pend' failed
make: *** [pend] Error 23
...the first rsync is failing. If it ends with a non-zero exit code then make will simply bail at that point. If you want make to continue with further commands in the rule then you can explicitly request that it should ignore non-zero exit codes by prefixing a command with a -...
pend:
-rsync -avhzPu /foo/ /bar/
-rsync -avhzPu /bob/ /alice/
This assumes, of course, that simply continuing with the commands is the desired behaviour. Otherwise you need to think more carefully about how to proceed.

Rsync calculate file count before transfer?

When transferring lots of small files in many folders with rsync, it tends to stay around ir-chk=1000/xxxxx, where xxxxx keeps counting up as it discovers new files, and the amount to check stays around 1000 until its on its last few folders.
How can I have it check for the entire file count before copying?
The command I was using to copy was:
rsync -av --progress source dest
rsync -av --progress --dry-run --stats source dest
Option --dry-run doesn't transfer any files but will show how many bytes will be transferred.
Option --stats shows a summary.
a sample output:
...
tests/
tests/__init__.py
tests/test_config.py
Number of files: 5,033 (reg: 2,798, dir: 2,086, link: 149)
Number of created files: 5,032 (reg: 2,798, dir: 2,085, link: 149)
Number of deleted files: 0
Number of regular files transferred: 2,798
Total file size: 26,035,530 bytes
Total transferred file size: 26,032,322 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.004 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 158,821
Total bytes received: 17,284
sent 158,821 bytes received 17,284 bytes 117,403.33 bytes/sec
total size is 26,035,530 speedup is 147.84 (DRY RUN)
Get the number of files will be transferred
rsync -av --progress --dry-run --stats source dest |
fgrep 'Number of files' |
cut -d' ' -f4 |
tr -d ,
Since version 3.0.0, you can use this option to explicitly turn off incremental recursion:
--no-i-r
Details from the rsync man page:
Beginning with rsync 3.0.0, the recursive algorithm used is now an
incremental scan that uses much less memory than before and begins the transfer after the scanning of the first few directories have been completed. This incremental scan only affects our recursion algorithm, and does not change a non-recursive transfer. It is also only possible when both ends of the transfer are at least version 3.0.0.
Some options require rsync to know the full file list, so these options
disable the incremental recursion mode. These include: --delete-before, --delete-after, --prune-empty-dirs, and --delay-updates. Because of this, the default delete mode when you specify --delete is now --delete-during when both ends of the connection are at least 3.0.0 (use --del or --delete-during to request this improved deletion mode explicitly). See also the --delete-delay option that is a better choice than using --delete-after.
Incremental recursion can be disabled using the --no-inc-recursive
option or its shorter --no-i-r alias.

Rsync ignores folders ending in slashes

I have this following content in a file named "/rsync/include.txt"
+ /home/**
+ /opt**
- *
I then call the rsync command as follows:
rsync -avr --include-from="/rsync/include.txt" . ../backup
This produces the following output:
sending incremental file list
created directory ../archive
./
opt/
opt/some-file
opt/include-me/
opt/include-me/me-too
opt/include-me/and-me/
sent 299 bytes received 106 bytes 810.00 bytes/sec
total size is 0 speedup is 0.00
The /home directory exists, and contains files.
Why does the + /home/** pattern not work? I do not want to use the + /home** pattern, as that could match other folder names, e.g., /homeopathy.
Can anybody help me understand why this command doesn't work, and point me in the direction of the working command?
EDIT: While I'd still like an answer to this question, I sincerely suggest using rdiff-backup as it uses similar filtering files and patterns, but is substantially easier to use. I've spent a good deal of time today on this issue with rsync, which was resolved in a few minutes using rdiff-backup.
General debugging info
The easiest way to see if your filter rules do what you want, is to use -vv (increase verbosity twice) in combination with -n (dry-run).
With double verbosity you will see which pattern caused which file to be ignored.
You may grep the output to only see the relevant parts.
Example
% rsync -avv --include 'opt/1' --exclude 'opt/*' -n ./ /tmp \
| grep '\[sender\]'
[sender] showing directory opt/1 because of pattern opt/1
[sender] hiding directory opt/2 because of pattern opt/*
[sender] hiding directory opt/3 because of pattern opt/*
Specific answer
Your example fails because + /home/** is excluded by - *.
man rsync states:
Note that, when using the --recursive (-r) option (which is implied by -a), every subdir component of every path is visited left to right, with each directory having a chance for exclusion before its content.
So the pattern /home/** will be evaluated after /home is traversed, but this will never happen, because - * excludes /home.
To include /home/ you just have to insert it before /home/**, so your exclude file becomes:
+ /home/
+ /home/**
+ /opt/
+ /opt/**
- *

Rsync include or exclude directories using text file

I'm using rsync to backup some data from a remote host.
this is how I'm using the rsync cmd:
rsync --dry-run -avhi -e ssh --include-from=/home/rsync_list/test.txt root#10.10.4.61:/ /mnt/BACKUP/my_BACKUP/
this is the file /home/rsync_list/test.txt
+ /usr/acs/conf/**
+ /usr/acs/bin/**
+ /raid0/opmdps/TEMP_folder/**
- *
I want to copy only the listed folders excluding the remaining files.
I always get
receiving file list ... done
sent 103 bytes received 48 bytes 302.00 bytes/sec
total size is 0 speedup is 0.00 (DRY RUN)
Could you tell me what I'm doing wrong? How should I write the rsync command if I would like to sync, for example, only /raid0/opmdps/TEMP_folder/ without its subfolders?
I wonder if you really only tried with the command you posted?
Not only you are using "--dry-run" option, even the output indicates this:
total size is 0 speedup is 0.00 (DRY RUN)
Please consult the manpage:
-n, --dry-run perform a trial run with no changes made
https://linux.die.net/man/1/rsync
May I suggest you give it a run without --dry-run?

rsync script expands variable incorrectly

I have a script that takes in a unique location number from a file. These are formatted like this 7325-05, 5269-09 and 7479-14, for example. The first four numbers are what the folder is called and the second number is the first two characters of the filename unique within each folder.
So I wrote this script to use locate and find to get the full path of the folder and then use a wildcard to download the specific file using rsync. Here's the script that I have right now:
#!/bin/bash
#IFS='
#'
oIFS=$IFS
IFS=$'\n'
while read line;
do
name=$line;
folder=${line:0:4}
track=${line: -2}
folderlocation="$(locate -r '/'$folder'$')"
filelocation="$(find "$folderlocation" -type f -name "$track*")"
rsync -vazhn --progress "$filelocation" /cygdrive/c/
# mkdir /cygdrive/c/test/"$folder"
# cp -rvi "$filelocation" /cygdrive/c/test/"$folder"
echo "";
done < $1
The code using cp that is commented out works just fine. I would just really prefer to use rsync, mainly due to better feedback and more accurate progress reporting, as far as I can tell.
Using the code as pasted above (with rsync) throws this error:
./filelocator classic-locations.txt
sending incremental file list
rsync: change_dir "/home/emil//\\sandrew-nas/SMMUSIC/MMIMUSIC/7001-8000/7201-7300/7252/Unknown Album (29-12-2012 09-52-02)" failed: No such file or directory (2)
sent 20 bytes received 12 bytes 64.00 bytes/sec
total size is 0 speedup is 0.00 (DRY RUN)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1]
sending incremental file list
rsync: change_dir "/home/emil//\\sandrew-nas/SMMUSIC/MMIMUSIC/7001-8000/7201-7300/7252/Unknown Album (29-12-2012 09-52-02)" failed: No such file or directory (2)
sent 20 bytes received 12 bytes 64.00 bytes/sec
total size is 0 speedup is 0.00 (DRY RUN)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1]
As you can see, my home folder (where I issue the command) is suddenly included in the script, leading me to believe that a variable or wildcard is being expanded in the local shell, but no amount of escape characters seem to accomplish what I want with rsync.

Resources