Good morning,
I want to transfer specific files in a directory to a remote machine while keeping the architecture of the subdirectories. Moreover, I only want to transfer files that have a peculiar extension (e.g. ".txt").
I have tried the following:
rsync -aP --include *.txt ./sourceDirectory user#hostIP:destDirectory
but it copies to the destination (destDirectory) all the files, and not only those which match the .txt pattern.
Could anybody help me with such riddle?
P.S.: obviously, the directory sourceDirectory contains subdirectories where are located my .txt files.
I used to have the same problem when rsync videos to home NAS. If you read the manual of rsync, the --include flag really means "Do not exclude". It actually has to work together with a --exclude flag.
The follow command will do your job:
rsync -aP --include="*/" --include="*.txt" --exclude="*" sourceDirectory destDirectory
The command literally means exclude everything, except for subfolders and txt file.
Related
I'm not allowed to use rsync on the cluster I'm working on so I need to use cp. I want to copy a large directory including all files and subfolders etc. but without any folders that have the name "outdir".
I tried cp -r -v ./!(outdir) ../target-directory/
but it still copies all folders and contents in deeper directories with the name outdir. It only included the outdir folders in the highest directory.
I also tried cp -r ./*/!(outdir) ../target-directory/ but that one copied all files into the folder without keeping any hirarchy or folders etc.
I also tried certain find commands but it didn't work, but maybe I was just doing something stupid. I'm a beginner with bash so if you could explain your answer and what the flags etc. do that would really be helpfull, I've been trying forever now, on what I think shouldn't be that hard to do.
Instead of cp, you can use tar with option --exclude to control what you want copied or not.
The full command is:
tar --exclude="outdir" -cvpf - . | (cd TARGET_DIRECTORY; tar -xpf -)
So any path that contains the "outdir" pattern will be excluded.
Without the --exclude option, it will copy the entire structure of your current directory under TARGET_DIRECTORY.
You can replace the . in the first tar by your desired source directory.
I have folders & subfolders organized by date in the name containing files .jpgs and .arw. I want to keep the same folder structure but move the ARW files to a copy of the folder structure AND delete them out of the source directory. Also if the folder does not contain any .ARW files it would not be copied.
I was working on something like this that I could run in the terminal screen:
rsync -av --exclude=.*/ --include=‘*/‘ --include=‘*.arw’ --exclude=‘*’ /Users/adam/test1/ /Users/Adam/test2/
But in my testing it's copying both the .jpg and .arw file to test 2. My assumption would be that I would add --remove-source-files once I got the code working with only moving the ARW file.
Thanks for your help!
The right command is:
rsync -avm --remove-source-files --include='*.arw' --include='*/' --exclude='*' /Users/adam/test1 /Users/adam/test2
The option 'm' prevents the copy of the empty directories.
You can run rsync with the '--dry-run' option to see which files would be copied without actually execute it:
rsync -avm --dry-run --remove-source-files --include='*.arw' --include='*/' --exclude='*' /Users/adam/test1 /Users/adam/test2
It appears rsync has an issue with .ARW file type. Changed requirement to move JPEG files instead to the new directory.
rsync -am --remove-source-files --include=‘.jpeg’ --include='.jpg' --include='/' --exclude='' /Users/adam/test\ 1/ /Users/Adam/test2/
Thanks!
I am using a NAS to backup my file server. The NAS exports /share/Backup via NFS, which is mounted on the fileserver as /mount/qnap. I want to keep track which files are rsynced but exclude the Backup-Dir, which contains many small files.Therefore I am running two instances of rsync, one with -v and another one without. The following command works as it should, after executing it the directory structure on /mount/qnap is identical to /mount/btrfs-raid.
rsync --delete -av --exclude Backup /mnt/btrfs-raid/ /mnt/qnap/
Rsyncing the Backup folder with the command
rsync --delete -av /mnt/btrfs-raid/Backup /mnt/qnap/Backup
produces the following directory structure on the NAS:
/mnt/qnap/Backup/Backup/..Subdirectories
To get the result I want I have to delete the last "Backup" from the target directory path:
rsync --delete -av /mnt/btrfs-raid/Backup /mnt/qnap/
Why does the second example not work like the first one?
Thanks
Stefan
Trailing slashes in paths are important for rsync. See the documentation.
rsync -avz foo:src/bar /data/tmp
This would recursively transfer all files from the directory src/bar on the machine foo into the /data/tmp/bar directory on the local machine. The files are transferred in "archive" mode, which ensures that symbolic links, devices, attributes, permissions, ownerships, etc. are preserved in the transfer. Additionally, compression will be used to reduce the size of data portions of the transfer.
rsync -avz foo:src/bar/ /data/tmp
A trailing slash on the source changes this behavior to avoid creating an additional directory level at the destination. You can think of a trailing / on a source as meaning "copy the contents of this directory" as opposed to "copy the directory by name", but in both cases the attributes of the containing directory are transferred to the containing directory on the destination. In other words, each of the following commands copies the files in the same way, including their setting of the attributes of /dest/foo:
I take delivery of files from multiple places as part of a publishing aggregation service. I need a way to move files that have been delivered to me from one location to another without losing the directory listings for sorting purposes.
Example:
Filepath of delivery: Server/Vendor/To_Company/Customer_Name/**
Filepath of processing: ~/Desktop/MM-DD-YYYY/Returned_Files/Customer_Name/**
I know I can move all of the directories by doing something such as:
find Server/Vendor/To_Company/* -exec mv -n ~/Desktop/MM-DD-YYYY/Returned_Files \;
but using that I can only run the script one time per day and there are times when I might need to run it multiple times.
It seems like ideally I should be able to create a copycat directory in my daily processing folder and then move the files from one to the other.
you can use rsync command with --remove-source-files option. you can run it as many times as needed.
#for trial run, without making any actual transfer.
rsync --dry-run -rv --remove-source-files Server/Vendor/To_Company/ ~/Desktop/MM-DD-YYYY/Returned_Files/
#command
rsync -rv --remove-source-files Server/Vendor/To_Company/ ~/Desktop/MM-DD-YYYY/Returned_Files/
reference:
http://www.cyberciti.biz/faq/linux-unix-bsd-appleosx-rsync-delete-file-after-transfer/
You could use rsync to do this for you:
rsync -a --remove-source-files /Server/Vendor/To_Company/Customer_Name ~/Desktop/$(date +"%y-%m-%d")/Returned_files/
Add -n to do a dry run to make sure it does what you want.
From the manual page:
--remove-source-files
This tells rsync to remove from the sending side the files (meaning non-directories) that are a part of the
transfer and have been successfully duplicated on the receiving side.
Note that you should only use this option on source files that are quiescent. If you are using this to move
files that show up in a particular directory over to another host, make sure that the finished files get renamed
into the source directory, not directly written into it, so that rsync can’t possibly transfer a file that is
not yet fully written. If you can’t first write the files into a different directory, you should use a naming
idiom that lets rsync avoid transferring files that are not yet finished (e.g. name the file "foo.new" when it
is written, rename it to "foo" when it is done, and then use the option --exclude='*.new' for the rsync trans‐
fer).
I'd like to be able to have rsync copy files I have listed in my bash script and then delete any files that do not exist in the source directory (if i change the file list in the script). Expanded the command looks like this:
rsync -axSR --delete src_dir1/file1 src_dir2/file2 dst_dir/
However, rsync is not deleting extraneous files. Is there a way to do this?
With this syntax, it could be because you are using multiple sources, and those sources are actual files.
Why not just sync the parent directory of src_dir to dst_dir?