After some reading and trying rsync copy over only certain types of files using include option I can't get seem to get it to work.
I run the following command:
rsync -zarv -vvv -e ssh --prune-empty-dirs --delete --include="*/" --include="*.csv"
--include="*.hdf5" --include="*.pickle" --include="*.tar.gz" --include="*.bin"
--include="*.zip" --include="*.npz" --exclude="*" . user#host.com:/rsync
But at the target it backups any file I have in the directory and subdirectories. delete-before and delete-after does not delete files like .txt or .py. I have also tried the --exclude="*" before the extension includes but I am running 2.6.9 so it should be after as far as I have understood it.
Deleting files on the host machine will just sync them again for whatever reason I don't know.
Your command looks fine, although try using --delete-excluded instead of --delete.
--delete-excluded - also delete excluded files from destination dirs
It should eliminate any files that are --excluded and not --included on the destination.
Sorry to have bothered. This was a bash issue and not a command issue.
As I was using the command:
exec $COMMAND
instead of
eval $COMMAND
This made god knows what for error but executing it manually (after print) and correctly in bash made it work. Deleting items still seems flaky but that I can try some.
Related
So I am cloning one folder to another using Bash. Currently my script is recursive and noclobber. Works great.
!cp -r -n /content/gdrive/Shared\ drives/Source/. /content/gdrive/Shared\ drives/Destination
This copies just fine. I just am looking for a way to delete any files if NOT on the Source drive but IS on the Destination drive. Maybe I need an entirely different script method?
Edit. I ended up using
!rsync -v -r --ignore-existing /gdrive/Shared\ drives/Source/. /gdrive/Shared\ drives/Destination --delete
Seems to be working for now. I was using -u but it seemed to be re-copying files just because the date changed, not the file itself. Thanks 1218985 for the help!
You can do it like this with rsync:
rsync --delete "/content/gdrive/Shared\ drives/Source/" "/content/gdrive/Shared\ drives/Destination/"
I am working on a script to copy a custom user profile over to the default. Part of the script uses rsync to copy the contents of the customized profile, named "profile" to the default profile located in: /System/Library/User Template/English.lproj. Each time I run the command interactively, it fails indicating "No such file or directory." I can browse to the directory in the Finder. I can navigate to the directory via the terminal. Why can't rsync find it? Here is the command:
rsync -av /Users/profile/* /System/Library/User\ Template/English.lproj
I tried a similar approach using cp -R instead of rsync and got a whole bunch of file not found error messages. Using the cd command to that same path also fails. I can step through each individual directory and arrive at English.lproj but I can't do it in one command. Any ideas where I am going wrong with my command?
Thanks
Jason
I take delivery of files from multiple places as part of a publishing aggregation service. I need a way to move files that have been delivered to me from one location to another without losing the directory listings for sorting purposes.
Example:
Filepath of delivery: Server/Vendor/To_Company/Customer_Name/**
Filepath of processing: ~/Desktop/MM-DD-YYYY/Returned_Files/Customer_Name/**
I know I can move all of the directories by doing something such as:
find Server/Vendor/To_Company/* -exec mv -n ~/Desktop/MM-DD-YYYY/Returned_Files \;
but using that I can only run the script one time per day and there are times when I might need to run it multiple times.
It seems like ideally I should be able to create a copycat directory in my daily processing folder and then move the files from one to the other.
you can use rsync command with --remove-source-files option. you can run it as many times as needed.
#for trial run, without making any actual transfer.
rsync --dry-run -rv --remove-source-files Server/Vendor/To_Company/ ~/Desktop/MM-DD-YYYY/Returned_Files/
#command
rsync -rv --remove-source-files Server/Vendor/To_Company/ ~/Desktop/MM-DD-YYYY/Returned_Files/
reference:
http://www.cyberciti.biz/faq/linux-unix-bsd-appleosx-rsync-delete-file-after-transfer/
You could use rsync to do this for you:
rsync -a --remove-source-files /Server/Vendor/To_Company/Customer_Name ~/Desktop/$(date +"%y-%m-%d")/Returned_files/
Add -n to do a dry run to make sure it does what you want.
From the manual page:
--remove-source-files
This tells rsync to remove from the sending side the files (meaning non-directories) that are a part of the
transfer and have been successfully duplicated on the receiving side.
Note that you should only use this option on source files that are quiescent. If you are using this to move
files that show up in a particular directory over to another host, make sure that the finished files get renamed
into the source directory, not directly written into it, so that rsync can’t possibly transfer a file that is
not yet fully written. If you can’t first write the files into a different directory, you should use a naming
idiom that lets rsync avoid transferring files that are not yet finished (e.g. name the file "foo.new" when it
is written, rename it to "foo" when it is done, and then use the option --exclude='*.new' for the rsync trans‐
fer).
I am a walking through a tutorial and it lets me delete two directories(im using one 10 deep, all empty) but once i try to remove the third it gives me that error message, even though there is not content in the directory and i am in the directory above it. Why is this? By the way i am using terminal.
That error is reported when the directory is not empty.
To find out what files are in that directory use ls -a. The -a flag tells ls to list hidden files (aka "dot files"). Remove those files then rmdir can be used.
Another option is to simply use rm -rf to recursively delete the directory and all of its files. NOTE: this can be dangerous if you give the wrong parameters - resulting in deleting more than you intended.
you can remove all hidden files by using rm -R ./.* you have to be in the directory the hidden file is in for it to work
On Windows I recently has the same problem, and deleting everything in {UserFolder}\AppData\Local\Composer\files didn't helped.
What I've done is to launch multiple times the composer install --dry-run command until it listed all dependencies, then I successfully ran the composer install command.
I'm trying to push changes to my server through ssh on windows (cygwin) using rsync.
The command I am using is:
rsync -rvz -e ssh /cygdrive/c/myfolder/ rsyncuser#192.168.1.110:/srv/www/prj112/myfolder/
/srv/www/prj112/myfolder/ is owned by rsyncuser. My problem is that eventhough with rsync the sub directories are create as they publish, each directory is assigned default permission of d--------- so rsync fails to copy any files inside it.
How do I fix this?
The option to ignore NTFS permissions has changed in Cygwin version 1.7. This might be what's causing the problem.
Try adding the 'noacl' flag to your Cygwin mounts in C:\cygwin\etc\fstab, for example:
none /cygdrive cygdrive user,noacl,posix=0 0 0
You can pass custom permissions via rsync using the 'chmod' option:
rsync -rvz --chmod=ugo=rwX -e ssh source destination
Your problem stems from the fact that the Unix permissions on that directory really are 0. All of the access information is stored in separate ACLs, which rsync does not copy. Thus, it sets the permissions on the remote copy to 0, and, obviously, is unable to write to that directory afterwards.
You can run
chmod -R 775
on that directory, which should fix your rsync problem.
After a look at the manpage I can tell that the chmod param is available in rsync since version ~2.6.8. But you have to use --chmod=ugo=rwX in combination with rsync -av
You should also try this command:
rsync -av <SOURCE_DIR> rsyncuser#192.168.1.110:/srv/www/prj112/myfolder
It would work on Linux at least. And note that rsync does not need to mention ssh--at least on Linux.
But if all fails and just to give an option you may take a look at this ready packed-up tool cwRsync
if you deploy a site from windows (for ex. octopress use rsync) it's possible set permission to 775 adding multiple chmod command:
rsync -avz --chmod=ug=rwx --chmod=o=rx -e ssh
To rsync from Windows to Unix/Linux you should provide a command like
SET BACKUP_SERVER=my.backup.server
SET SSH_USER=theUnixUserName
SET STORAGEPATH=/home/%SSH_USER%/Backup/
SET STORAGEURI=%BACKUP_SERVER%:%STORAGEPATH%
SET SSH_ID=/cygdrive/c/Users/theWindowsUserName/Documents/keyfiles/id_dsa
SET EXCLUDEFILE=backup_excludes.txt
SET BACKUPLOGFILE=/cygdrive/c/Users/theWindowsUserName/Backuplogs/backup-%DATE%-%TIME::=-%.log
The ssh command then is
SET BACKUP=rsync -azvu --chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r --rsh="ssh -l %SSH_USER% -i '%SSH_ID%'" --exclude-from=%EXCLUDEFILE% --delete --delete-excluded --log-file="%BACKUPLOGFILE%"
with backup_excludes.txt containing lines of ignored elements like
.git
.svn
.o
\Debug
\Release
Then you would use this in a script with
%BACKUP% /cygdrive/c/mySensibleData %STORAGEURI%
%BACKUP% /cygdrive/c/myOtherSensibleData %STORAGEURI%
%BACKUP% /cygdrive/c/myOtherSensibleData2 %STORAGEURI%
and so on. This will backup your directories mySensibleData, myOtherSensibleData and myOtherSensibleData2 with the permissions 755 for directories and 644 for files. You also get backup logs in your %BACKUPLOGFILE% for each backup.
Cygwin rsync will report permission denied when some process has the target file open. Download and run Process Explorer and find out if anything else is locking the file or simply try renaming the file and see if you get the Windows error about some other process having the file open.
Also, you can try to create a (global) environment variable CYGWIN and set its value to nontsec