Sync files from Windows to Freebox(Linux) : prevent file recopy - bash

I try to sync files from my Windows 7 computer to my Freebox (running Linux
I used rsync from unix like terminal (MobaXterm) and xcopy directly from windows bash as well.
In both case, whatever the set of option I used (inclusing operations on file attributes), I am unable to sync properly.
Problem is that each file is considered as new at each sync, and hence copied again, even if unmodified.

Problem comes from the file system. Windows is usually NTFS while Freebox (and some Linux) is FAT32 (in general not NTFS). File comparison based on dates concludes on differences between files. Using robocopy command on window with special option /fft solves the issue.
Example:
robocopy src dest /fft /e /purge
Will copy all files recursively from src to dest, removing on dest the non existing ones on src and handling correctly the files date attributes through different file systems, avoiding unecessary recopy.

Related

Synchronize windows folders

I have set up remote sync in my eclipse to copy jsp and js files at different locations. I am observing that sometimes because of this sync (I need to keep build auto option enabled) eclipse is hanging and I need to kill the process.In windows do we have any option to sync two local folders. I searched but options I am getting through third party software. I am using my office laptop, so don't want to use any third party software and want to check if windows provide any easy option for that.
I miss unix and rsync :(
Currently I am using a bat file to copy these files. I think robocopy is a good option too.
xcopy /s /d /y source folder destination foler
Fortunately, there is a useful little program in Windows called robocopy that ships with Win7 and above.
https://technet.microsoft.com/en-us/library/cc733145.aspx
robocopy <source> <destination> /mir /copyall
This is what you can use to copy a source directory to a target directory including all subdirectories, files, and metadata. It's uni-directional so won't check both ways, but there are time and change based triggers you can set with a windows startup task so you can make the folders auto-sync as you work.
If you read the link, you'll find the /mot: and /mon: flags which will watch the folders for changes/wait a certain amount of time then copy again.
In addition, it is really good at logging output, and is excellent when used as a system startup process.

Difference between xcopy and robocopy

I'm kind of new to batch scripting.
As a newbie I find both both of them useful while scripting
What are the key differences between xcopy and robocopy?
Robocopy replaces XCopy in the newer versions of windows
Uses Mirroring, XCopy does not
Has a /RH option to allow a set time for the copy to run
Has a /MON:n option to check differences in files
Copies over more file attributes than XCopy
Yes i agree with Mark Setchell, They are both crap. (brought to you by Microsoft)
UPDATE:
XCopy return codes:
0 - Files were copied without error.
1 - No files were found to copy.
2 - The user pressed CTRL+C to terminate xcopy. enough memory or disk space, or you entered an invalid drive name or invalid syntax on the command line.
5 - Disk write error occurred.
Robocopy returns codes:
0 - No errors occurred, and no copying was done. The source and destination directory trees are completely synchronized.
1 - One or more files were copied successfully (that is, new files have arrived).
2 - Some Extra files or directories were detected. No files were copied Examine the output log for details.
3 - (2+1) Some files were copied. Additional files were present. No failure was encountered.
4 - Some Mismatched files or directories were detected. Examine the output log. Some housekeeping may be needed.
5 - (4+1) Some files were copied. Some files were mismatched. No failure was encountered.
6 - (4+2) Additional files and mismatched files exist. No files were copied and no failures were encountered. This means that the files already exist in the destination directory
7 - (4+1+2) Files were copied, a file mismatch was present, and additional files were present.
8 - Some files or directories could not be copied (copy errors occurred and the retry limit was exceeded). Check these errors further.
16 - Serious error. Robocopy did not copy any files. Either a usage error or an error due to insufficient access privileges on the source or destination directories.
There is more details on Robocopy return values here: http://ss64.com/nt/robocopy-exit.html
The most important difference is that robocopy will (usually) retry when an error occurs, while xcopy will not. In most cases, that makes robocopy far more suitable for use in a script.
Addendum: for completeness, there is one known edge case issue with robocopy; it may silently fail to copy files or directories whose names contain invalid UTF-16 sequences. If that's a problem for you, you may need to look at third-party tools, or write your own.
The differences I could see is that Robocopy has a lot more options, but I didn't find any of them particularly helpful unless I'm doing something special.
I did some benchmarking of several copy routines and found XCOPY and ROBOCOPY to be the fastest, but to my surprise, XCOPY consistently edged out Robocopy.
It's ironic that robocopy retries a copy that fails, but it also failed a lot in my benchmark tests, where xcopy never did.
I did full file (byte by byte) file compares after my benchmark tests.
Here are the switches I used with robocopy in my tests:
**"/E /R:1 /W:1 /NP /NFL /NDL"**.
If anyone knows a faster combination (other than removing /E, which I need), I'd love to hear.
Another interesting/disappointing thing with robocopy is that if a copy does fail, by default it retries 1,000,000 times with a 30 second delay between each try. If you are running a long batch file unattended, you may be very disappointed when you come back after a few hours to find it's still trying to copy a particular file.
The /R and /W switches let you change this behavior.
With /R you can tell it how many times to retry,
/W let's you specify the wait time before retries.
If there's a way to attach files here, I can share my results.
My tests were all done on the same computer and
copied files from one external drive to another external,
both on USB 3.0 ports.
I also included FastCopy and Windows Copy in my tests and each test was run 10 times. Note, the differences were pretty significant. The 95% confidence intervals had no overlap.
Its painful to hear people are still suffering at the hands of
*{COPY} whatever the version.
I am a seasoned batch and Bash script writer and I recommend rsync , you can run this within cygwin (cygwin.org) or you can locate some binaries floating around .
and you can redirect output to 2>&1 to some log file like out.log for later analysing.
Good luck people its time to love life again .
=M. Kaan=
I have written lot of scripts to automate daily backups etc. Previously I used XCopy and then moved to Robocopy. Anyways Robocopy and XCopy both are frequently used in terms of file transfers in Windows. Robocopy stands for Robust File Copy. All type of huge file copying both these commands are used but Robocopy has added options which makes copying easier as well as for debugging purposes.
Having said that lets talk about features between these two.
Robocopy becomes handy for mirroring or synchronizing directories. It
also checks the files in the destination directory against the files
to be copied and doesn't waste time copying unchanged files.
Just like myself, if you are into automation to take daily backups
etc, "Run Hours - /RH" becomes very useful without any interactions.
This is supported by Robocopy. It allows you to set when copies
should be done rather than the time of the command as with XCopy. You
will see robocopy.exe process in task list since it will run
background to monitor clock to execute when time is right to copy.
Robocopy supports file and directory monitoring with the "/MON" or
"/MOT" commands.
Robocopy gives extra support for copying over the "archive" attribute
on files, it supports copying over all attributes including
timestamps, security, owner, and auditing information.
Hope this helps you.
They are both rubbish! XCOPY was older and unreliable, so Microsoft replaced it with ROBOCOPY, which is still rubbish.
http://answers.microsoft.com/en-us/windows/forum/windows_8-files/robocopy-appears-to-be-broken-in-windows-8/9a7634c4-3a9d-4f2d-97ae-093002a638a9
Don't worry though, it is a long-standing tradition that was started by the original COPY command, which to this day, still needs the /B switch to get it to actually copy properly!

How to copy and delete files using a bat file with arguements

I want to copy files across computers on the network using a bat file (I want to run the bat file in windows scheduler) but only have files copied based on their last modified dates, e.g. only copy files across to another computer only if the last modified date is older than 3 months then when the coping is done have the copied files delete on the source computer.
Robocopy (included in Windows 7+, downloadable for older versions from Microsoft's site) has options that do exactly what you need.
Robocopy /mov /minage:90 [src] [dest]
would seem to be close to what you need.

Windows cmd: Copy over entire directory *including* parent directory. Solution without specifying same parent directory name

In Windows, how do you copy an entire directory, INCLUDING the parent directory folder?
For instance, let's say we have the directory c:\Folder and want to copy it over to d: .
The only way right now would be:
xcopy /E c:\Folder d:\Folder
Is there a way to do this without specifying the same end directory (Folder)?
I'm told that an application called RoboCopy can do it, however, I believe it's part of a series of server 2003/2008 tools -- I can't speak to its capabilities or whether it will work on your version of Windows. That said, there are a set of tools (basically UNIX commands ported to DOS) located here that will do what you need -- specifically the "cp" command. My apologies for not being able to assist further.

RSync copies only folder directory structure not files

I am using RSync to copy tar balls to an external hard drive on a Windows XP machine.
My files are tar.gz files (perms 600) in a directory (perms 711).
However, when I do a dry-run, only the folders are returned, the files are ignored.
I use RSync a lot, so I presume there is no issue with my installation.
I have tried changing permissions of the files but this makes no difference
The owner of the files is root, which is also the user which the script logs in as
I am not using Rsync's CVS option
The command I am using is:
rsync^
-azvr^
--stats^
--progress^
-e 'ssh -p 222' root#servername:/home/directory/ ./
Is there something I am missing to get my files copied over?
I can think of only a single possibility: My experience with rsync is that it creates the directory structure before copying files in. Rsync may be terminating prematurely, but after this directory step has been completed.
Update0
You mentioned that you were running dry run. Rsync by default only shows the directory names when the directory and all its contents are not present on the receiver.
After a lot of experimentation, I'm only able to reproduce the behaviour you describe if the directories on the source have later modification dates than on the receiver. In this instance, the modification times are adjusted on the receiver.
I had this problem too, and it turns out that backing up to a windows drive from linux doesn't seem to copy the temp files in place, after they are transferred over.
Try adding the --inplace flag, when rsyncing to windows drives.

Resources