I am trying to copy a large database backup file over network.
The .net FileInfo.CopyTo or xcopy failed with different binary resulted. Also tried robocopy without parameter, and resulted a failure. Just did another attempt with robocopy using /zb parameter (restart and backup mode). It took much longer but resulted in a success.
My question is, is backup mode in robocopy really designed to copy large / backup file? Have searched through net and couldn't find a clear answer.
Would appreciate if any experienced user could give me a hint or better solution on large file copy over network. Thanks.
Just want to share an update on resolving the issue above.
In my case, xcopy failed to copy the file over 10GB across servers in different domain and server location.
On the other side, robocopy with
/zb - Uses Restart mode. If access is denied, this option uses Backup mode.
can successfully copy. It increased the time from 1hour to 2.5hours though.
--
After re-arranging server, the file is copied across servers in the same domain and server location now. And using xcopy is alright too.
--
So my theory on this would be probably about the stability connection between servers. If the connection is not robust (with occasionally drop out causing an access issue), when copying large file like my case, a corruption likely occurs any time during the long process; robocopy with restart and backup can recover the copy pretty well. Time spent on recovery is probably the down side.
And as a side note, FTP instead of copy should be used if it's going to be a routine task.
Backup mode only allows Robocopy to back up and restore all files, regardless of their own individual permissions (NTFS ACLs) on those files.
Related
I need to automate the upload of some files from client PCs to a central server. We're building central statistics for an online gaming community, processing game replay files.
target is my own small VPS server running ubuntu
upload file size 2-3MB
20-40 different clients running windows spread around the globe
I expect ~6GB of wanted data to be uploaded over the course of 7 weeks (a season in our game) and 5-10x that amount of "unwanted" data.
The files are processed on the server, and then they're not required anymore, and ought to be deleted to not run out of disk space eventually. I also only need some of the files, but due to the files requiring very complex processing including decryption, so i can only determine that after the server processed it.
My initial idea was to use a scriptable client such as WinSCP, and use some Windows scheduler entry to automate it. WinSCP documentation looks very nice. I am a bit hesitant because I see the following problems:
after deletion on the server, how to prevent re-upload ?
ease of setup to technical novices
reliability of the solution
I was thinking maybe someone has done the same before and can give some advice.
There's article on WinSCP site that deals with all this:
How do I transfer new/modified files only?
For advanced logic, like yours, it uses PowerShell script with use of WinSCP .NET assembly.
Particularly, there is a section that you will be interested in: Remembering the last timestamp – It shows how to remember the timestamp of the last uploaded file, so that the next time you will transfer only newer files, even if the previously uploaded files are not on the server anymore.
The example is for downloads with Session.GetFiles, but it will with small changes work for uploads with Session.PutFiles too.
It also points to another article: Remember already downloaded files so they are not downloaded again, which shows another method – To store names of already transferrer file to a file and use it the next time to decide, which files are new.
My FreeNAS server is slowly dying and before that happens i need to migrate all data in the NAS to a windows server.
The FreeNAS has ZFS Snapshots and i need to restore data from a few days ago to the Windows server.
I have done some research and i can't think of the best way to do this. (i am not linux/Zfs savvy)
So the things i need to do is,
Restore ZFS Snaptshot from a few days ago to a windows Server
I mounted a windows share to the Freenas using mount_smbfs //username:password#server.name/share_name share_name/
I can copy and create files on that share just fine. So I was wondering if it was possible to restore an entire data set from an snapshot to the windows share.
Any help, tips is much appreciated.
Note. I could easily copy all data on a freenas volume to the windows share, but what makes it complicated for me, is restoring data from a snapshot without overwriting the current data on the volume and moving that data to the windows share.
You have two sensible possibilities:
Access the ZFS dataset (shared over SMB) from your Windows Server, then right-click on it in Explorer and choose "Previous Versions". You will get (after a short time depending on the number of snapshots) a list of all snapshots with their dates. You can then either explore them and copy some files over, or you can choose to copy all to another location (e. g. your new share).
Mount the Windows share on FreeNAS like you did, then go to <pool>/<filesystem>/.zfs/snapshot/ (path completion on the shell might be turned off for the .zfs directory, so type it in manually). There you'll find all your snapshots (like you would have on Windows' Previous Versions) and you can copy some or all files over to the new directory.
I would suggest the first way, because you have the GUI and cannot do any harm to the FreeNAS system this way.
On the other hand, have you thought about the possibility of rescuing the system? You did not specify why it's dying, but things like hard drives or mainboards can be swapped quite easily without requiring setting up everything anew. Maybe this would help you more than moving the data off to another, unconfigured system?
Are there any FTP programs which can automatically copy (or rather 'move') the contents of a folder to a remote server? I have of course googled this but only really found one or two ancient products which look really clunky and unmaintained. I was wondering if there's a way to do this from the command line or any better solution to the base problem.
In more detail, new files get written to a folder every few hours. These new files need to be FTP'd elsewhere and then deleted. Mirroring or synchonisation systems are probably out of the picture as we need to delete the source files once they've been successfully transferred.
If it's easier, the 'solution' could pull the files off the server (rather than the server pushing them to the client). The computers will both be Windows OS.
You could use any off the shelf FTP program that supports command line and schedule a task on Windows Scheduler to run every 10 minutes. Check the folder, and move any files to the FTP site.
In the end I used a program called FTP Auto Sync: http://ftp-auto-sync.com/
I'm using a windows copy command to perform auto file backup of Microsoft outlook files for a domain users.
the problem exists after 6 month of using this batch file is that the .pst file size for each user is growing and the copy operation is getting too long to complete.
i want to know are there any way to copy only the changed bits in the .pst file couse what i'm doing is copying the whole file each time a user login.
copy C:\"Documents and Settings"\%USERNAME%\"Local Settings"\"Application Data"\Microsoft\Outlook\ \\storage\folder1\folder2\%USERNAME%\%DESTDIR% /y /v
thanks
This page might help you in doing that : link
In a word: with the regular Windows copy command, no.
You've a few alternatives, though. Probably the best is to ask your users (or get your administrator) to reduce the size of the mailbox.
Another is to try the robocopy utility, which is part of the Windows Server Resource Kit and which replaces xcopy in versions of Windows above Server 2008 R2. In my experience this is a little faster. It also includes an option to skip the copy if the file hasn't changed, which may help if your users only use email infrequently.
rsync (details here, Windows version here) does a true differential copy, so only the changed bytes between the files are copied. However this needs to be installed as a Windows service on the listening side and needs to be installed as a program on each client.
I am thinking about a script/program that can run in background, and attempt to backup or synchronize a given filesystem path to a mirror location (probably located on an external/separate storage device).
This should apply to Windows but it could as well be used under Linux.
Differential/incremental backups are a bonus.
Windows System State backups are a bonus too.
Keeping the origin free of meta-data is essential. (unlike version control)
Searching by file or activity date could be interesting (like version control)
Backup repositories should be easy to browse and take little space.
Deleted files should be available for recovery for a period of time.
Windows Backup is tedious and bloated and limited.
Tar-gzipping is not accessible.
User interaction during backup should be nonexistent.
Amanda is the ultimate full-featured open-source backup solution, and there's a (relatively) new Zmanda Windows Client.
Duplicity is free and creates encrypted, incremental, compressed offsite backups. It's a linux app, but you could run it in cygwin or a small virtual machine.
I've written a perl script that runs it via a cronjob to backup several very big directories over DSL and it works great.
Check out AJCBackup. Does an excellent job at a good price.
Acronis True Image is great. It's not free but the Home edition is pretty cheap for what it does and it works reliably. Does image- and file- based backups, scheduling, instant backup of chosen folders accessible from explorer context menu, incremental/differential backups, can mount the backup files as Windows volumes and browse them, copy files out etc. It has saved my ass a few times already.