Is there a way to copy, at the same time, a single directory to multiple hard drives without the copy processes having each to read the source files? (Something like a Raid 1 perhaps)
In the specific, where I work we need to deploy (even multiple times a day) a folder containing around ~50gb of data from a PC (with Windows 7) to multiple others... using USB drives... (yup, usb drives, can't do anything about that). Often serially copying the files to each USB takes ages and it creates LONG dead times, especially for the last people to receive their copy.
Since the source PC in question has 8 USB 3.0 ports, would it be possible somehow to copy at the same time the source directory to USB drives in all 8 ports? (Of course without having 8 copy processes fighting for the limited read speed of the source hard drive... just readying each file and copying it to all destinations)
I tried searching for an answer but all I got were answers for linux or networked machines.
http://sourceforge.net/projects/n2ncopy/
I've searched it on google, i have never tryed the program
Related
I have created a dataset of millions (>15M, so far) of images for a machine-learning project, taking up over 500GB of storage. I created them on my Macbook Pro but want to get them to our DGX1 (GPU cluster) somehow. I thought it would be faster to copy to a fast external SSD (2x nvme in raid0) and then plug that drive directly into local terminal and copy it to the network scratch disk. I'm not so sure anymore, as I've been cp-ing to the external drive for over 24 hrs now.
I tried using the finder gui to copy at first (bad idea!). For a smaller dataset (2M images), I used 7zip to create a few archives. I'm now using the terminal in MacOS to copy the files using cp.
I tried "cp /path/to/dataset /path/to/external-ssd"
Finder was definitely not the best approach as it took forever at the "preparing" to copy stage.
Using 7zip to archive the dataset increased the "file" transfer speed, but it took over 4 days(!) to extract the files, and that for a dataset an order of magnitude smaller.
Using the command line cp, started off quickly but seems to have slowed down. Activity monitor says I'm getting 6-8k IO's on the disk. It's been 24 hours and it isn't quite halfway done.
Is there a better way to do this?
rsync is the preferred tool for this kind of workload. It is used for both local and network copies.
Main benefits are (excerpt from manpage):
delta-transfer algorithm, which reduces the amount of data sent
if it is interrupted for any reason, then you can restart it easily with very little cost. It can even restart part way through a large file
options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied.
Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.
Regarding command usage and syntax, for local transfers is almost the same as cp:
rsync -az /path/to/dataset /path/to/external-ssd
I'm facing the problem of frequently copying multiple hundred huge media files (each way bigger than 100GB) from a Synology NAS to a local Thunderbolt RAID via LAN using a Mac. I've tried many different options ranging from Finder to rsync. None of which seem to work great (or even good).
Surprisingly just copying using Mac OS Finder is the fastest option that I tested so far. It averages at about 100MB/s. But for many reasons (error handling, no checksums, .DS-Store problems, etc) this method is not at all satisfactory.
Although I do not need to sync anything (just copy entire directories from A to B), rsync has exactly what I need in terms of features but it seems to be way too slow for this kind of task. It averages at about 50-70MB/s (-vaEWh --progress).
My question is: Am I holding it wrong or is rsync not the right tool for this? What else could be used to copy many very large files at speeds above 100MB/s with a decent amount of monitoring and logging?
Thanks a lot!
Our wedding videographer gave us our wedding video (all 200GB of it) on an external hard drive formatted for Mac.
My wife and I each have PCs and we have 2 external hard drives formatted for PCs. We have a Mac that's half dead, but we could use it for group sharing or something, but it doesn't have any internal space to store the large files.
Is there any way to get the files from this external hard drive onto one of our PCs?
Not exactly a programming problem, but anyhow...
OSX uses HFS plus, not NTFS so a quick google search and you'll see that there are products available for what you want: -
http://www.pcadvisor.co.uk/how-to/windows/3369574/how-read-mac-os-hfs-drives-in-windows-for-free/
When I boot up Linux Mint from a Live CD, I am able to save files to the "File System". But where are these files being saved to? Can't be the disc, since it's a CDR. I don't think it's stored in the RAM, because it can only hold so much data and isn't really intended to be used as a "hard drive". The only other option is the hard drive... but it's certainly not saving to any partition on the hard drive I know about, since none of them are mounted. Then where are my files being saved to??
Believe it or not, it's a ramdisk :)
All live distros mount a temporary hard disk in RAM memory. The process is completely user-transparent and is all because of the magic of Linux kernel.
The OS, in fact, first allocates an area of your RAM memory into a virtual device, then mounts it as a regular hard drive in your file system.
Once you reboot, you lose all your data from that ramdrive.
Ramdrive is needed by almost all software running on Live CDs. In fact, almost all programs, in particular desktop managers, are designed in order to write files, even temporary, during their execution.
As an example, there are two ways to run KDE on a Live CD: either modify its code deeply in order to disallow you to change wallpaper etc. (the desktop settings are stored inside ~/.kde) or redeploy it onto a writable file system such as a ramdrive in order to avoid write fails on read-only file systems.
Obviously, you can mount your real HDD or any USB drive into your virtual file system and make all writes to them permanent, but by default no live distro mounts your drives into the root file system, instead they usually mount into specific subdirectories like /mnt, /media, /windows
Hope to have been of help.
It does indeed emulate a disk using RAM; from Wikipedia:
It is able to run without permanent
installation by placing the files that
typically would be stored on a hard
drive into RAM, typically in a RAM
disk, though this does cut down on the
RAM available to applications.
RAM. In Linux, and indeed most unix systems, any kind of device is seen as a file system.
For example, to get memory info on linux you use cat /proc/meminfo, where cat is used to read files. Then, there's all sorts of strange stuff like /dev/random (to read random crap) and /dev/null (to throw away crap). ;-)
To make it persistent - use a USB device - properly formatted and with a special name. See here:
https://help.ubuntu.com/community/LiveCD/Persistence
To make things easier when switching between machines (my workstation at the office and my personal laptop) I have thought about trying an external hard drive to store my working directory on. Specifically I am looking at Firewire 800 drives (most are 5400 rpm 8mb cache). What I am wondering is if anyone has experience with doing this with Visual Studio projects and what sort of performance hit they see.
It depends on the size of the project. The throughput is low and the latency is high, so you're going to get hit every which way, but due to the latency you'll be hit harder if you have a lot of little files rather than a few large ones.
Have you considered simply carrying around a GIT or other distributed repository and updating the machine repositories as you move around? Then you can compile locally and treat the drive and a roving server. Since only changes will be moved across, it should be faster, and your code will be 'backed up' in more places.
If you forget the drive, it breaks, or is lost/stolen, then you can still sit down at a PC and program with no code missing if you're at the last PC you used, or very little code missing (which will be updated later with a resync anyway).
And it's just a hop skip and a jump away from simply using the network to move the changes between the systems if you don't want to carry the drive around later.
I use vmware and the virtual machines are on an external usb drive. Performance is fine. You might have some issues with the drive name changing - not an issue if you use virtual machines.
Granted I work in an industry were Personal Information and Intellectual Property are king, but I don't like that idea at all. That hard drive disappears and you have a big problem.
Why not Remote Desktop into the work machine?
EDIT Stipud Spelingg