I've searched large and deep, but nothing is available, as far as I can see.
TLDR: How can I use rsync with a SharePoint installation? (Or something like rsync)
Long description
We have a large install base of Macs (~50%), Windows (~40%), and Linux (~10%), so our environment is pretty heterogeneous. Being an experimental job we produce a considerable amount of experimental datasets that we need to share, and more importantly, backup.
Right now we use external hard drives to store these files and folders, since our computers cannot hold these amount of data (50GB++, for instance, per dataset). And when we need to share, we "physically" share. We mainly we use rsync with some kind of backend (what kind is not important), but this solution requires computers to be left turned on, and act as servers.
For reasons that I will not bother you with, we cannot leave a computer on after work.
Having OneDrive for Business seemed a very promising technology to use, since we have more than 1TB per user. We could start syncing out datasets from our computers and hard drives, and we could share even when computers are turned off.
We are aware that we may hit some drawbacks, as not being able to actually share, having some limits about the number of objects (files/directories), but we will handle them later.
I prefer rsync, but right now we're open to any solution.
OneDrive for Business has a download that will allow you to synchronize a directory locally. https://onedrive.live.com/about/en-us/download/
For a Linux platform, you should be able to use onedrive-d found here:
https://github.com/xybu/onedrive-d
I know that it's an old question, but it's unanswered. Maybe a solution could be https://rclone.org/. Rclone is a command line program to sync files and directories to and from the cloud.
Related
Going to be working with a medium sized remote group on a large (but independent) project that will be generating many GB to TB of data.
To keep users from having to store 500GB of data on their personal machines, and to keep everyone in sync, we need a command-line/python utility to control selective syncing of dependencies on multiple operating systems: or at least osx and linux.
So example, someone who needs to work on the folder:
startrek/startrekiii
May require the folders:
startrek/nimoy/common
startrek/nimoy/[user]
startrek/shatner/common
startrek/shatner/[user]
but not:
startrek/startrekii, startrek/nimoy/[some_other_user], etc
From their command line (or a UI) they would run:
sync startrekiii
And they'd also receive startrek/nimoy/common, etc
likewise we'll have an unsync command that, as long as those dependent folders are not in use by another sync, will be unsynced and removed from the user's HD.
Of cloud sync/storage solutions, dropbox seems to offer the most granular control over this, allowing you to sync specific folders and subfolders - however from everything I can find this granular control is strictly limited to their UI.
We're completely open to alternative solutions if you have them, we just need something as easily deployable as possible and don't have the budget for Aspera or something to that effect.
Two other important notes:
Because of one very central part of our pipeline which pulls files
from those dependent folders (over which we have limited API
control), the paths need to be consistent on their respective
platform. So ~/Dropbox/startrek/nimoy can never be ~/Dropbox/startrek/startrekiii/nimoy
Many of the people using this will be artists and otherwise non-technical people, the extent of who's experience using csh or bash is for simple things like changing directories and moving files around.
Has anyone found a way to hack into Dropbox's selective sync, and/or know of a better alternative?
My parents got two shiny new iMac's (2015, 27") for Christmas, and for the first time ever they have separate computers. The problem is, they want everything between them to be the same. For now, I've copied both Desktop and Documents folders from the old computer (the only folders they really used), amounting to about 70GB of stuff. The problem is keeping things synced after I leave to head back to school.
Dropbox, SugarSync, SkyDrive, etc are out, as the amount of data is way past their limits, and privacy issues are there.
I was thinking of doing rsync in a cron job, to mimic how something like SyncBack Free may do this, but am very open to suggestions. Ideally, there would be two way synchronization, with most recent edits being written to the computer with the older version.
What would such a script look like? Can it be done for free?
In most of the games and programs you download, you just get the installer.
Some .exe files can be ran straightly, though (it's probably cause they don't have much source files to extract, huh?).
I was wondering, what's the difference between an installer, that just extracts the files, and a zip (rar, iso..) file, that you could download ,just depending on your internet speed, in up to few seconds. And where does a, maybe 200mb, installer fetch the, let's say 5gb of, files, offline?
I've never heard about this, and I'm learning to program, so I'd appreciate if you could answer me properly.
What you're really asking is:
How does an installer work?
A bit of background.
In the Before Times, man did not have such things as "installers." Software was run directly off of floppy disks (and none of that rigid 3.5" crap, I'm talking disks that flopped), like God intended.
Then came the first home computers with persistent hard drives. For the first time, it made sense to copy a program off a disk and have it stick around.
But programs still worked the way "portable" applications do today: you copied them as-is and ran them as-is.
Then operating systems began to get more complicated.
Windows introduced this notion of a registry: a central location where program and operating system configuration could be stored. Software authors began using this registry. Its arcane architecture and user-hostile editing utility (the infamous regedit.exe) made it the perfect place to store shareware information -- how many days you have left on your trial, for example.
This happened around the same time that programs began to be too large to fit -- uncompressed -- on a single floppy disk. A way was needed to split a program onto multiple disks. Since it wasn't very user-friendly to require the user to have e.g. a ZIP extractor installed (remember, this was before ubiquitous Internet), Windows programs began to be shipped with installers. You can think of these as basically portable versions of WinZIP whose sole purpose was to reassemble and extract a compressed file.
These days, installers serve a number of other purposes:
providing a convenient user interface
prompting the user to accept a click-through end-user license agreement (EULA)
prompting the user for CD keys (though this is being phased out for many systems in favor of digital distribution)
asking the user to register their software
and so on. They may also serve as DRM vehicles, validating CDs and decrypting data to prevent villainous individuals (yarr) from brrreakin' ye olde DMCA.
At their heart, they aren't any more complex than in the Windows 95 days -- a glorified unzip program.
Sidenote: Where does the installer get 5GB of data from 200MB of archives if not the Internet?
That's high, though there are plenty of ways you could get that compression ratio. Imagine a complex game whose world is defined in verbose XML -- that's readily compressible. You could even get that back in the old WinZIP days.
A zip file can only hold some files and then you unzip and get those files as is.
An installer however can be a very complicated program. It can create the needed files or folders structures, It can register the required dlls on your system, give you the options of the features that can be installed, Check your system for the compatibility and also be used as a wizard to guide you, step by step, to custom install you application.
An Installer (esp. Windows Installer) can make automatic Registry entries, as well as unpack and write files to a directory. With the Zip, you have to manually extract the files, and get no automatic registry edits.
The advantage to a zip is that it guarantees (most of the time) that the application is portable, that all necessary files are included in the unzipped directory.
The advantage of an installer is pretty obvious: automated, UI.
As for the 200mb -> 5gb....compressing the files into an exe can add another layer of more/better/smaller compression than that of just simply throwing the files into a zipped folder, however 200mb -> 5gb is a pretty big jump, not impossible, just pretty big. For most installers that do have instructions for large external (online) downloads, they typically let you know before hand that they are about to download a large chunk of data and to not disconnect from the internet during install....
An Installer or EXE Can Be Easily Get Affected By Virus But if there is ZIP archive than there are less chances for virus affection and using zip is more flexible too because it can be protected using you own password too.
Another Normal Benefit is that ZIP compress the files too.
Hope You are getting me.
So, I need to make a file storage for our team. Also I have SVN server. Opportunity to do rollbacks and control on who created or deleted file is very neccessary and important for our project.
Any ideas? Maybe without SVN. I can connect using WebDAV but only in read-only mode (because there is no LOCKS support in it).
You can set up the SVN server to allow exactly that.
Read the chapter in the SVN book about WebDAV and Autoversioning
So, what you want is the ability to roll back changes, and limit who can make the changes, but without the bother of checking in and out files?
Maybe Subversion isn't for you. I've done similar sharing with Dropbox and there's now BoxNet that's suppose to be like Dropbox on Steroids. Dropbox (and I assume box.net too) has some features that are very nice:
You can setup folder sharing between particular teams. That way, you can say who can and cannot access these files.
Dropbox automatically saves each and every version of a file, so you can always go back to previous versions -- even if that file has been deleted.
Files are stored locally. All a user has to know is to save a particular file in a particular folder, and everyone has access to it. I've successfully used Dropbox to collaborate with managers that make the Pointed Hair boss in Dilbert look like a high tech genius.
There's also Skydrive and Google Drive, but I don't find them as universal as Dropbox or as easy to use. It's possible to use Dropbox without ever going to the Dropbox website. To the non-geek, it appears to be magic as files I've written and edited appear on their drive. It took me a few weeks to train one person that he didn't have to email me his document when he made changes because I already had it.
Dropbox gives you 2 Gb of space for free which doesn't sound like a lot. However, my first hard drive was a whopping 20Mb which was twice the size of the standard 10Mb drive at that time. If you're not storing a lot of multimedia presentations or doing a lot of Photoshop, 2Gb might be more than enough for your project.
I know Windows 7 and later has some sort of versioning system built into it. I know this because anytime someone mentions that Mac OS X has time machine, some Wingeek pipes in stating that Windows has the same thing, but only better!. Unfortunately, Windows is not my forte, so I don't know too much about this specific feature. I believe the default is once per day, but it can be changed. This might be the perfect solution if everyone is on Windows.
Subversion can do autoversioning as Stefan stated. Considering his position in the Subversion community (especially his work on TortoiseSVN), he knows his stuff. Unfortunately I don't know too much about it since I've never used or seen this feature implemented. It's probably due to the fact that I work mainly with developers who know what a version control system is, and therefore have no need for something that does the versioning for them.
Also don't forget to check if you can use your corporate Sharepoint which does something very much what you want. I am not too impressed with Sharepoint, but if the facility is there, and your company can give you the support, it is something you probably want to look into.
I've been looking at the DropBox Mac client and I'm currently researching implementing a similar interface for a different service.
How exactly do they interface with finder like this? I highly doubt these objects represented in the folder are actual documents downloaded on every load? They must dynamically download as they are needed. So how can you display these items in finder without having actual file system objects?
Does anyone know how this is achieved in Mac OS X?
Or any pointer's to Apple API's or other open source projects that have a similar integration with finder?
Dropbox is not powered by either MacFUSE or WebDAV, although those might be perfectly fine solutions for what you're trying to accomplish.
If it were powered by those things, it wouldn't work when you weren't connected, as both of those rely on the server to store the actual information and Dropbox does not. If I quit Dropbox (done via the menu item) and disconnect from the net, I can still use the files. That's because the files are actually stored here on my hard drive.
It also means that the files don't need to be "downloaded on every load," since they are actually stored on my machine here. Instead, only the deltas are sent over the wire, and the Dropbox application (running in the background) patches the files appropriately. Going the other way, the Dropbox application watches for the files in the Dropbox folder, and when they change, it sends the appropriate deltas to the server, which propagates them to any other clients.
This setup has some decided advantages: it works when offline, it is an order of magnitude faster, and it is transparent to other apps, since they just see files on the disk. However, I have no idea how it deals with merge conflicts (which could easily arise with one or more clients offline), which are not an issue if the server is the only copy and every edit changes that central copy.
Where Dropbox really shines is that they have an additional trick that badges the items in the Dropbox folder with their current sync status. But that's not what you're asking about here.
As far as the question at hand, you should definitely look into MacFUSE and WebDAV, which might be perfect solutions to your problem. But the Dropbox way of doing things, with a background application changing actual files on the disk, might be a better tradeoff.
Dropbox is likely using FSEvents to watch for changes to the file system. It's a great API and can even bundle up changes that happened while your app was not running. It's the same API that Spotlight uses. The menubar app likely does the actual observing itself (since restarting it can fix uploads being hung, for instance).
There's no way they're using MacFUSE, as that would require installing the MacFUSE kernel extension to make Dropbox work, and since I definitely didn't install it, I highly doubt they're using it.
Two suggestions:
MacFUSE
WebDAV
The former will allow you to write an app that appears as a filesystem and does all the right things; the latter will allow you move everything server-side and let the user just mount your service as a file share.
Dropbox on the client is written in python.
The client seems to use a sqlite3 database to index files.
I suppose Dropobox split a file in chunks, to reduce bandwith usage.
By the way, it two people has the same file, even if they do not know each other, the server can optimize and avoid to transfer the file more times, only copying it on the server side
To me it feels like a heavily modified revision control system. It has all the features: updates files based on deltas, options to recover or restore old revisions of files. It almost feels like they are using git (GitFS?), or some filesystem they designed.
You could also give File Conveyor a try. It's a Python daemon capable of instantly detecting FS changes (on Linux through inotify, on OS X through FSEvents), processing the files and syncing them to one or more destinations.
Supported protocols: FTP, SFTP, Amazon S3 (CloudFront is also supported), Rackspace Cloud Files. Can easily be extended. Uses django-storages.
"processing files": e.g. optimizing images, transcoding videos — this was originally conceived to be used for sending static assets to a CDN in the context of speeding up websites)