So I am interested in a way (ideally cross platform) to have a zip file which is uploaded over annoyingly slow uplink connections (think ADSL) - where only the delta is uploaded (assuming a recent version is on the server and minimal changes to be uploaded).
Now rsync can work - with gzip rsync-aware support (ie you compress the file with gzip - but telling it to be rsync friendly) - but this is, well, a bit of a pain on windows.
Has this been solved before? or is rsync/gzip combo the state of the art?
(note that this network is asymetric - downloads are an order of magnitude faster - so not a bidirectional sync issue).
rdiff-backup is available for Windows as well, and pretty much intended for these kind of problems. Also handles binary diffs pretty well it seems. Only use it if it's non-mission-critical data, as the Windows-variant is not that well tested.
I use it for Linux, no experience with Windows whatsoever. It would probably be a good idea to compare hashes on local / remote location to be sure.
You have to implement some cronjob (schedule a job in the taskbar, don't know how it's called in Windows) to clean older files if you don't want incremental backups though.
Not sure if it fits your needs, but I think it comes very close and is definitely worth checking out!
The best solution seems to be "use rsync, even on windows"
Related
Here's my problem. I have OSX Lion and I do Web development, BUT I have no real comprehension of what I'm doing when I'm using brew, pear, and the terminal in general. I am working on leveling up, but I still have to work in the meantime. That's why I very often mess up my system files (just tried to install PHPUnit, didn't work, so I deleted other pear directories, still didn't work, and now I end up with a mess).
It would feel better and relieve a lot of stress to know I can revert back my changes when I mess up. So my question is, can I set up a version control like git on all my computer files themselves, so that before any big change, I can save the state of my computer? Or is there any other way to get that same result?
I think creating different users for my mac is not enough, cause I want to build up my system, and add things to it, so it doesn't really help. And I'm not sure, but Time Machine is made just to get some files, not to revert my system to some previous state, or can it do it?
Help would be greatly greatly appreciated, thanks!
Seems to me you need to use a VM.
Take snapshots and work without worries. If you mess up you just revert to your last known good snapshot
You can do this - you can version control anything... but I wouldn't recommend it (at least not with GIT/SVN/etc - perhaps there's some software designed for this purpose that I'm unaware of).
You'll be tracking version changes for tons of files, temporary, setting files, binaries, etc. Files would be changing all the time and you'd need to stay on top of commits and so forth. Instead I'd recommended just copying folders (backup), making changes, verifying your changes work, then deleting the backups.
It's very easy to overuse version control.
Having an external drive with time machine and allowing it to sync often will allow you to revert certain parts (or all) of the file system to a certain date.
Since you're under OS X, I'd suggest Time Machine - it is more adapted to what you want to do than a source control versioning. TM is pretty decent at backuping, but there are other solutions if this one doesn't fit your needs.
EDIT: as commented by #dstarh, brew isolates everything it installs and uses symbolic links when needed. So use it whenever you can, it leaves your system cleans. There's instructions on how to uninstall a software, and in the worst of the cases, you could look at the source of your software's formula and find out what to delete.
Long story short : yes you could, but there's way easier and painless ways to do this.
For some Webhost issues I have to write a file backup/syncronisation tool for a common OS in the server sector (Windows/Linux). Most Linux root-servers offers the ssh-interface for secure communication so I could use the SSH File Transfer Protocol, but what's the best solution on the windows side (on the fly) ? And are there good D libraries (or C alternatives)
I'm writing here and not in the admin or windows stack because there is one reason: It's important that there are existing libraries. So an easy implementation is more important than the existence of an interface or protocol. The simplicity and the language features and not the possibilities have priority.
All in all I am looking for an easy way to implement an os independent tool for a file exchange. For the synchronisation work it has to be possible to access some file information ( last write time, modified time, filesize, etc.)
edit:"My Version" of a synconisation tool should work on a new system without extra sotfware installation (maybe some automated installation over ssh-windows-equivalent [if there is one])
You only enter your access data and it should work. Furthermore I also need a protocol and this is the biggest problem. Because ssh doesn't work on windows on the fly - is there an equivalent?
rsync is a popular file synchronization tool, best suited to files being added, deleted, or extended. It's been very well debugged and is quite simple to set up. (rsync -avzP username#hostname:/path/to/source/ /path/to/dest/ or rsync -avzP /path/to/source/ username#hostname:/path/to/dest/ are common.)
rsync is frequently tunneled over ssh; it does have its own protocol if you don't mind it being publicly open.
But if you've got a lot of data that is being slightly moved, or frequent renames, a tool like git can make much better use of bandwidth. It does carry the downside of keeping history on both sides, which might be less disk-efficient than you'd like, but it can more than compensate if your bandwidth is a bit less amazing.
git is also frequently tunneled over ssh; it also has its own protocol if you don't mind it being publicly open.
I doubt either one has D library bindings, but C bindings ought to be easy to come by. :)
When I develop web applications I'm frequently need to sync files from a working folder to external server or another folder. I like keeping my code separated from the web sever.
In open source world there is the eclipse with file sync that does the job pretty well. Unfortunately I can't find any good replacement for Visual Studio.
I've only found two generic solutions:
- Winscp which is pretty good but stucks when a file is locked and ask for confirmation. Which is quite annoying.
- DSynchronize which works pretty well (ie. doesn't ask questions) but doesn't have filters so I can't tell it not to sync my .svn files or web.conf :(.
Do you know any good way to achieve realtime synchronization in Visual Studio or windows?
I doens't have to have gui in fact I would love to see a command line solution like a powershell command that outputs modified files.
I've ended up using Mercurial (to skip the .svn files) and DSynchronize to sync files
I would give a try to immortal classic - rsync. There is cygwin enabled implementation for Windows called cwrsync: http://www.itefix.no/i2/node/10650 . With proper configuration (potentially with some fine tuning with scripting as well) it will do perfectly.
If you would like to have bi directional synchronization, the Unison may be the answer:
http://www.cis.upenn.edu/~bcpierce/unison/
If you are looking for something even fancier, you might give a try to one of distributed file systems available, like CODA (I'm afraid decent Windows systems aren't supported yet): http://www.coda.cs.cmu.edu or native DFS solution from Microsoft, however I'm afraid the set up is too hassling (if not impossible in your case) since it's targeted for enterprise solutions:
http://technet.microsoft.com/en-us/library/cc753479(WS.10).aspx
Of course DFS option probably won't support filtering you are interested in.
At work I use ClearCase and SourceSafe, but have found some time to do some time to code for myself enroute thanks to a disposable laptop.
However, I wish I had a lightweight VCS on my system using which I would be able to make changes to my code during the commute and then push/grab them from my Linux systems.
I use git on my home system, but I can't really get it working on Windows. I don't want all that cygwin hack.
If it does not run natively on Windows, it just won't do.
What have you guys tried on your Windows system? Something that YOU use.
The big player at the moment seems to be Mercurial?
What would be best for a one (or maybe two) man team?
I just need to maintain :
Versioned copies of source code.
Checking in and out should be as less obtrusive as possible.
I am looking forward to a multiple Undo kind of feature (like that in an EMacs buffer) but persistent.
I really like the way git keeps track of lines moving between files in a source code set
I should be able to move part(s)/sub tree(s) of the source tree (each sub tree implies a module/plugin to my the main software I am building) to an archival system either completly or partially and restore them back from the archive as and when required and the system should track any changes to this tree as well.
I actually want to experiment with my code as much as possible without me manually keeping track of what I modified and what I need to undo once I try out some idea, so that I am back to where I want to continue from.
Notes : A similar topic came up a year ago : DVCS Choices - What's good for Windows?
I hope things have changed, and I really want people to share their own, real life experiences. Not something they recommend without using it or they think will work.
Bazaar and Mercurial both work very well on Windows. I posted in the question you linked, and since then, both have improved their Windows support even more. Using them is easy and flawless, and they even have GUIs if you swing that way.
I for one have switched from bazaar to git, and I've been pleased.
If you've a Clearcase background, why don't you take a look at Plastic SCM? Check this link, it will show you how it works on a distributed setup (and of course all the basic operations) http://codicesoftware.blogspot.com/2010/03/distributed-development-for-windows.html.
You won't miss any of the "good" clearcase features but all the shortcomings are simply gone (faster, installs on 45seconds, no cumbersome setup to use on a mixed Win/Linux scenario, built-in ACLs, excellent branching and merging, much better common ancestor algorithm, visualizations, better GUI, and you still have "selectors" in case you miss config_specs, but not being mandatory)
I've been looking at the DropBox Mac client and I'm currently researching implementing a similar interface for a different service.
How exactly do they interface with finder like this? I highly doubt these objects represented in the folder are actual documents downloaded on every load? They must dynamically download as they are needed. So how can you display these items in finder without having actual file system objects?
Does anyone know how this is achieved in Mac OS X?
Or any pointer's to Apple API's or other open source projects that have a similar integration with finder?
Dropbox is not powered by either MacFUSE or WebDAV, although those might be perfectly fine solutions for what you're trying to accomplish.
If it were powered by those things, it wouldn't work when you weren't connected, as both of those rely on the server to store the actual information and Dropbox does not. If I quit Dropbox (done via the menu item) and disconnect from the net, I can still use the files. That's because the files are actually stored here on my hard drive.
It also means that the files don't need to be "downloaded on every load," since they are actually stored on my machine here. Instead, only the deltas are sent over the wire, and the Dropbox application (running in the background) patches the files appropriately. Going the other way, the Dropbox application watches for the files in the Dropbox folder, and when they change, it sends the appropriate deltas to the server, which propagates them to any other clients.
This setup has some decided advantages: it works when offline, it is an order of magnitude faster, and it is transparent to other apps, since they just see files on the disk. However, I have no idea how it deals with merge conflicts (which could easily arise with one or more clients offline), which are not an issue if the server is the only copy and every edit changes that central copy.
Where Dropbox really shines is that they have an additional trick that badges the items in the Dropbox folder with their current sync status. But that's not what you're asking about here.
As far as the question at hand, you should definitely look into MacFUSE and WebDAV, which might be perfect solutions to your problem. But the Dropbox way of doing things, with a background application changing actual files on the disk, might be a better tradeoff.
Dropbox is likely using FSEvents to watch for changes to the file system. It's a great API and can even bundle up changes that happened while your app was not running. It's the same API that Spotlight uses. The menubar app likely does the actual observing itself (since restarting it can fix uploads being hung, for instance).
There's no way they're using MacFUSE, as that would require installing the MacFUSE kernel extension to make Dropbox work, and since I definitely didn't install it, I highly doubt they're using it.
Two suggestions:
MacFUSE
WebDAV
The former will allow you to write an app that appears as a filesystem and does all the right things; the latter will allow you move everything server-side and let the user just mount your service as a file share.
Dropbox on the client is written in python.
The client seems to use a sqlite3 database to index files.
I suppose Dropobox split a file in chunks, to reduce bandwith usage.
By the way, it two people has the same file, even if they do not know each other, the server can optimize and avoid to transfer the file more times, only copying it on the server side
To me it feels like a heavily modified revision control system. It has all the features: updates files based on deltas, options to recover or restore old revisions of files. It almost feels like they are using git (GitFS?), or some filesystem they designed.
You could also give File Conveyor a try. It's a Python daemon capable of instantly detecting FS changes (on Linux through inotify, on OS X through FSEvents), processing the files and syncing them to one or more destinations.
Supported protocols: FTP, SFTP, Amazon S3 (CloudFront is also supported), Rackspace Cloud Files. Can easily be extended. Uses django-storages.
"processing files": e.g. optimizing images, transcoding videos — this was originally conceived to be used for sending static assets to a CDN in the context of speeding up websites)