I am needed to move entire directories from one computer in the network to the other (In a platform independent way). Basically I am working on some automation tool to help the developers do Build Verification Tests, for this; I am directed to automate the installation and un-installation of the product on multiple platforms. So, I will need to first copy the files!
And this is where I needed some help in both conceptual and practical knowledge.
Firstly, let me mention that using something like FileZilla or WinSCP is out of the question since I need things to happen automatically and not through button clicks. But please let me know if these tools have any command line utilities!
I tried Perl's NET::FTP, and while it looked promising, I was wondering whether it was the best way to go. Also, I want to know what are the pre-requisites before I can run FTP, I mean would I need perl installed on the other end as well ? I constantly read that the commands from perl's FTP actually try to connect to a FTP host, does this mean its not going to work if I haven't configured the remote host in some way? And if I am right, then what is this extra piece of configuration to be done?
Apart from this, is there any other way I could solve my problem ? I mean I am looking for API's here that would help me do platform independent file transfers. But once again, I cannot use tools that would need button clicks and stuff, because I am doing automation and everything needs be dome programmatic-ally and automatically.
Also, I think this is a very generic problem-statement: "Moving files across a computers connected by LAN"; So, it would be wonderful if we can have a list of (possibly) many options (ways to solve the problem) in the form of answers to this post.
Thanks in advance for any help that you wish to provide.
If nearly all of the files in your directory have changed, creating an archive, sending it over the network, and unarchiving makes sense. Actually, if your LAN is fast enough, though, it may be faster not to compress the archive--just use tar.
If only some of the files have changed, rsync, a command line tool, will only download the changes. It can be used with ssh like this:
rsync -ae ssh username#hostname:/path/to/files /store/here/locally
http://www.thegeekstuff.com/2010/09/rsync-command-examples/
On Linux and OS X, cron and crontab allow you to schedule scripts to run periodically. Windows provides the Windows Task Scheduler.
FTP is fine if you don't care about encryption over your LAN. Otherwise, SSH would be preferable.
rsync is available on OS X and Linux, but I think you can use it on Windows through Cygwin.
I suggest making an archive (e.g. a .tar.gz file) on the source host, transferring it with scp, and unarchive it on the target host.
You could also use unison or rsync
I would suggest you to develop your own FTP client in .NET. This way you will have complete control over the application, and instead of button-clicks you can schedule it using windows-scheduler. Here is an article about how to create your own FTP client in VB.NET:
http://dot-net-talk.blogspot.com/2008/12/how-to-create-ftp-client-in-vbnet.html
Related
I need to pull log files from a Windows server. The log files appear every minute and I am trying to ftp to the server pull the file back to the linux box and process it. I have been looking around and found about inotify, I am not sure how to use it in a bash script after ftp to the Windows server. I am open to other implementation it does not have to be ftp/inotify but I am not sure how this could be done.
any ideas??
Two products come to mind depending on how you plan to approach the solution.
I personally use Splunk on a variety of platforms (Windows & Linux servers/local Linux & OSX dev environments). It is a real-time log aggregator that features an API and the ability to query. Even if this doesn't solve your problem, the free version has some very robust features that you should consider: http://www.splunk.com
The second approach would be synchronization of your web directories using something like RSync. I've used RSync on Linux boxes and always appreciated what it can do. I even see it now has a Windows port: https://www.itefix.no/i2/cwrsync
I'm working with Textmate and Transmit.
I work locally and sync with the server every now and then.
At the moment I'm doing this by myself, it's slow and takes some time.
What are the best ways to sync between servers?
Also, is it possible to archive the directory and unpack it at the server? This would be so much faster...
Thanks!
What are the best ways to sync between
servers?
rsync? It's pretty much designed to do that exact task (syncing changes between files) and comes standard on most *nix OSes (including Mac OS X).
If you have, or could obtain, ssh access you could use something like SSHFS with MacFUSE
This would let you remote mount the server filesystem though Finder, as if it were local, but only using a ssh secure shell remote login.
Option 1
rsync --help
Option 2
Get an editor which is able to work with remote files.
Option 3
Mount remote directory and use your favorite editor.
Yummy FTP is for Mac and much faster than Transmit. It's the best you can get if you have only a FTP connection.
I'm working on a windows platform and want to be able to auto sync my files one way 'on change' to my virtual windows or linux web server - also need to be able to filter file types. i can connect to the remote machine via network drives.
i'm ideally looking for a free, easy to set up solution - a commercial product that does what I need is called ViceVersa but its a little overkill and costs :)
Thanks
Josh
I'd use rsync - simple, easy to setup, and provides the filters you need. Also very low on bandwidth after the first pass.
Here is a link explaining how to get it working in Windows
Whilst rsync doesn't allow 'on-change' auto-syncing, it is very fast when it scans a sync'ed directory (even very large ones), so you could schedule a frequent sync to overcome this.
Edit: You could combine it with a program like this, to trigger an rsync on folder contents change. Cheaper than viceversa
For other users, its worth mentioning lsyncd, it will auto sync on changes between two machines (by default deferring to rsync). Will only work on Linux though, but if thats not a problem it works great.
It also seems that Sparkleshare has finally released some working code (Dropbox clone). Havent tried it myself but does cross-platform synching and you can setup your own server.
Is there any way, any free software capable of automatic file upload? Let's say I edit php code on my local computer with my favorite IDE. I won't change my IDE, it's great. I want something that would detect a file is changed in my project directory and upload it with FTP/SFTP onto remote server. That's it - just that simple.
What I've already tried:
FTPDrive + FileSync Eclipse Plugin - it's quite slow, uploads ALL the files way to often, works buggy under Vista and Windows 7.
WinSCP automatic synchronization - bugs again, refuses to upload files randomly. Would be the best if it worked right.
Eclipse's native SFTP support - it's USELESS! You cannot use PDT projects with this feature. PDT without projects is no better than Notepad++.
Aptana FTP feature. It's worse than manual! Gawd, it sucks!
Running my own PHP/MySQL server under windows. First, it took me ages to set it up, then, it didn't work EXACTLY as my production environment - I hadn't been able to test my code correctly.
How it should work? I change file here, and it's uploaded there. It would be best, if it sit quietly in tray and bother me only if upload error occured.
Ok, if it's not free, maybe there's something cheap at least?
If there's nothing like it, is there something like FTPDrive?
rsync does exactly what you're asking.
Well, almost: it doesn't watch your filesystem and automatically upload files - you'd have to set up a task to run it every minute or whatever. But it does efficiently upload only the changes. If you're on Linux, lsyncd does the watching part and drives rsync to do the efficient upload part.
In the rails world, we tend to use source control and a deployment tool like vlad or capistrano. It's a bit safer and more consistent than FTP. This is a guide on how to use it with svn and php http://www.simplisticcomplexity.com/2006/08/16/automated-php-deployment-with-capistrano/.
You really should try to get your development server running on your personal machine. It's a much better way and it is worth the initial pain of trying to make it work. There are good tutorials on that out there somewhere.
You can use WebDrive or ExpanDrive, mount a complete remote directory as a local disk drive and directly edit your files on the server. However this highly depends on your connection and how your tools are written. Another approach could be to use one of these tools and with another tool sync all the changes asynchronously.
I like the functionality of dreamweaver where you can add a site and define an ftp and then when you save a file it saves a local copy and also uploads a file via ftp. I am trying to get similar functionality with linux. What I have thought of doing is have inotify monitor a local folder and upload any new or changed files to an ftp site, but I am having a hard time finding information on this. Any ideas on how I can accomplish this?
Also, I do not want to install any programs on the ftp server.
Thanks
Dean
You might want to take a look at cron scheduling an rsync job, which will efficiently copy changed files across a network at a chosen interval. rsync will use ssh or rsh (not ftp), so this might not work, but would seem a better way in most cases.
I'd throw together a python script which uses inotify and scp/ftp.
These are all common and should be supported by whatever distro your using. They're also all pretty well documented.