When uploading new files to FTP server, how to prevent reupload of files that were deleted on the server meanwhile - ftp

I need to automate the upload of some files from client PCs to a central server. We're building central statistics for an online gaming community, processing game replay files.
target is my own small VPS server running ubuntu
upload file size 2-3MB
20-40 different clients running windows spread around the globe
I expect ~6GB of wanted data to be uploaded over the course of 7 weeks (a season in our game) and 5-10x that amount of "unwanted" data.
The files are processed on the server, and then they're not required anymore, and ought to be deleted to not run out of disk space eventually. I also only need some of the files, but due to the files requiring very complex processing including decryption, so i can only determine that after the server processed it.
My initial idea was to use a scriptable client such as WinSCP, and use some Windows scheduler entry to automate it. WinSCP documentation looks very nice. I am a bit hesitant because I see the following problems:
after deletion on the server, how to prevent re-upload ?
ease of setup to technical novices
reliability of the solution
I was thinking maybe someone has done the same before and can give some advice.

There's article on WinSCP site that deals with all this:
How do I transfer new/modified files only?
For advanced logic, like yours, it uses PowerShell script with use of WinSCP .NET assembly.
Particularly, there is a section that you will be interested in: Remembering the last timestamp – It shows how to remember the timestamp of the last uploaded file, so that the next time you will transfer only newer files, even if the previously uploaded files are not on the server anymore.
The example is for downloads with Session.GetFiles, but it will with small changes work for uploads with Session.PutFiles too.
It also points to another article: Remember already downloaded files so they are not downloaded again, which shows another method – To store names of already transferrer file to a file and use it the next time to decide, which files are new.

Related

Working efficiently on remote projects in Visual Studio

On occasion I telecommute and need to work on the project files on my box at the office via my laptop. I bounce back and forth between two methods of doing this:
I remote into the machine at my office and work on the instance of Visual Studio on that machine through RDP.
I have the folder containing the solution files set up as a network share and load the solution on my local (laptop) install of Visual studio
These options are far from perfect. Loading the solution via network share makes for a responsive experience unless VS needs to read/write files, which happens with more actions than one might realize. The RDP method avoids the network latency of reading and writing files, but makes for a laggy experience, plus I have to reconfigure VS's layout everytime I log in to accommodate a single monitor.
What I'd like to do is continue to work on my local/laptop copy of VS but figure out a way to speed it up substantially. For a variety of reasons, I do not want to have to commit to and pull from source control to share across machines.
Any suggestions on how to speed up working with Visual Studio when loading remote project files?
Storing your local files under a Dropbox installation can help with synchronizing your files, but it comes with a couple caveats:
On each machine that you synchronize, you must be able to store the files under your /Dropbox directory. This may be feasible on your laptop but not on your office box, for example.
Dropbox synchronizes at its own casual pace. This is most noticeable on its initial sync, but also imposes a short delay on subsequent synchronizations. When you open your laptop at home, you'll need to keep an eye on the taskbar icon to ensure it is "up to date" before you begin working.
Given those caveats, I love using Dropbox for my personal projects for a couple reasons:
Since I don't use source control on those projects, it is relatively easy to overwrite something you want to keep. Dropbox gives you fast access to your entire revision history, allowing you to roll back whenever needed. This does not count against my free quota.
Even when I delete or remove files, Dropbox continues to store them and allows me, years later, to undelete and recover them. Again, all of this happens without impacting my free quota.
As the OP mentioned in the question's comments, Google Drive offers a solution similar to Dropbox.

Any problems having a visual studio project on Google Drive?

I often make copies of my Visual Studio project files to Google Drive as "oh crap" backups. What if I just had the whole project on the Google drive and just worked on the project directly on the Google Drive?
Anyone try this? Any issues crop up from it?
TLDR version: As long as you develop in serial it's fine. Parallel development across devices = bad.
More info:
I use it for sharing development projects between devices (Mac and various Win 10 PCs), and its a plus/minus affair overall (as of Jan 2016):
1) An integrated GIT folder seems to work perfectly fine being transferred between machines back and forth, as long as you are using only one machine at a time, commit only from one machine at a time, and use one machine as a focus.
2) Separately from GIT, using devices in parallel is bad (i.e. dev on both simultaneously for multiplayer testing & development), and leads to source files being confused and duplicated, as GD will try to sync them before work on them is done or committed (every time a file is saved). Develop only on one machine at a time.
3) Large projects have trouble syncing quickly because of the large number of small files (>1510 files changed over the course of a day or two, e.g. the OBJ folders, the Bin folders with different build configs etc.. etc.. ), so assume an increase in electricity costs keeping machines on to keep the sync up to date.
4) Executing projects also has an issue, as Google Drive will lock files for upload as VS makes them and then VS can't copy them, the build will time out every single time. It's best to pause Google Drive when actively developing on a machine (then turn on during the 5 minute tea breaks etc..).
5) It's great for keeping Content in sync, i.e. art files (3DS MAX, Photoshop, Paint, etc..), music, textures, language files, all the stuff that's too big/impractical/problematic to use GIT for.
I keep my personal Visual Studio Projects in Google Drive. Not for backup but for sharing between computers (PC/laptop/tablet).
This Works pretty well. The only problem i notice is when i run some Projects Visual Studio deletes some files and Google Drive keeps asking if i really want to delete the files.
Google Backup and Sync will keep syncing your bin and obj folders as you build/debug your program. This consumes unnecessary network, disk, and CPU usage. To prevent that, see: https://stackoverflow.com/a/68704427/10355922

Are there any FTP programs which can automatically send the contents of a folder to a remote server?

Are there any FTP programs which can automatically copy (or rather 'move') the contents of a folder to a remote server? I have of course googled this but only really found one or two ancient products which look really clunky and unmaintained. I was wondering if there's a way to do this from the command line or any better solution to the base problem.
In more detail, new files get written to a folder every few hours. These new files need to be FTP'd elsewhere and then deleted. Mirroring or synchonisation systems are probably out of the picture as we need to delete the source files once they've been successfully transferred.
If it's easier, the 'solution' could pull the files off the server (rather than the server pushing them to the client). The computers will both be Windows OS.
You could use any off the shelf FTP program that supports command line and schedule a task on Windows Scheduler to run every 10 minutes. Check the folder, and move any files to the FTP site.
In the end I used a program called FTP Auto Sync: http://ftp-auto-sync.com/

Automatic file upload

Is there any way, any free software capable of automatic file upload? Let's say I edit php code on my local computer with my favorite IDE. I won't change my IDE, it's great. I want something that would detect a file is changed in my project directory and upload it with FTP/SFTP onto remote server. That's it - just that simple.
What I've already tried:
FTPDrive + FileSync Eclipse Plugin - it's quite slow, uploads ALL the files way to often, works buggy under Vista and Windows 7.
WinSCP automatic synchronization - bugs again, refuses to upload files randomly. Would be the best if it worked right.
Eclipse's native SFTP support - it's USELESS! You cannot use PDT projects with this feature. PDT without projects is no better than Notepad++.
Aptana FTP feature. It's worse than manual! Gawd, it sucks!
Running my own PHP/MySQL server under windows. First, it took me ages to set it up, then, it didn't work EXACTLY as my production environment - I hadn't been able to test my code correctly.
How it should work? I change file here, and it's uploaded there. It would be best, if it sit quietly in tray and bother me only if upload error occured.
Ok, if it's not free, maybe there's something cheap at least?
If there's nothing like it, is there something like FTPDrive?
rsync does exactly what you're asking.
Well, almost: it doesn't watch your filesystem and automatically upload files - you'd have to set up a task to run it every minute or whatever. But it does efficiently upload only the changes. If you're on Linux, lsyncd does the watching part and drives rsync to do the efficient upload part.
In the rails world, we tend to use source control and a deployment tool like vlad or capistrano. It's a bit safer and more consistent than FTP. This is a guide on how to use it with svn and php http://www.simplisticcomplexity.com/2006/08/16/automated-php-deployment-with-capistrano/.
You really should try to get your development server running on your personal machine. It's a much better way and it is worth the initial pain of trying to make it work. There are good tutorials on that out there somewhere.
You can use WebDrive or ExpanDrive, mount a complete remote directory as a local disk drive and directly edit your files on the server. However this highly depends on your connection and how your tools are written. Another approach could be to use one of these tools and with another tool sync all the changes asynchronously.

Do you have performance problems when you work on Visual Studio projects via a network share?

We have tremendous problems with Visual Studio (2008, if that matters) locking up and slowing down when accessing projects over a network drive. It can take several minutes to open a large Web site project through a mapped drive, and saving even a single file can take a minute or more.
I fired up Wireshark and watched the traffic. VS, it seems, requests massive amounts of files from the network -- there's an enormous amount of SMB traffic. I've done some research, and this traffic seems to stem from two situations.
VS has to have everything in its own process to provide Intellisense.
VS needs to have all the source in order to compile the project.
All the advice I've read seems to boil down to the same thing: work locally, not on a remote machine, then push your code to an integration server via source control.
This would sure solve our problems (VS is quite fast working locally), but what if you can't work locally? What if the project and the infrastructure required to run it is too large and complicated to be replicated on everyone's individual machines?
We've gone 'round this problem a couple times, and the only way we can figure to work on these projects is direct access via a mapped drive. However, the VS slowness and lockups are really becoming a problem.
One solution: we installed VS on the server and work on the projects directly on the servers via RDP. Seriously.
So, I ask:
What does everyone else do? Do you work via the network, or do you replicate projects locally? If remotely, do you suffer from VS performance issues.
We work locally and use SVN to keep all our code on the server.
I find VS 2008 quite slow working locally sometimes so I wouldn't fancy working on a network share.
Trying to compile over a network share is horribly slow using visual studio. Your start times will be bad as the intellisense database is regenerated. Each compilation has to go over the network multiple times. Linking takes forever.
If you need the output of your compilation on the network, I'd recommend doing your compile locally and defining a post-build command to copy the results to your share.
If, as you say, you cannot pull everything locally then I'd suggest your project is too big and needs to be broken up into more manageable chunks. For a multi-tier application, break it up by tier and invest in some form of continuous integration (e.g. CruiseControl) to automatically build individual pieces. In this way you can work locally on an particular piece and pull the pre-build portions from CI for the other pieces of the application.
I'm not terribly surprised that using VS to load projects over a network share has performance issues. VS (in any language) is constantly getting information from files in the project. Once you start loading this over a network you're at the mercy of the underlying network connection. All lags and access issues will directly translate into VS having an issue loading file contents.
I would advise copying the solution locally and using some form of source code control to sync the project on the share.
If the code is too complicated to install on everyone's machine, then don't put it on everyone's machine. Does everyone need to have everything in order to do productive work?
I have 79 projects in my solution that I work with. Several hundred thousand lines of code. I pull my source down everyday from TFS and build it; it's a lot of code, but it's a far better solution than trying to work over a network share.
A more legitimate situation of having the source code on a share is when one has a non-Windows host on which a (number of) virtual Windows machine is running.
I have this exact situation where my desktop machine (the host) is running Debian and I use VMware to run various virtual Windows machines (the guests), including one that has Visual Studio installed so that I can target Windows OS's. Having the source code on a Samba share on the host machine has the following pro's:
The source is not duplicated, so there is no way to confuse different copies while working on several virtual machines at the same time.
I have full control over the source from my preferred OS.
I can turn on and off any of the virtual machines, or roll back to a snapshot, without the risk of loosing changes.
I can build (etc.) from the same source on several machines without having to commit changes before the source fully tested (reason: I have to use Subversion <1.5).
The only problem with this setup is that Visual Studio (6,7,8,9) is painfully slow.
I have mounted the partition (on which the share lies) with "relatime" and this works in as far as the disk activity on the share moderate, but Visual Studio keeps the (virtual) network card occupied all the time.
Any solutions to this would be very appreciated.
I encountered similar problems everytime I worked (work = anything else then just copy / paste files) over a network drive. The problem occured with ZendStudio and Eclipse.
Why not use any kind of source control?
When working on Windows based projects I've always worked locally.
Once at a unix shop (AIX iirc) developers would work via NFS mount and checkin/checkout via RCS...
I'm using VS2005 across to a network share and not having any performance issues. However, it is a new server (Windows Server 2008). I don't have any other data points for VS since using it at work is relatively new for me.
However, some datapoints from using Netbeans for previous projects on a network share... Local build time for my project was 2 minutes on Vista, on a fast dual-core AMD 64-bit machine. For a network share project, on a Server 2003 box, it was 20 minutes. Building that same project from an ancient Tablet PC (1ghz, single core) running XP locally was around 5 minutes. Interestingly enough, the Tablet PC could build on the Server 2003 box in the same 5 minutes.
For those asking "why" on the network share. The network share is automatically backed up, archived, etc. Also, that way I can very easily look at the same projects from multiple machines without having to worry about pushing back into the repository, etc. Once you've gone to having your dev stuff on a device where you can get to it from anywhere/anything, you'll never want to do local storage again!
I have performance problems via network anything, they just aren't good enough yet.
I thought it was common knowledge that disk-speed is one of the major "slowness" factors when it comes to using VS in Windows. Most dev machines I've built have had projects located on 10k RPM RAID0 drives, or at least a single 10k RPM drive. And even then it seems slow sometimes. Just the way it is, I suppose, until VS2009/VS2010 fixes it? :)
From my experience, this lag when working on a network share is 99% due to Intellisense. Disable it and you'll see.
disabling Intellisense indeed speeds up saving and opening files trough a UNC share dramatically
http://blogs.msdn.com/saraford/archive/2007/12/03/did-you-know-how-to-turn-off-intellisense-by-default.aspx
but then again, as stated in other comments, you might as well use a good text editor
I've also experienced the problems with performance mentioned above. It seems to vary from project to project, but I did find one way of speeding up performance significantly for some project types.
Following the advice in this article made a previously unusable project on a network location (it would take minutes to open one file) perform almost like a local project. The basic gist is that you need to grant FULL TRUST to the network location:
To grant permission to all your projects in your Visual Studio Projects folder located on the network, follow these 8 steps:
Open Microsoft .NET Framework 1.1 (or 2.0) Configuration which you'll
find under Administrative Tools in the Control Panel.
Expand Runtime Security Policy | Machine, | Code Groups | All_Code |
LocalIntranet_Zone In the right-hand pane, click Add a Child Code
Group.
In the dialog that follows choose Create a new code group and fill in
a Name like Visual Studio Projects.
Optionally, provide a Description for the Code Group. (You'll see the
description when you click a Code Group in the left tree, helping you
identify the various Code Groups you may have) .
In the Condition Type drop down, choose URL
For the URL field, type something like this:
file://YourServer/My Documents/Visual Studio Projects/*
Under Use existing permission set, choose FullTrust (that is, if you
trust your own applications. If you don't, choose a different
permission set or create a new one).
Not sure why this works, but it made a previously unusable NET 2.0 project perform significantly better.
Original article: http://imar.spaanjaars.com/364/how-do-i-allow-my-visual-studio-net-projects-to-run-from-a-network-location
I was having the same problem. I have a local copy of our build system, which expects certain drive letters, and was also experiencing slowness.
I have solved the problem by adding the following registry keys:
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\DOS Devices]
"R:"="\DosDevices\D:\devel\build
"S:"="\DosDevices\D:\devel\src"
Note that the double '\'s above are part of the .reg file format. When using regedit use single '\' throughout.
My build times were divided by 3. :)
I found the info in the wikipedia article on the SUBST command.

Resources