I need to manipulate hundreds of zip files ranging from 50Mb to 1500Mb.
Downloading them (particularly hard on heroku with their SquashFS transient system) doesn't seem viable.
Is there a way to remotely unzip files?
In short, no (at least not without SSH)
Extrapolating from Bobby's excellent answer here:(https://superuser.com/questions/479661/how-to-unzip-files-via-an-ftp-connection)
"It is not possible to unzip files remotely. FTP stands for "File Transfer Protocol", which was designed to transfer and partly manage files on the remote end, but not to execute commands. To unpack an archive you'd have to execute a program like tar, bzip2 or similar, but that's not possible via a FTP connection.
You need another session which allows you to execute commands, like SSH. Or you unpack the archive on your machine and transfer the contents via FTP, which will be considerable slower if you have a large number of small files because of the overhead of FTP."
Hope this helps.
Related
I need to automate the upload of some files from client PCs to a central server. We're building central statistics for an online gaming community, processing game replay files.
target is my own small VPS server running ubuntu
upload file size 2-3MB
20-40 different clients running windows spread around the globe
I expect ~6GB of wanted data to be uploaded over the course of 7 weeks (a season in our game) and 5-10x that amount of "unwanted" data.
The files are processed on the server, and then they're not required anymore, and ought to be deleted to not run out of disk space eventually. I also only need some of the files, but due to the files requiring very complex processing including decryption, so i can only determine that after the server processed it.
My initial idea was to use a scriptable client such as WinSCP, and use some Windows scheduler entry to automate it. WinSCP documentation looks very nice. I am a bit hesitant because I see the following problems:
after deletion on the server, how to prevent re-upload ?
ease of setup to technical novices
reliability of the solution
I was thinking maybe someone has done the same before and can give some advice.
There's article on WinSCP site that deals with all this:
How do I transfer new/modified files only?
For advanced logic, like yours, it uses PowerShell script with use of WinSCP .NET assembly.
Particularly, there is a section that you will be interested in: Remembering the last timestamp – It shows how to remember the timestamp of the last uploaded file, so that the next time you will transfer only newer files, even if the previously uploaded files are not on the server anymore.
The example is for downloads with Session.GetFiles, but it will with small changes work for uploads with Session.PutFiles too.
It also points to another article: Remember already downloaded files so they are not downloaded again, which shows another method – To store names of already transferrer file to a file and use it the next time to decide, which files are new.
I am working on building a application that will pull data from SFTP.
Basically I have written a shell script that's run on a cron job daily.
Now I want to know if I can implement some logic in the shell script that will scan the files (for security threats – such as: software viruses, worms, Trojans, adware, etc.) before downloading. Is that possible and how?
You basically want to "remote control" your SFTP session. There are several ways to do this, but I don't see, how you can scan a file for a virus using the (S)FTP protocol, without downloading it. This would require to execute a program on the remote side, and for what I know, FTP does not support this.
Maybe ssh would be the tool of choice. First open a ssh session, do all the checking of the files, then transfer them with FTP. If you are really paranoid, you can calculate the MD5 sum of the file before the checking, and after the download, and verify that they are identical.
I need to transfer webserver-log-like-files containing periodically from windows production servers in the US to linux servers here in India. The files are ~4 MB in size each and I get about 1 file per minute. I can take about 5 mins lag between the files getting written in windows and them being available in the linux machines. I am a bit confused between the various options here as I am quite inexperienced in such design:
I am thinking of writing a service in C#.NET which will periodically archive, compress and send them over to the linux machines. These files are pretty compressible. WinRAR can convert 32 MB of these files into a 1.2 MB archive. So that should solve the network transfer speed issue. But then how exactly do I transfer files to linux? I could mount linux drive on windows server using samba, or should I create an ftp server, or send the file serialized as a POST request. Which one would be good? Also, I have to minimize the load on the windows server.
Mount the windows drive on linux instead. I could use the mount command or I could use samba here (What are the pros and cons of these two?). I can then write the compressing and copying part in linux itself.
I don't trust the internet connection to be very stable, so there should be a good retry mechanism and failure protection too. What are the potential gotchas in these situations, and other points that I must be worried about?
Thanks,
Hari
RAR is bad. Stick to 7zip or bzip2. Transfer it using ssh, probably with rsync since it can be link-failure-tolerant.
WinSCP can help you transfer files from Windows to Linux in batch with script. Then configure Windows Task Scheduler to run the script periodically.
I learnt from this post step by step: https://techglimpse.com/batch-script-automate-file-transfer-winscp/
Are there any FTP programs which can automatically copy (or rather 'move') the contents of a folder to a remote server? I have of course googled this but only really found one or two ancient products which look really clunky and unmaintained. I was wondering if there's a way to do this from the command line or any better solution to the base problem.
In more detail, new files get written to a folder every few hours. These new files need to be FTP'd elsewhere and then deleted. Mirroring or synchonisation systems are probably out of the picture as we need to delete the source files once they've been successfully transferred.
If it's easier, the 'solution' could pull the files off the server (rather than the server pushing them to the client). The computers will both be Windows OS.
You could use any off the shelf FTP program that supports command line and schedule a task on Windows Scheduler to run every 10 minutes. Check the folder, and move any files to the FTP site.
In the end I used a program called FTP Auto Sync: http://ftp-auto-sync.com/
I like the functionality of dreamweaver where you can add a site and define an ftp and then when you save a file it saves a local copy and also uploads a file via ftp. I am trying to get similar functionality with linux. What I have thought of doing is have inotify monitor a local folder and upload any new or changed files to an ftp site, but I am having a hard time finding information on this. Any ideas on how I can accomplish this?
Also, I do not want to install any programs on the ftp server.
Thanks
Dean
You might want to take a look at cron scheduling an rsync job, which will efficiently copy changed files across a network at a chosen interval. rsync will use ssh or rsh (not ftp), so this might not work, but would seem a better way in most cases.
I'd throw together a python script which uses inotify and scp/ftp.
These are all common and should be supported by whatever distro your using. They're also all pretty well documented.