I have a bash script that utilises inotify-tools to wait for .zip files to be dropped in a substructure under the root. From there they are unzipped into a another directory.
When I copy the .zip files in with WinSCP the script executes correctly. Copying the .zip files with Filezilla leads to this error however:
End-of-central-directory signature not found. Either this file is not
a zipfile, or it constitutes one disk of a multi-part archive. In the
latter case the central directory and zipfile comment will be found on
the last disk(s) of this archive.
I've googled this error and the two main issues seem to be an old version of Linux's unzip functionality, which I have a newer version of, and trying to copy files that are > 2gb (this file isn't)
Anyone know the issue here, it seems to me that Linux is trying to unzip the script before it is fully copied to disk? Like I said, only filezilla has this error, I don't get it with winSCP
I believe your main issue is you try to process the ZIP when it is still being transfered. Probably what happens is that as soon as the transfer is initiated WinSCP creates a temporary files to store the transfered data. That event would fire your script before the zip file is complety transfered.
That would explain why you get this error :
End-of-central-directory signature not found. Either this file is not
a zipfile,
So the solution would be to have two folders one for transfer and one for compelete. They should be in the same file system. On transfer complete just move from one folder to another.
Related
This may be asking too much from an already very powerful tool, but is there a chance that lftp mirror can execute a command during the mirroring process (from remote directory to the local machine)?
Specific example: lftp is asked to mirror a remote directory with xml files into a local folder and as soon as each file is downloaded/updated, it converts the file to JSON format using xml2json.
I can think of a solution that relies on monitoring the local copy of the mirrored folder for changes via find and then executing xml2json on the new/updated files, but perhaps there is a simpler way?
You can use xfer:verify and xfer:verify-command settings to run a local command on every transferred file.
I am trying to update a module to a newer version. In the past I have manually uploaded each file carefully into the new directory and overwritten older files using FTP. However I wanted to use SSH to try and do this more easily and without any file permission problems.
I have:
Uploaded the .tgz file to the root folder (/http) on the server
Logged into the server via SSH
Changed the directory to the correct directory
Run the following command: tar -zxvf fishpig_splash.tgz
In the command line I was then given a list of all the files that had been extracted. However if I use FTP to go to any of these files I can see that they are still the older version and have not been overwritten.
I was expecting that the files would extract into the correct directories and overwrite any that already existed. I have tested the extraction by creating a temporary directory and extracting into that and everything worked fine.
Is there another part to this script I need to use to overwrite the files?
Thanks
Glynn
Sorry this was just me being stupid! When extracting the tar file there was a subfolder within it for the extension, I completely missed it. I just went down a level in the file and zipped up the contents only then extracted them at the root and everything worked fine. Thanks for the help though!
I'm just trying to curl this zip file, then unzip it
curl -sS https://www.kaggle.com/c/word2vec-nlp-tutorial/download/labeledTrainData.tsv.zip > labeledTrainData.tsv.zip
unzip labeledTrainData.tsv.zip labeledTrainData.tsv
but I keep getting the error;
Archive: labeledTrainData.tsv.zip
End-of-central-directory signature not found. Either this file is not
a zipfile, or it constitutes one disk of a multi-part archive. In the
latter case the central directory and zipfile comment will be found on
the last disk(s) of this archive.
I'm using the same syntax as found in this response, I think. Is there something wrong with the file I'm downloading? I feel like I'm making a noob mistake. I run these two commands in a shell script
I am able to replicate your error. This sort of error generally indicates one of two things:
The file was not packaged properly
You aren't downloading what you think you're downloading.
In this case, your problem is the latter. It looks like you're downloading the file from the wrong URL. I'm seeing this when I open up the alleged zip file for reading:
<html><head><title>Object moved</title></head><body>
<h2>Object moved to here.</h2>
</body></html>
Long story short, you need to download from the alternate URL specified above. Additionally, Kaggle usually requires login credentials when downloading, so you'll need to specify your username/password as well.
How do I make LFTP download a file from a remote server only if this file was created TODAY (the same day I run LFTP) ?
Use mirror.
It has this --newer-than=SPEC option to download only files newer than specified time. For your specific needs, use --newer-than=now-1days. Now - 1 day should be yesterday therefore lftp will download all the file newer than yesterday.
Refer here for more info: http://lftp.yar.ru/lftp-man.html
EDIT: While I was tweaking my script, I notice there's an --only-newer option which download only newer file which is also useful for your case but with slight changes. --only-newer check the destination folder and download any files from source that's not in the destination folder while --newer-than download any files that's newer than the time you specified without checking the destination folder.
I have a scenario where I am inserting the data from FTP file into various systems. I have a couple of questions regarding FTP in Mule
-Is it possible to have a temp/work directory in FTP Inbound endpoint? The moveToDirectory works as a processed folder, I would like to move my source file into temp directory first and then have Mule download it.
-On success, the scenario demands, the original file to be moved from source/temp folder to success Folder. On an occurrence of an error, the original file should be moved to Failure Folder. Is it possible to only move and rename original file rather than storing the file in the payload and write it back in success or failure scenario?
Thanks,
Varada