I'm new to the above topics and struggling to find the necessary material I need. I've a task to complete where I need to publish from my broker (raspberry pi 3) to a client also a (raspberry pi 3) through a zip file and unzip it at the
Client side, through base64(encoder/decoder). I am able to do this through ssh no problem but I'm unsure the protocol for implementation in MQTT. Any advice at all would be greatly appreciated thank you.
1) You need to convert the zip file encoded using base 64 to binary format
2) You can publish the binary data to any topic in MQTT.
3) Another client subscribed to this topic must get the binary data and then convert to back to the zip format.
The limitation of pay load size allowed through mqtt is 256 MB. So first make sure your zip file is less than the limit specified.
Related
My goal is to download a large zip file (15 GB) and extract it to Google Cloud using Laravel Storage (https://laravel.com/docs/8.x/filesystem) and https://github.com/spatie/laravel-google-cloud-storage.
My "wish" is to sort of stream the file to Cloud Storage, so I do not need to store the file locally on my server (because it is running in multiple instances, and I want to have the disk size as small as possible).
Currently, there does not seem to be a way to do this without having to save the zip file on the server. Which is not ideal in my situation.
Another idea is to use a Google Cloud Function (eg with Python) to download, extract and store the file. However, it seems like Google Cloud Functions are limited to a max timeout of 9 mins (540 seconds). I don't think that will be enough time to download and extract 15GB...
Any ideas on how to approach this?
You should be able to use streams for uploading big files. Here’s the example code to achieve it:
$disk = Storage::disk('gcs');
$disk->put($destFile, fopen($sourceZipFile, 'r+'));
I have several FTP servers (4 servers), where there are files that are generated by an application.
This application generates the same type of file with the same structure in the 4 servers.
With Talend, I want to when any change to a file in one of the servers I need to recover their data and put in in Active MQ.
What could you suggest ? Because in tFTP I don't have tWaitForFile
Staying within that architectural approach... You could poll the ftp servers to detect a change in a file's updated Timestamp or size .
I am using FtpClient of Apache Commons Net to upload videos to FTP server. To check if the file has really been successfully transferred, I want to calculate the checksum of remote file, but unfortunately I found there is no related API I could use.
My question is: Whether there is a need to calculate file checksum in ftp server? If the answer is yes, how to get checksum in FtpClient?
If the answer is no, how do FtpClient know if the file has really been successfully and completely transferred?
With FTP, I'd recommend to verify the upload, if possible.
The problem is that there's no widespread standard API for calculating checksum with FTP.
There are many proposals for checksum calculation command for FTP. None were accepted yet.
The latest proposal is:
https://datatracker.ietf.org/doc/html/draft-bryan-ftpext-hash-02
As a consequence, different FTP servers support different checksum commands, with a different syntax. HASH, XSHA1, XSHA256, XSHA512, XMD5, MD5, XCRC, to name some. You need to check what, and if any at all, your FTP server supports.
You can test that with WinSCP. The WinSCP supports all the previously mentioned commands. Test its checksum calculation function or checksum scripting command. If they work, enable logging and check what command and what syntax WinSCP uses against your server.
> 2015-04-28 09:19:16.558 XSHA1 /test/file.dat
< 2015-04-28 09:19:22.778 213 a98faefdb2c36ca352a2d9b01668aec6b641cf4b
Then execute the command using Apache Commons Net sendCommand method:
if (FTPReply.isPositiveCompletion(ftpClient.sendCommand("XSHA1", "filename"))
{
String[] reply = ftpClient.getReplyStrings();
}
(I'm the author of WinSCP)
If your server does not support any of the checksum commands, you do not have many options:
Download the file back and check it locally.
When using encryption (TLS/SSL), chances of the file being corrupted during transfer are significantly lower. The receiving party (server in this case) would otherwise fail to decrypt the data. So if you are sure that the file transfer completed (no decryption errors and the size of the uploaded file is the same as size of the original local file), you can be pretty sure that the uploaded file is correct.
Just a addition of how I implemented this. When dealing with standard ftp servers without any additionak modules loaded for checksum checking, all i did was creating a list of MD5 CRC hashes about each file into an SFV file. Say its called uploads.sfv (just in the same format as sfv generator would do). This allows you to do further checksum checks.
Examples about the server side support checksum checking support:
PZS-ng for cuftpd, glftpd
mod_digest for ProFTPD
Of course as #MartinPrikryl highlighted, none of these are standardized.
That's a long shot, but if the server supports php, you can exploit that.
Save the following as a php file (say, check.php), in the same folder as your name_of_file.txt file:
<? php
echo md5_file('name_of_file.txt');
php>
Then, visit the page check.php, and you should get the md5 hash of your file.
Related questions:
FTP: copy, check integrity and delete
How to perform checksums during a SFTP file transfer for data integrity?
https://serverfault.com/q/98597/401691
This question already has answers here:
How to confirm SFTP file delivery?
(3 answers)
Closed 1 year ago.
I'm uploading the file through Sftp to destination server using bash scripts.
How I can be sure that the file which is uploaded is complete upload in the case sftp will not return anything or network connection could be broken?
I see that I can get the size of the file before uploading to the server and then I can compare it with the existing size for the file on the server.
Perhaps you can mention about other better options?
Thank you.
I think getting the size is a good option.
What I could imagine :
Client side :
- Put the size of the file, and its md5 in a file, like ".fileinfo"
- Send the fileinfo to the server
- Send the (interesting) File to the server
Server side :
- Check periodically files of a folder (with "watch ls" command for example)
- If a ".fileinfo" exists, read it, and check if the size corresponds to an existing file of the same name (without ".filefome"). If the size corresponds, do an "md5sum" of the file, and check if it corresponds. If yes, move your file into your final destination folder, and delete the ".fileinfo" file. If not reiterate.
Many sites for downloading softwares will provide both the software and its checksum.
we can use the same technique to check our uploading file.
upload the file together with its checksum, on the server side compare the file's checksum with uploaded checksum,
if the two don't match, you will know
The file uploaded is corrupted, or
The checksum uploaded is corrupted, or
Both the checksum and file uploaded are corrupted.
Test exit code of sftp. If it returns 0 you can be pretty sure that everything is ok (assuming you are using OpenSSH sftp). This works only when you use -b switch (what I assume you are doing).
SFTP protocol allows checksum calculation, but I suppose you are stuck with OpenSSH (or either or both sides) that does not support this.
To be 100% sure, you can download the file back and compare with original.
Note: I'm using Windows file servers and .NET
If I were to create a TAR file from files on a remote file server (meaning, the TAR file would be created on the remote file server, where the original files are), would the bytes need to come to my machine and then go back to the file server (since my machine is running the code that's generating the TAR), or would they stay on the file server? I'm asking about the best possible (theoretical) implementation.
Thank you!
The bytes need to be where they are processed.
If you process them on your remote system, they must be transferred.
If you process them on your server, they don't need to be transferred.
If your goal is to minimize bandwidth usage, your best bet would be to have a script on your server that will generate the tar files for you when triggered by your remote system.
The best possible implementation really depends on what your goals and constraints are.
The bytes would have to be read into your machine. The only way I know that you can just do the TARing on the remote server is to have the remote server generate the TAR. For example, you could connect via SSH and run a shell command on the remote server.
Unfortunately, in the scenario described, the TAR operation will use network bandwidth. You need to run the tar program on the file server to avoid using bandwidth.