I want to make the file to zip . I can make the file to zip file but I don't want to create temporary zip file. Is any way to make the file to zip and to byte[] ?
Thanks
zip files are typically generated with streams anyway, so there's no need to temporarily store them in a file - might as well be in memory or streamed directly to a remote recipient (with only a small memory buffer to avoid a large memory footprint).
Ref Sample helperclass in the accepted answer section How can I generate zip file without saving to the disk with Java?
Related
I've set up the pipeline and it works (I followed this documentation https://learn.microsoft.com/en-us/azure/connectors/connectors-create-api-ftp), it downloads the zip file and loads it to the blob storage.
however the resulted zip file is corrupted. it has a slightly different size than the original file.
I set the infer content type to YES. Also tried this setting to no but didn't change result.
I tried with hardcoded and dynamic naming.
Hi I'm using nifi as an ETL tool.
Process IMG
This is my current process. I use TailFile to detect CSV file and then send messages to Kafka.
It works fine so far, but i want to delete CSV file after i send contents of csv to Kafka.
Is there any way?
Thanks
This depends on why you are using TailFile. From the docs,
"Tails" a file, or a list of files, ingesting data from the file as it is written to the file
TailFile is used to get new lines that are added to the same file, as they are written. If you need to a tail a file that is being written to, what condition determines it is no longer being written to?
However, if you are just consuming complete files from the local file system, then you could use GetFile which gives the option to delete the file after it is consumed.
From a remote file system, you could use ListSFTP and FetchSFTP which has a Completion Strategy to move or delete.
I have a Nifi flow where I am fetching files from S3. A pair of files are fetched through S3 and later passed into a MergeContent processor. Next, there is a README file that needs to go with each pair of files.
This README file is always same and I have stored it locally. I have a ExecuteStreamCommand that takes in content from the MergeContent processor.
I have tried passing in the README file into the MergeContent processor using the ListFile/FetchFile combination but its not working as expected. I guess the final result that I am looking for is a MergeContent package that contains a pair of files downloaded from S3 + the README file.
I think in this case you will want to use GetFile for the README -- the path is static, and you can set the Keep Source File setting to true in order to constantly retrieve the same content.
ListFile/FetchFile probably isn't working because once ListFile retrieves a filename from the directory, it stores the timestamp in its local state and won't retrieve files older than that on the next execution.
I know Spring has MultipartFile component.
I am wondering if there is any API to unzip files or read zip files to do some processing?
I have a zip file that following a certain format.
photos\
audio\
report.xml
when the user upload it via web, I wish to scan the zip file and do some processing.
Is there a solution for this issue?
I do not know spring have any such type of API,
but you can use other API for ZIP or UNZIP files.
1) http://commons.apache.org/compress/
2) java.util.zip
and also see
What is a good Java library to zip/unzip files?
There are a couple of Java SE APIs for reading ZIP files:
java.util.zip.ZipInputStream - gives you a one-pass reader
java.util.zip.ZipFile - gives you a reader that allows you to read the entries and the files in any order.
You should be able to use one or the other of these, depending on the nature of your processing.
If the processing requires the images to be in actual files, you would have to create the directories and write the files yourself. In this case, it would probably be simpler to use an external command to do the ZIP extraction.
Can any1 provide me an Idea, How to implement unzipping of .gz format file through Worker. If i try to write unzipping of file then, where i need to store unzipped file(i.e one text file
) , Will it be loaded in any location in azure. how can i specify the path in Windows Azure Worker process like current execting directory. If this approach doesnot work, then i need to create one more blob to store unzipped .gz file i.e txt.
-mahens
In your Worker Role, it is up to you how a .gz file arrive (downloaded from Azure Blob storage) however on the file is available you can use GZipStream to compress/uncompress a .GZ file. You can also find code sample in above link with Compress and Decompress function.
This SO discussion shares a few tools and code to explain how you can unzip .GZ using C#:
Unzipping a .gz file using C#
Next when you will use Decompress/Compress code in a Worker Role you have ability to store it directly to local storage (as suggested by JcFx) or use MemoryStream to store directly to Azure Blob Storage.
The following SO article shows how you can use GZipStream to store unzipped content into MemoryStream and then use UploadFromStream() API to store directly to Azure Blob storage:
How do I use GZipStream with System.IO.MemoryStream?
If you don't have any action related to your unzipped file then storing directly to Azure Blob storage is best however if you have to do something with unzipped content you can save locally as well as storage to Azure Blob storage back for further usage.
This example, using SharpZipLib, extracts a .gzip file to a stream. From there, you could write it to Azure local storage, or to blob storage:
http://wiki.sharpdevelop.net/GZip-and-Tar-Samples.ashx