I've set up the pipeline and it works (I followed this documentation https://learn.microsoft.com/en-us/azure/connectors/connectors-create-api-ftp), it downloads the zip file and loads it to the blob storage.
however the resulted zip file is corrupted. it has a slightly different size than the original file.
I set the infer content type to YES. Also tried this setting to no but didn't change result.
I tried with hardcoded and dynamic naming.
Related
I would like to know if there is a way to read a file on a remote server using url? For example i have been shared a url to a file on a remote server, i need to read it using bash commands or any tool to retrieve data (eg: view first 50 rows and write to a file) without downloading the original file to local system.
The use case is to avoid downloading/uploading huge files located on the remote server to local systems instead access the file content using the url.
Any resources on this would help.
I want to make the file to zip . I can make the file to zip file but I don't want to create temporary zip file. Is any way to make the file to zip and to byte[] ?
Thanks
zip files are typically generated with streams anyway, so there's no need to temporarily store them in a file - might as well be in memory or streamed directly to a remote recipient (with only a small memory buffer to avoid a large memory footprint).
Ref Sample helperclass in the accepted answer section How can I generate zip file without saving to the disk with Java?
I have a Nifi flow where I am fetching files from S3. A pair of files are fetched through S3 and later passed into a MergeContent processor. Next, there is a README file that needs to go with each pair of files.
This README file is always same and I have stored it locally. I have a ExecuteStreamCommand that takes in content from the MergeContent processor.
I have tried passing in the README file into the MergeContent processor using the ListFile/FetchFile combination but its not working as expected. I guess the final result that I am looking for is a MergeContent package that contains a pair of files downloaded from S3 + the README file.
I think in this case you will want to use GetFile for the README -- the path is static, and you can set the Keep Source File setting to true in order to constantly retrieve the same content.
ListFile/FetchFile probably isn't working because once ListFile retrieves a filename from the directory, it stores the timestamp in its local state and won't retrieve files older than that on the next execution.
We are new to Windows azure and have used Windows azure storage for blob objects while developing sitefinity application but the blob files which are uploaded to this storage via publishing to azure from Visual Studio uploads files with only the file names and do not maintain the prefix folder name and slash. Hence we have to rename all files manually on the windows azure management portal and put the folder name and slash in the beginning of each file name so that the page which is accessing these images can show the images properly otherwise the images are not shown due to incorrect path.
Though in sitefinity admin panel , when we upload these images/blob files in those pages , we upload them inside a folder and we have configured to leverage sitefinity to use azure storage instead of database.
Please check the file attached to see the screenshot.
Please help me to solve this.
A few things I would like to mention first:
Windows Azure does not support rename functionality. Rename blob functionality = copy blob followed by delete blob.
Copy blob operation is asynchronous so you must wait for copy operation to finish before deleting the blob.
Blob storage does not support folder hierarchy natively. As you may have already discovered, you create an illusion of a folder by prepending a blob name (say logo.png) with the name of folder you want (say images) and separate them with slash (/) so your blob name becomes images/logo.png.
Now coming to your problem. Needless to say that manually renaming the blobs would be a cumbersome exercise. I would recommend using a storage management tool to do that. One such example would be Azure Management Studio from Cerebrata. If you use that tool, essentially what you can do is create an empty folder in the container and then move the files into that folder. That to me would be the fastest way to achieve your objective.
If you wish to write some code to do that, here are the steps you will take:
First you will list all blobs in a blob container.
Next you will loop over this list.
For each blob (let's call it source blob), you would get its name and prepend the folder name that you want and create an instance of a CloudBlockBlob object.
Next you would initiate a copy blob operation on that blob using StartCopyFromBlob on this new blob where source is your source blob.
You would need to wait for the copy operation to finish. Once the copy operation is finished, you can safely delete the source blob.
P.S. I would have written some code but unfortunately I'm stuck with something else. I might write something later on (but please don't hold your breath for that :)).
Can any1 provide me an Idea, How to implement unzipping of .gz format file through Worker. If i try to write unzipping of file then, where i need to store unzipped file(i.e one text file
) , Will it be loaded in any location in azure. how can i specify the path in Windows Azure Worker process like current execting directory. If this approach doesnot work, then i need to create one more blob to store unzipped .gz file i.e txt.
-mahens
In your Worker Role, it is up to you how a .gz file arrive (downloaded from Azure Blob storage) however on the file is available you can use GZipStream to compress/uncompress a .GZ file. You can also find code sample in above link with Compress and Decompress function.
This SO discussion shares a few tools and code to explain how you can unzip .GZ using C#:
Unzipping a .gz file using C#
Next when you will use Decompress/Compress code in a Worker Role you have ability to store it directly to local storage (as suggested by JcFx) or use MemoryStream to store directly to Azure Blob Storage.
The following SO article shows how you can use GZipStream to store unzipped content into MemoryStream and then use UploadFromStream() API to store directly to Azure Blob storage:
How do I use GZipStream with System.IO.MemoryStream?
If you don't have any action related to your unzipped file then storing directly to Azure Blob storage is best however if you have to do something with unzipped content you can save locally as well as storage to Azure Blob storage back for further usage.
This example, using SharpZipLib, extracts a .gzip file to a stream. From there, you could write it to Azure local storage, or to blob storage:
http://wiki.sharpdevelop.net/GZip-and-Tar-Samples.ashx