Do you know best way to upload too many files to Azure Blob container?
I am currently do something to upload multiple files to Azure blob storage. The number of files may be huge, like 30,000 or more(each file could be sized of 10KB~1MB). Firstly, I have a list of files locations, then I would use Parallel.Foreach to upload the files. code snippet like this:
`List locations=...
Parallel.Foreach(locations, location=>
{
...
UploadFromStream(...);
...
});`
The codes run to inconsistence results.
Sometimes it runs well, I can see all files uploaded to the Azure blob container.
Sometimes, I will got exceptions like this:
Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature., Inner Exception: The remote server returned an error: (403) Forbidden
Sometimes, I got a timeout exception.
I have worked against the issue for several days, unfortunatly, I havn't got a perfect solution yet. So I want to know how do you do when you handling similar scenario, how do you do when upload too many files to Azure blob storage?
Finally, I have not found what's wrong with my code.
However, I have found solution for this issue. I expire Parallel.Foreach, just use common foreach. Then I use BeginuploadFromStream method instead of UploadFromStream, it actually upload files asynchronously.
So far, it runs prefectly, more stable, without any exception happens.
Related
I have Laravel backend with Intervention package to handle images.
All images are stored in S3 bucket and are accessed through CDN - Cloudfront distribution.
When user requests image from my app I use something like this line to fetch image and return it to user.
return Image::make("https://*********.cloudfront.net/S3_image_path_and_filename.jpg")->response()->header('Cache-Control', 'max_age=3600');
This code worked without any troubles for more than a year without any changes but recently I started to see that some images on client side are empty and there are errors saying
Intervention\Image\Exception\NotReadableException Unable to init from given url
I looked up for issues with missing images but they exist in S3 bucket and are accessible through Cloudfront. I even see them on client side after clearing cache or waiting for some time - so this issue is flacky, can't reproduce it. I even don't see any error associated with images in Cloudfront distribution.
My questions are
what could be the reasons behind this issue?
how I can debug it?
maybe I need to handle the exception above but I am not sure what should be returned to user in that case?
I am working on a solo project where admin users will be able to manage Virtual Machines via a web application run on laravel 7. I am currently creating the creation controller which requires uploading large files (.ovas). I am trying to use streams (I think) in a store method.
Specifically, a view has a form with a standard file inclusion that sends the request to the controller. The form gets validated and then I try to move the file into storage as such:
$data = $this->validator($request);
Storage::put('files', fopen($data['file'], 'r+'));
Where $data['file'] is the uploaded file. I can see the file being uploaded locally (while monitoring hdd usage), but after it is sent, laravel returns the following error
fopen(/var/www/html/devel/ocl/storage/app/files): failed to open stream: Is a directory
The error seems pretty obvious, so I was curious and decided to return the fopen() call before storage to see if I could invoke the same error by changing the method to:
$data = $this->validator($request);
return var_dump(fopen($data['file'], 'r+'));
However, no error is invoked and I get resource(8) of type (stream) as expected (I think). The tmp resource disappears and no errors are thrown. I am a bit baffled on why this is occurring as I would assume that function call resolves before the storage. I feel like I'm missing something fundamental with PHP/laravel and cannot think of how to continue to explore the issue.
Why does the first call to fopen() seem to see $data['file'] as a path, but the second call to just fopen() work as intended? I am hoping the answer can help remedy the issue.
Thanks!
I'm trying to create an app in InTune using the Microsoft Graph REST API. I'm able to create the app, the contentversion, upload a file to Azure Storage, and call the commit action. After that, I'm waiting for uploadState 'commitFileSuccess', but it returns 'commitFileFailed'.
I saw a similar question, but that's assuming the file encryption is wrong:
commitFileFailed during mobileAppContentFile Commit
However, I have no clue where the error lies. Is there anybody with experience on this particular subject?
If you need more info, please let me know.
I found out what I was doing wrong. I used the PowerShell sample from Github as a base to write my own Ruby version, but I overlooked one thing.
https://github.com/microsoftgraph/powershell-intune-samples/tree/master/LOB_Application
I was uploading my apk file at once to the azureStorageUri, but I needed to upload it in chunks. After doing that the uploadState got updated to commitFileSuccess.
I will share my Ruby script once I've cleaned it up!
I have written a Firefox Extension using Web Extension APIs. It has passed the Preliminary review but the reviewer said that he cannot proceed with the full review cause when he installs it, he gets the following error -
"Unable to parse JSON data for extension storage"
Upon inspecting for quite sometime, I figured that Firefox creates a file called "storage.js" in the profile folder for each extension where it writes and reads from, all the local storage data for that particular extension. And if the extension tries to write to this file before this file is created, the error "Unable to write JSON data to extension storage" is thrown and if the extension code tries to read from this file before this file is created, the error "Unable to parse JSON data for extension storage" is thrown.
Now, my concern is how do I know for sure that the file has been created and that it can be written to or read from?
PS : This happens when the extension is just installed. For consequent sessions, this error wont come as that file is no longer missing.
This seems to be a bug in the current Firefox implementation, and your assessment is spot on:
The underlying ExtStorage module will always call read before get, set etc. even write and clear.
read will unconditionally try to access the underlying, extension specific storage file, that may not exist yet for freshly installed add-ons using the storage API for the first time.
This will therefore result in the logging of one such Unable to parse JSON data for extension storage message, no matter what you do with the storage API.
Therefore triggering the message cannot be avoided.
I suggest you do the following:
Contact the editors team, requesting they re-evaluate your add-on based on:
The message in question is really only a warning (when appearing after first access of the storage API by your addon).
Even when the message would be an actual error (the storage is corrupt), it would still not be your error, as the storage API implementation by mozilla needs to be more resilient then and there is nothing you can do anyway.
The message being issued on first regular use of the storage API, unrelated to what WebExtensions add-on uses that API and in what way, is a mozilla bug, and not something you caused or can fix yourself or at least work around.
Therefore denying a full review just because a mozilla bug erroneously logs a spurious message once without any other severe effects is... questionable.
File a bug about this so mozilla developers can address this issue. You'll wanna CC at least Bill McCloskey (:billm) since he wrote that code ;)
I have an mp3 file in an S3 bucket. I am fetching this file via ajax GET request for html5 audio playback. Intermittently, the get request will fail to download the file and thus the track will not play. The request returns "206 partial content." Oddly, it will work several times before failing and then continuing to fail.
If I disable caching in my browser (chrome), the file will download and play appropriately.
Have I configured s3 incorrectly? How can I get this mp3 file to download and play consistently?
specific file is located here: https://s3.amazonaws.com/1m40s_dev/assets/music/walden.mp3
thanks!
I've found this often relates to the MIME type set on the S3 hosted file.
Setting the correct MIME type seems to fix things.
On a side note, I struggled with a single binary file always breaking in IE. Its MIME type was application/octet-stream. I changed the MIME to binary/octet-stream and that seemed to fix downloads from IE. Not sure why.
use amazon cloudfront solve the problem
I solved this by appending a timestamp to the end of the mp3 url on page load. This forced a new download of the content each time and eliminated the caching error.
This feels more like a work around than a fix. I still don't know the root cause of the issue but if you find yourself having a similar problem and just need to move on, add a timestamp or random number as a param at the end of the url
.../assets/music/walden.mp3?[timestamp]
One other workaround I've found is, if you're using rails, turning off turbolinks makes this go away on chrome. I'll add more to my answer as I discover more.