I created an Azure Storage account. I have a 400 megabytes .zip file that I want to put into blob storage for later use.
How can I do that without writing code? Is there some interface for that?
Free tools:
Visual Studio 2010 -- install Azure tools and you can find the blobs in the Server Explorer
Cloud Berry Lab's CloudBerry Explorer for Azure Blob Storage
ClumpsyLeaf CloudXplorer
Azure Storage Explorer from CodePlex (try version 4 beta)
There was an old program called Azure Blob Explorer or something that no longer works with the new Azure SDK.
Out of these, I personally like CloudBerry Explorer the best.
The easiest way is to use Azure Storage PowerShell. It provided many commands to manage your storage container/blob/table/queue.
For your mentioned case, you could use Set-AzureStorageBlobContent which could upload a local file into azure storage as a block blob or page blob.
Set-AzureStorageBlobContent -Container containerName -File .\filename -Blob blobname
For details, please refer to http://msdn.microsoft.com/en-us/library/dn408487.aspx.
If you're looking for a tool to do so, may I suggest that you take a look at our tool Cloud Storage Studio (http://www.cerebrata.com/Products/CloudStorageStudio). It's a commercial tool for managing Windows Azure Storage and Hosted Service. You can also find a comprehensive list of Windows Azure Storage Management tools here: http://blogs.msdn.com/b/windowsazurestorage/archive/2010/04/17/windows-azure-storage-explorers.aspx
Hope this helps.
The StorageClient has this built into it. No need to write really anything:
var account = new CloudStorageAccount(creds, false);
var client = account.CreateCloudBlobClient();
var blob = client.GetBlobReference("/somecontainer/hugefile.zip");
//1MB seems to be a pretty good all purpose size
client.WriteBlockSizeInBytes = 1024;
//this sets # of parallel uploads for blocks
client.ParallelOperationThreadCount = 4; //normally set to one per CPU core
//this will break blobs up automatically after this size
client.SingleBlobUploadThresholdInBytes = 4096;
blob.UploadFile("somehugefile.zip");
I use Cyberduck to manage my blob storage.
It is free and very easy to use. It works with other cloud storage solutions as well.
I recently found this one as well: CloudXplorer
Hope it helps.
There is a new OpenSource tool provided by Microsoft :
Project Deco - Crossplatform Microsoft Azure Storage Account Explorer.
Please, check those links:
Download binaries: http://storageexplorer.com/
Source Code: https://github.com/Azure/deco
You can use Cloud Combine for reliable and quick file upload to Azure blob storage.
A simple batch file using Microsoft's AzCopy utility will do the trick. You can drag-and-drop your files on the following batch file to upload into your blob storage container:
upload.bat
#ECHO OFF
SET BLOB_URL=https://<<<account name>>>.blob.core.windows.net/<<<container name>>>
SET BLOB_KEY=<<<your access key>>>
:AGAIN
IF "%~1" == "" GOTO DONE
AzCopy /Source:"%~d1%~p1" /Dest:%BLOB_URL% /DestKey:%BLOB_KEY% /Pattern:"%~n1%~x1" /destType:blob
SHIFT
GOTO AGAIN
:DONE
PAUSE
Note that the above technique only uploads one or more files individually (since the Pattern flag is specified) instead of uploading an entire directory.
You can upload files to Azure Storage Account Blob using Command Prompt.
Install Microsoft Azure Storage tools.
And then Upload it to your account blob will CLI command:
AzCopy /Source:"filepath" /Dest:bloburl /DestKey:accesskey /destType:blob
Hope it Helps.. :)
You can upload large files directly to the Azure Blob Storage directly using the HTTP PUT verb, the biggest file I have tried with the code below is 4,6 Gb. You can do this in C# like this:
// write up to ChunkSize of data to the web request
void WriteToStreamCallback(IAsyncResult asynchronousResult)
{
var webRequest = (HttpWebRequest)asynchronousResult.AsyncState;
var requestStream = webRequest.EndGetRequestStream(asynchronousResult);
var buffer = new Byte[4096];
int bytesRead;
var tempTotal = 0;
File.FileStream.Position = DataSent;
while ((bytesRead = File.FileStream.Read(buffer, 0, buffer.Length)) != 0
&& tempTotal + bytesRead < CHUNK_SIZE
&& !File.IsDeleted
&& File.State != Constants.FileStates.Error)
{
requestStream.Write(buffer, 0, bytesRead);
requestStream.Flush();
DataSent += bytesRead;
tempTotal += bytesRead;
File.UiDispatcher.BeginInvoke(OnProgressChanged);
}
requestStream.Close();
if (!AbortRequested) webRequest.BeginGetResponse(ReadHttpResponseCallback, webRequest);
}
void StartUpload()
{
var uriBuilder = new UriBuilder(UploadUrl);
if (UseBlocks)
{
// encode the block name and add it to the query string
CurrentBlockId = Convert.ToBase64String(Encoding.UTF8.GetBytes(Guid.NewGuid().ToString()));
uriBuilder.Query = uriBuilder.Query.TrimStart('?') + string.Format("&comp=block&blockid={0}", CurrentBlockId);
}
// with or without using blocks, we'll make a PUT request with the data
var webRequest = (HttpWebRequest)WebRequestCreator.ClientHttp.Create(uriBuilder.Uri);
webRequest.Method = "PUT";
webRequest.BeginGetRequestStream(WriteToStreamCallback, webRequest);
}
The UploadUrl is generated by Azure itself and contains a Shared Access Signature, this SAS URL says where the blob is to be uploaded to and how long time the security access (write access in your case) is given. You can generate a SAS URL like this:
readonly CloudBlobClient BlobClient;
readonly CloudBlobContainer BlobContainer;
public UploadService()
{
// Setup the connection to Windows Azure Storage
var storageAccount = CloudStorageAccount.FromConfigurationSetting("DataConnectionString");
BlobClient = storageAccount.CreateCloudBlobClient();
// Get and create the container
BlobContainer = BlobClient.GetContainerReference("publicfiles");
}
string JsonSerializeData(string url)
{
var serializer = new DataContractJsonSerializer(url.GetType());
var memoryStream = new MemoryStream();
serializer.WriteObject(memoryStream, url);
return Encoding.Default.GetString(memoryStream.ToArray());
}
public string GetUploadUrl()
{
var sasWithIdentifier = BlobContainer.GetSharedAccessSignature(new SharedAccessPolicy
{
Permissions = SharedAccessPermissions.Write,
SharedAccessExpiryTime =
DateTime.UtcNow.AddMinutes(60)
});
return JsonSerializeData(BlobContainer.Uri.AbsoluteUri + "/" + Guid.NewGuid() + sasWithIdentifier);
}
I also have a thread on the subject where you can find more information here How to upload huge files to the Azure blob from a web page
The new Azure Portal has an 'Editor' menu option in preview when in the container view. Allows you to upload a file directly to the container from the Portal UI
I've used all the tools mentioned in post, and all work moderately well with block blobs. My favorite however is BlobTransferUtility
By default BlobTransferUtility only does block blobs. However changing just 2 lines of code and you can upload page blobs as well. If you, like me, need to upload a virtual machine image it needs to be a page blob.
(for the difference please see this MSDN article.)
To upload page blobs just change lines 53 and 62 of BlobTransferHelper.cs from
new Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob
to
new Microsoft.WindowsAzure.Storage.Blob.CloudPageBlob
The only other thing to know about this app is to uncheck HELP when you first run the program to see the actual UI.
Check out this post Uploading to Azure Storage where it is explained how to easily upload any file via PowerShell to Azure Blob Storage.
You can use Azcopy tool to upload the required files to the azure default storage is block blob u can change pattern according to your requirement
Syntax
AzCopy /Source : /Destination /s
Try the Blob Service API
http://msdn.microsoft.com/en-us/library/dd135733.aspx
However, 400mb is a large file and I am not sure a single API call will deal with something of this size, you may need to split it and reconstruct using custom code.
Related
I want to create an app that supports multiple users. Every user can upload multiple files and each file uploaded needs to be processed by a pipeline (for example processing1.exe -> processing2.py -> processing3.exe). The results must be made available to the user after processing.
The backend is in Laravel.
I have the following questions:
I tried running the processing1.exe binary on Azure Functions but I got "access denied". I guess this is because the default Windows image does not have the necessary dependencies installed (Windows SDK). From what I read there are other offerings of "serverless" in Azure like Logic Apps, Custom containers etc. Is there a way to use "serverless"/Azure Functions with that binary? What are my options?
Everytime a new file upload is detected (by Laravel), how should I trigger Azure to start processing it? Should I Azure Storage Queues, Azure Service Bus, Event Hub or Event Grid (or something else)?
Ideally the system shouldn't process all the files from users in a FIFO manner but instead use round-robin (that way if user1 uploads 1000 files and user2 uploads 3 files, user2 wouldn't have to wait for all of user1 files to finish).
Few of the workarounds for running the .exe file on Azure Functions:
Workaround 1:
There is one of my practical workarounds on running and getting .exe output in Azure Functions, please refer this SO Thread.
Workaround 2:
My .exe file contains SQL database query that insert record into database using Azure Function App:
cmd.CommandText = "insert into [dbo].[debug]([Name]) values('test')";
run.csx:
using System;
public static void Run(TimerInfo myTimer, TraceWriter log)
{
System.Diagnostics.Process process = new System.Diagnostics.Process();
process.StartInfo.FileName = #"D:\home\site\wwwroot\TimerTriggerclass1\demofunction.exe";
process.StartInfo.Arguments = "";
process.StartInfo.UseShellExecute = false;
process.StartInfo.RedirectStandardOutput = true;
process.StartInfo.RedirectStandardError = true;
process.Start();
string output = process.StandardOutput.ReadToEnd();
string err = process.StandardError.ReadToEnd();
process.WaitForExit();
log.Info($"C# Timer trigger function executed at: {DateTime.Now}");
}
Before running the Function App in Azure, we need to upload the .exe file in the Azure Functions App > Functions (TimerTriggerclass1) > Upload File option.
Workaround 3:
using System;
using System.Diagnostics;
using System.Threading;
public static void Run(TimerInfo myTimer, TraceWriter log)
{
System.Diagnostics.Process process = new System.Diagnostics.Process();
string WorkingDirectoryInfo =#"D:\home\site\wwwroot\TimerTriggerClass1";
string ExeLocation = #"D:\home\site\wwwroot\TimerTriggerClass1\MyApplication.exe";
Process proc = new Process();
ProcessStartInfo info = new ProcessStartInfo();
try
{
info.WorkingDirectory = WorkingDirectoryInfo;
info.FileName = ExeLocation;
info.Arguments = "";
info.WindowStyle = ProcessWindowStyle.Minimized;
info.UseShellExecute = false;
info.CreateNoWindow = true;
proc.StartInfo = info;
proc.Refresh();
proc.Start();
proc.WaitForInputIdle();
proc.WaitForExit();
}
catch
{
}
log.Info($"C# Timer trigger function executed at: {DateTime.Now}");
}
Everytime a new file upload is detected (by Laravel), how should I trigger Azure to start processing it? Should I Azure Storage Queues, Azure Service Bus, Event Hub or Event Grid (or something else)? Ideally the system shouldn't process all the files from users in a FIFO manner but instead use round-robin (that way if user1 uploads 1000 files and user2 uploads 3 files, user2 wouldn't have to wait for all of user1 files to finish).
For this process, the pure serverless approach like Azure Functions with Blob Storage Trigger or Azure Logic Apps with Blob Storage Trigger is better.
Whenever the new file is uploaded, then the Blob Storage will trigger the Azure Function to process the files immediately.
It will not make wait of the user 2 until the user1 finishes all files processing. User 2 can get the processed result at the same time when files are uploaded.
In Comparison, Logic Apps also does the same functionality whenever the files are added or modified but Logic Apps has built-in processing methods and connectors like after processing, share or save it to some other storages or mails whereas if you need customized functionality before/on/after processing the files uploaded on Azure Storage, Azure functions is the best approach to write the custom optimized code.
I am utilizing Syncfusion's PdfViewerControl and PdfLoadedDocument classes to generate thumbnail images of a PDF. However, once I moved the project to an Azure App Service, the PdfViewerControl is throwing an exception when being initialized. I am curious if it is attempting to use system memory and Azure is blocking this. Below is the method GenerateThumbnails I've created and the exception is being thrown when creating a new PdfViewerControl. If anyone has a work around for this or has experienced something similar when moving to Azure, any assistance would be greatly appreciated.
Along with that, if someone knows of another tool to create thumbnails from a PDF in this manner that'd be very helpful as well. Thanks!
Exception:
System.AccessViolationException: 'Attempted to read or write protected memory. This is often an indication that other memory is corrupt.'
Method:
public static List<Byte[]> GenerateThumbnails(Byte[] file)
{
Int32 resizedHeight;
Int32 resizedWidth;
List<Byte[]> thumbnails = new List<Byte[]>();
using (PdfViewerControl pdfViewerControl = new PdfViewerControl())
using (PdfLoadedDocument pdfLoadedDocument = new PdfLoadedDocument(file, true))
{
// The PDF Viewer Control must load the PDF from a PdfLoadedDocument, rather than directly from the filename because
// when loaded from the filename, it is not disposed correctly and causes a file lock.
pdfViewerControl.Load(pdfLoadedDocument);
for (Int32 i = 0; i < pdfViewerControl.PageCount; ++i)
{
using (Bitmap originalBitmap = pdfViewerControl.ExportAsImage(i))
{
if (pdfViewerControl.LoadedDocument.Pages[i].Size.Width > pdfViewerControl.LoadedDocument.Pages[i].Size.Height)
{
resizedHeight = (PdfUtility.TARGET_THUMBNAIL_WIDTH_LANDSCAPE * originalBitmap.Height) / originalBitmap.Width;
resizedWidth = PdfUtility.TARGET_THUMBNAIL_WIDTH_LANDSCAPE;
}
else
{
resizedHeight = PdfUtility.TARGET_THUMBNAIL_HEIGHT_PORTRAIT;
resizedWidth = (PdfUtility.TARGET_THUMBNAIL_HEIGHT_PORTRAIT * originalBitmap.Width) / originalBitmap.Height;
}
using (Bitmap resizedBitmap = new Bitmap(originalBitmap, new Size(resizedWidth, resizedHeight)))
using (MemoryStream memoryStream = new MemoryStream())
{
resizedBitmap.Save(memoryStream, ImageFormat.Jpeg);
thumbnails.Add(memoryStream.ToArray());
}
}
}
}
return thumbnails;
}
Update
Web App for Containers on Windows is now supported. This allows you to bring your own docker container that runs outside of the sandbox, so the restrictions described below won't affect your application.
There are restrictions in the sandbox that the app is running in that prevents certain API calls.
Here is a list of frameworks and scenarios that have been found to be
not be usable due to one or more of the restrictions above. It's
conceivable that some will be supported in the future as the sandbox
evolves.
PDF generators failing due to restriction mentioned above:
Syncfusion Siberix Spire.PDF The following PDF generators are
supported:
SQL Reporting framework: requires the site to run in Basic or higher
(note that this currently does not work in Functions apps in
Consumptions mode) EVOPDF: See
http://www.evopdf.com/azure-html-to-pdf-converter.aspx for vendor
solution Telerik reporting: requires the site to run in Basic or
higher. More info here Rotativa / wkhtmltopdf: requires the site to
run in Basic or higher. NReco PdfGenerator (wkhtmltopdf): requires
subscription plan Basic or higher Known issue for all PDF generators
based on wkhtmltopdf or phantomjs: custom fonts are not rendered
(system-installed font is used instead) because of sandbox GDI API
limitations that present even in VM-based Azure Apps plans (Basic or
higher).
Other scenarios that are not supported:
PhantomJS/Selenium: tries to connect to local address, and also uses
GDI+.
https://github.com/projectkudu/kudu/wiki/Azure-Web-App-sandbox
Im creating a web extension and porting from XUL. I used to be able to easily read files with
var dJsm = Components.utils.import("resource://gre/modules/Downloads.jsm").Downloads;
var tJsm = Components.utils.import("resource://gre/modules/Task.jsm").Task;
var fuJsm = Components.utils.import("resource://gre/modules/FileUtils.jsm").FileUtils;
var nsiPromptService = Components.classes["#mozilla.org/embedcomp/prompt-service;1"].getService(Components.interfaces.nsIPromptService);
....
NetUtil.asyncFetch(file, function(inputStream, status) {
if (!Components.isSuccessCode(status)) {
return;
}
var data = NetUtil.readInputStreamToString(inputStream, inputStream.available());
var data = window.btoa(data);
var encoded_data_to_send_via_xmlhttp = encodeURIComponent(data);
...
});
This above will be deprecated.
I can use the downloads.download() to know what was the last download but I can NOT read the file and then get the equivalent for encoded_data_to_send_via_xmlhttp
Also in Firefox 57 onwards, means that I have to try to fake a user action by a button click or something, or upload a file.
Access to file:// URLs or reading files without any explicit user input
isnt there an easy way to read the last downloaded file?
The WebExtension API won't allow extensions to read local files anymore. You could let the extension get CORS privilege and read the content directly from the URL via fetch() or XMLHttpRequest() as blob and store directly to IndexedDB or memory, then encode and send to server. This comes with many restrictions and limitations such as to which origin you can read from and so forth.
Also, this would add potentially many unneeded steps. If the purpose is, as it seem to be in the question at the moment, to share the downloaded file with a server, I would instead suggest that you obtain the last DownloadItem object, extract the URL (.url) from that object and send the URL back to server.
This way the server can load directly from that URL (and encode it on server if needed). The network load will be about the same (a little less actually since there is no Base64 encoding involved which adds 33% to the size), and much less load on the client. The server would read the data as a binary/byte data stream; about the same as if the data was sent directly from the extension.
To obtain the last downloaded file you would do the following from a privileged script:
browser.downloads.search({
limit: 1,
orderBy: ["-startTime"]
})
.then(getLastDownload);
function getLastDownload(downloads) {
if (downloads.length) {
var url = downloads[0].url;
// ... send url to the server and let server fetch the data from it directly
}
}
According to this support mozilla question.
(2) Local file security
Firefox limits access from pages on web servers to pages on local disk or UNC paths. [...]).
Which solution ?
Use local-filesystem-links firefox addon (not tested)
and/or
run a small local webserver on client side, supposing server was run with sufficient privileges, you may finally access any local content via http:// (but still cannot with file:///)
Is it possible to upload image or file to SkyDrive fom Metro Style App?
I have already found how to browse the file from SkyDrive. But I haven't found regarding uploading file to SkyDrive. If you reply me, it will be very thankful..
I don't think the file picker method works unless the user has the desktop app installed.
You should use a Sharing contract. If you add a data file (Storage Item) to share, then SkyDrive will be listed as a share target and the user gets a UI where they can choose where in their SkyDrive they want to save. This is how I implemented it in my app.
For more info...
http://msdn.microsoft.com/en-us/library/windows/apps/hh771179.aspx
You can use FileSavePicker to save files. This will of course give the user a chance to select where he wants to save to local documents folder or sky drive. The user is in control.
FileSavePicker savePicker = new FileSavePicker();
savePicker.SuggestedStartLocation = PickerLocationId.DocumentsLibrary;
savePicker.DefaultFileExtension = ".YourExtension";
savePicker.SuggestedFileName = "SampleFileName";
savePicker.FileTypeChoices[".YourExtension"] = new List<string>() { ".YourExtension"};
StorageFile file = await savePicker.PickSaveFileAsync();
if (file != null)
{
await FileIO.WriteTextAsync(file, "A bunch of text to save to the file");
}
Please note that in the sample code I am creating the content of the file in code. If you want the user to select an existing file from the computer then you will have to first use FileOpenPicker, get the file and then use FileSavePicker to save the contents of the selected file to the SkyDrive
Assuming that you are using XAML/JavaScript, the suggested solution is to use FilePicker.
The following link may help you.
http://msdn.microsoft.com/en-us/library/windows/apps/jj150595.aspx
Thanks Mamta Dalal and Dangling Neuron, but there is problem. But it looks like I can't use FileSavePicker. I have to upload file(documnet, photo) not only text file. I have to copy from one path to another. If I use FileSavePicker, I have to write every file content (text, png, pdf, etc) and can't copy. Currently I am using FolderPicker. But unfortunately, FolderPicker doesn't support SkyDrive.My Code is As follow:
>FolderPicker saveFolder = new FolderPicker();
>saveFolder.ViewMode = PickerViewMode.Thumbnail;
>saveFolder.SuggestedStartLocation = PickerLocationId.Desktop;
>saveFolder.FileTypeFilter.Add("*");
>StorageFolder storagefolderSave = await saveFolder.PickSingleFolderAsync();
>StorageFile storagefileSave = [Selected storagefile with file picker];
>await storagefileSave.CopyAsync(storagefolderSave,storagefileSave.Name,NameCollisionOption.ReplaceExisting);
It will be greate that if FolderPicker supports SkyDrive or can copy file using FileSavePicker.
I have Console application which submit messages to Azure queue. Now I am trying to migrate this application to Mobile, but met reference problem with CloudStorageAccount. It requires Windows dll version but mine is Mobile.
Do you guys have any idea how I can initialize CloudStorageAccount object alternative way?
public Initializator()
{
CloudStorageAccount.SetConfigurationSettingPublisher((configName, configSetter) =>
{
// for a console app, reading from App.config
configSetter(ConfigurationManager.ConnectionStrings[configName].ConnectionString);
});
CloudStorageAccount storageAccount = CloudStorageAccount.FromConfigurationSetting("QueueStorage");
CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();
queueIn = queueClient.GetQueueReference("queuein");
queueOut = queueClient.GetQueueReference("queueout");
queueIn.CreateIfNotExist();
queueOut.CreateIfNotExist();
}
One of the easiest ways, in my opinion, to work with Windows Azure storage (tables, blobs, & queues) from Windows Phone is to use the Phone.Storage NuGet package (http://www.nuget.org/packages/Phone.Storage). This makes working with storage on the phone nearly identical to working with storage from a server (or console app).
Be sure to check out Wade Wegner's blog post at http://www.wadewegner.com/2011/11/nuget-packages-for-windows-azure-and-windows-phone-developers/ for some additional info on the NuGet packages.
There's also a Phone.Storage.Sample package that may be worth taking a look at.