Write csv data directly to azure blob storage - azure-blob-storage

Question

I have a large array with around 1 million data objects. I have found lots of samples for uploading files to azure blob storage. I guess you could do it with a memory stream, but I haven't found samples for doing it from objects. I am not sure with that size of data whether you should write line by line of what options I have. All input are welcome, would be perfect with some samples. The goal is to write the data objects to a csv file in Azure Blob Storage.

Answer 1

I assume you are writing the code in C# with the latest version of Azure Storage SDK for .NET (9.3.3).
Here is my code for realizing your needs to write a large array of data objects directly to Azure Blob Storage.
using System;
using System.Collections;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;
namespace WriteCSVDataToBlob
{
class Record
{
string[] cols;
public Record(string[] cols)
{
this.cols= cols;
}
override public string ToString()
{
return String.Join(',', cols);
}
}
class Program
{
static void Main(string[] args)
{
var arr = new ArrayList();
arr.Add(new Record(new string[]{ "A", "B","one" }));
arr.Add(new Record(new string[] { "C", "D","two"}));
string storageConnectionString = "<your storage connection string>";
var storageAccount = CloudStorageAccount.Parse(storageConnectionString);
var client = storageAccount.CreateCloudBlobClient();
var container = client.GetContainerReference("test");
var blob = container.GetBlockBlobReference("data.txt");
using (CloudBlobStream x = blob.OpenWriteAsync().Result)
{
foreach(var rec in arr)
{
x.Write(System.Text.Encoding.Default.GetBytes(rec.ToString()+"\n"));
}
x.Flush();
x.Close();
}
}
}
}

Answer 2

#Peter Pan's solution works for Microsoft.Azure.Storage.Blob v11.1.0.
In newer version of azure blob storage you can use something like this:
using (var writer = new StreamWriter(new MemoryStream()))
using (var csv = new CsvWriter(writer, CultureInfo.InvariantCulture))
{
csv.Configuration.Delimiter = userInputModel.Separator;
csv.WriteRecords(arr);
writer.Flush();
writer.BaseStream.Seek(0, SeekOrigin.Begin);
client.AppendBlock(writer.BaseStream);
}