How to get the name of all the folders and subfolders of an Azure container - azure-blob-storage

I have this container Ofertas with 3 folders
within each of the 3 main folders there are hundreds of subfolders and within each of these subfolders are the files or blobs
I need to get the list of these subfolders but I only know how to get the principal folder names
string connectionString = _configuration.GetConnectionString("ArchivadoHistoricoOfertas");
BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient("ofertas");
var blobFolders = containerClient.GetBlobsByHierarchy(BlobTraits.None, BlobStates.None, "/").Where(b => b.IsPrefix).ToList();
blobFolders.ForEach(f => {
Console.WriteLine("Folder Prefix: " + f.Prefix);
});
I have tried this
But here the code stops and nothing happens
Any idea, please?
Thanks

I tried in my environment and got below results:
If you need to get all folder, subfolder and files in structure you can use below code.
Code:
class Program
{
static async Task Main(string[] args)
{
// Get a connection string to our Azure Storage account.
string connectionString = "<connect strng>";
string containerName = "test";
Console.WriteLine($"Getting blobs and virtual directories for container '{containerName}'");
BlobContainerClient container = new BlobContainerClient(connectionString, containerName);
await ListBlobsForPrefixRecursive(container,"", 0);
}
public static async Task ListBlobsForPrefixRecursive(BlobContainerClient container, string prefix, int level)
{
string spaces = new string(' ', level);
Console.WriteLine($"{spaces}- {prefix}");
await foreach (Page<BlobHierarchyItem> page in container.GetBlobsByHierarchyAsync(prefix: prefix, delimiter: "/").AsPages())
{
foreach (var blob in page.Values.Where(item => item.IsBlob).Select(item => item.Blob))
{
Console.WriteLine($"{spaces} {blob.Name}");
}
var prefixes = page.Values.Where(item => item.IsPrefix).Select(item => item.Prefix);
foreach (var s in prefixes)
{
await ListBlobsForPrefixRecursive(container, s, level + 1);
}
}
}
}
Output:
If you need to get the particular root folder to get subfolder you use the below code:
Code:
using Azure.Storage.Blobs;
namespace storage326
{
class Program
{
static void Main(string[] args)
{
string connectionString = "< storage account>";
string rootfolder = "demo";
List<string> subfolder = new List<string>();
BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
string containerName = "test";
BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient(containerName);
var blobs = containerClient.GetBlobs(prefix: rootfolder);
foreach (var blob in blobs)
{
//Console.WriteLine(blob.Name);
string[] sub_names = blob.Name.Split('/');
if (sub_names.Length > 2 && !subfolder.Contains(sub_names[1]))
{
subfolder.Add(sub_names[1]);
}
}
foreach (var s in subfolder)
{
Console.WriteLine(rootfolder+"/"+ s);
}
}
}
}
Console:

Related

C# :Move blobs from one container to another using Azure.Storage.Blobs client library

I am using the recent Azure.Storage.Blobs client library . I have seen many examples for copy and delete in CloudblockBlob Client using the StartCopyAsync() method but for the newer version I am not able to find anything .
I need to move files from one container to another in the same storage account .
This is the old version
CloudBlobClient blobClient = StorageAccount.CreateCloudBlobClient();
CloudBlobContainer sourceContainer = blobClient.GetContainerReference(SourceContainer);
CloudBlobContainer targetContainer = blobClient.GetContainerReference(TargetContainer);
CloudBlockBlob sourceBlob = sourceContainer.GetBlockBlobReference(fileToMove);
CloudBlockBlob targetBlob = targetContainer.GetBlockBlobReference(newFileName);
await targetBlob.StartCopyAsync(sourceBlob);
Please try the following code. It makes use of Azure.Storage.Blobs (12.9.1).
using System;
using System.Threading.Tasks;
using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Models;
namespace SO68668289
{
class Program
{
private const string connectionString = "DefaultEndpointsProtocol=https;AccountName=account-name;AccountKey=account-key==";
private const string sourceContainer = "source";
private const string targetContainer = "target";
private const string blobName = "blob-name.txt";
static async Task Main(string[] args)
{
BlobServiceClient serviceClient = new BlobServiceClient(connectionString);
BlobContainerClient sourceContainerClient = serviceClient.GetBlobContainerClient(sourceContainer);
BlobContainerClient targetContainerClient = serviceClient.GetBlobContainerClient(targetContainer);
BlobClient sourceBlobClient = sourceContainerClient.GetBlobClient(blobName);
BlobClient targetBlobClient = targetContainerClient.GetBlobClient(blobName);
Console.WriteLine("Sending copy blob request....");
var result = await targetBlobClient.StartCopyFromUriAsync(sourceBlobClient.Uri);
Console.WriteLine("Copy blob request sent....");
Console.WriteLine("============");
bool isBlobCopiedSuccessfully = false;
do
{
Console.WriteLine("Checking copy status....");
var targetBlobProperties = await targetBlobClient.GetPropertiesAsync();
Console.WriteLine($"Current copy status = {targetBlobProperties.Value.CopyStatus}");
if (targetBlobProperties.Value.CopyStatus == CopyStatus.Pending)
{
System.Threading.Thread.Sleep(1000);
}
else
{
isBlobCopiedSuccessfully = targetBlobProperties.Value.CopyStatus == CopyStatus.Success;
break;
}
} while (true);
if (isBlobCopiedSuccessfully)
{
Console.WriteLine("Blob copied successfully. Now deleting source blob...");
await sourceBlobClient.DeleteAsync();
}
}
}
}

C# Azure.Storage.Blobs SDK How to list and zip all files in a container and store the zip in another container

I have around 100 files in a container . I want to call a function to zip all files and send it to another container . I am using the Azure.Storage.Blobs version 12.9.1
var blob = container.GetBlockBlobReference(outputFilename);
using (var stream = await blob.OpenWriteAsync())
using (var zip = new ZipArchive(stream, ZipArchiveMode.Create))
{
for (int i = 0; i < 2000; i++)
{
using (var randomStream = CreateRandomStream(2))
{
var entry = zip.CreateEntry($"{i}.zip", CompressionLevel.Optimal);
using (var innerFile = entry.Open())
{
await randomStream.CopyToAsync(innerFile);
}
}
}
}
This is for the older version . How can we achieve it using the new sdk
Can you please try with this code.
Solution 1: Try this code to generate Zip file. After generating zip file upload it to container
public void ZipFilesToResponse(HttpResponseBase response, IEnumerable<Asset> files, string zipFileName)
{
using (var zipOutputStream = new ZipOutputStream(response.OutputStream))
{
zipOutputStream.SetLevel(0); // 0 - store only to 9 - means best compression
response.BufferOutput = false;
response.AddHeader("Content-Disposition", "attachment; filename=" + zipFileName);
response.ContentType = "application/octet-stream";
foreach (var file in files)
{
var entry = new ZipEntry(file.FilenameSlug())
{
DateTime = DateTime.Now,
Size = file.Filesize
};
zipOutputStream.PutNextEntry(entry);
storageService.ReadToStream(file, zipOutputStream);
response.Flush();
if (!response.IsClientConnected)
{
break;
}
}
zipOutputStream.Finish();
zipOutputStream.Close();
}
response.End();
}
For more details refer this link
Solution 2:Also try with this code
using System;
using System.Threading.Tasks;
using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Models;
using System.IO;
using ICSharpCode.SharpZipLib.Zip;
namespace SO68566758
{
class Program
{
private const string connectionString = "DefaultEndpointsProtocol=https;AccountName=account-name;AccountKey=account-key";
private const string sourceContainer = "source-container";
private const string targetContainer = "target-container";
private const string outputBlobName = "backup.zip";
static async Task Main(string[] args)
{
BlobServiceClient serviceClient = new BlobServiceClient(connectionString);
BlobContainerClient sourceContainerClient = serviceClient.GetBlobContainerClient(sourceContainer);
BlobContainerClient targetContainerClient = serviceClient.GetBlobContainerClient(targetContainer);
var blobs = sourceContainerClient.GetBlobsAsync();
using (var fs = new FileStream(outputBlobName, FileMode.OpenOrCreate))
{
using (var zipOutputStream = new ZipOutputStream(fs))
{
await foreach (var blob in blobs)
{
var blobName = blob.Name;
var blobClient = sourceContainerClient.GetBlobClient(blobName);
//var downloadResponse = await blobClient.DownloadAsync();
//var streamContent = downloadResponse.Value.Content;
var entry = new ZipEntry(blobName);
zipOutputStream.PutNextEntry(entry);
await blobClient.DownloadToAsync(zipOutputStream);
}
}
}
BlobClient targetBlob = targetContainerClient.GetBlobClient(outputBlobName);
using (FileStream fs = new FileStream(outputBlobName, FileMode.Open))
{
await targetBlob.UploadAsync(fs);
}
}
}
}

Programmatically cloning Octopus Deploy Process steps and modifyng the cloned steps

We are developing a Pipeline for which we have to add over 100 steps and modify two things for each step: Step Name and PackageID.
Rather than going through the pain of doing this via the UI, we’d like to do this programmatically.
Below is some C# I’ve sketched out for this (I’m a C# developer with extremely limited PowerShell skills, that’s why I did this in C#).
The lines above the comment “From here on is where I'm fuzzy” are working code, but the lines below the comment are just pseudocode.
Can someone explain to me how to write the lines below the comment (or, the PowerShell equivalent)?
I wasn’t able to find API calls for this.
Thanks
namespace ODClientExample
{
class Program
{
static void Main(string[] args)
{
List<string> ListOfWindowsServices = new List<string>();
ListOfWindowsServices.Add("svc1");
ListOfWindowsServices.Add("svc2");
ListOfFWindowsServices.Add("svc3");
var server = "https://mysite.whatever/";
var apiKey = "API-xxxxxxxxxxxxxxxxxx"; // I generated this via the Octopus UI
var endpoint = new OctopusServerEndpoint(server, apiKey);
var repository = new OctopusRepository(endpoint);
var project = repository.Projects.FindByName("Windows Services");
// From here on is where I'm fuzzy:
//
var procesSteps = GetProcessSteps(project);
var processStepToClone = GetProcesStepByName(processSteps, "SomeProcessStep");
foreach (string svcName in ListofSvcNames)
{
processStepToClone.StepName = svcName;
processStepToClone.PackageID = svcName;
}
}
}
}
I've made a little more progress. I'm now able to access the Steps in the Process, and add a Step. However, when my code calls repository.DeploymentProcesses.Modify, I get this exception:
Please provide a value for the package ID.
Please select the feed that this package will be downloaded from.
Please select one or more roles that 'svc1' step will apply to.
Here's my latest code:
static void Main(string[] args)
{
List<string> ListOfFexWindowsServices = new List<string>();
ListOfFexWindowsServices.Add("svc2");
ListOfFexWindowsServices.Add("svc3");
ListOfFexWindowsServices.Add("svc4");
string server = "https://mysite.stuff/";
string apiKey = "API-xxxxxxxxxxxxxxxxxxxxxxxx"; // I generated this via the Octopus UI
OctopusServerEndpoint endpoint = new OctopusServerEndpoint(server, apiKey);
OctopusRepository repository = new OctopusRepository(endpoint);
ProjectResource projectResource = repository.Projects.FindByName("MyProject");
DeploymentProcessResource deploymentProcess = repository.DeploymentProcesses.Get(projectResource.DeploymentProcessId);
var projectSteps = deploymentProcess.Steps;
DeploymentStepResource stepToClone = new DeploymentStepResource();
foreach (DeploymentStepResource step in projectSteps)
{
if (step.Name == "svc1")
{
stepToClone = step;
break;
}
}
foreach (string serviceName in ListOfFexWindowsServices)
{
DeploymentStepResource newStep = new DeploymentStepResource();
PopulateNewStep(newStep, stepToClone, serviceName);
deploymentProcess.Steps.Add(newStep);
repository.DeploymentProcesses.Modify(deploymentProcess);
}
}
static void PopulateNewStep(DeploymentStepResource newStep, DeploymentStepResource stepToClone, string serviceName)
{
newStep.Name = serviceName;
newStep.Id = Guid.NewGuid().ToString();
newStep.StartTrigger = stepToClone.StartTrigger;
newStep.Condition = stepToClone.Condition;
DeploymentActionResource action = new DeploymentActionResource
{
Name = newStep.Name,
ActionType = "Octopus.TentaclePackage",
Id = Guid.NewGuid().ToString(),
};
PopulateActionProperties(action);
newStep.Actions.Add(action);
// ISSUE: Anything else to do (eg, any other things from stepToClone to copy, or other stuff to create)?
newStep.PackageRequirement = stepToClone.PackageRequirement;
}
static void PopulateActionProperties(DeploymentActionResource action)
{
action.Properties.Add(new KeyValuePair<string, PropertyValueResource>("Octopus.Action.WindowsService.CustomAccountPassword", "#{WindowsService.Password}"));
// TODO: Repeat this sort of thing for each Action Property you see in stepToClone.
}
void Main()
{
var sourceProjectName = "<source project name>";
var targetProjectName = "<target project name>";
var stepToCopyName = "<step name to copy>";
var repo = GetOctopusRepository();
var sourceProject = repo.Projects.FindByName(sourceProjectName);
var targetProject = repo.Projects.FindByName(targetProjectName);
if (sourceProject != null && targetProject != null)
{
var sourceDeploymentProcess = repo.DeploymentProcesses.Get(sourceProject.DeploymentProcessId);
var targetDeploymentProcess = repo.DeploymentProcesses.Get(targetProject.DeploymentProcessId);
if (sourceDeploymentProcess != null && targetDeploymentProcess != null)
{
Console.WriteLine($"Start copy from project '{sourceProjectName}' to project '{targetProjectName}'");
CopyStepToTarget(sourceDeploymentProcess, targetDeploymentProcess, stepToCopyName);
// Update or add the target deployment process
repo.DeploymentProcesses.Modify(targetDeploymentProcess);
Console.WriteLine($"End copy from project '{sourceProjectName}' to project '{targetProjectName}'");
}
}
}
private OctopusRepository GetOctopusRepository()
{
var octopusServer = Environment.GetEnvironmentVariable("OCTOPUS_CLI_SERVER");
var octopusApiKey = Environment.GetEnvironmentVariable("OCTOPUS_CLI_API_KEY");
var endPoint = new OctopusServerEndpoint(octopusServer, octopusApiKey);
return new OctopusRepository(endPoint);
}
private void CopyStepToTarget(DeploymentProcessResource sourceProcess, DeploymentProcessResource targetProcess, string sourceStepName, bool includeChannels = false, bool includeEnvironments = false)
{
var sourceStep = sourceProcess.FindStep(sourceStepName);
if (sourceStep == null)
{
Console.WriteLine($"{sourceStepName} not found in {sourceProcess.ProjectId}");
return;
}
Console.WriteLine($"-> copy step '{sourceStep.Name}'");
var stepToAdd = targetProcess.AddOrUpdateStep(sourceStep.Name);
stepToAdd.RequirePackagesToBeAcquired(sourceStep.RequiresPackagesToBeAcquired);
stepToAdd.WithCondition(sourceStep.Condition);
stepToAdd.WithStartTrigger(sourceStep.StartTrigger);
foreach (var property in sourceStep.Properties)
{
if (stepToAdd.Properties.ContainsKey(property.Key))
{
stepToAdd.Properties[property.Key] = property.Value;
}
else
{
stepToAdd.Properties.Add(property.Key, property.Value);
}
}
foreach (var sourceAction in sourceStep.Actions)
{
Console.WriteLine($"-> copy action '{sourceAction.Name}'");
var targetAction = stepToAdd.AddOrUpdateAction(sourceAction.Name);
targetAction.ActionType = sourceAction.ActionType;
targetAction.IsDisabled = sourceAction.IsDisabled;
if (includeChannels)
{
foreach (var sourceChannel in sourceAction.Channels)
{
targetAction.Channels.Add(sourceChannel);
}
}
if (includeEnvironments)
{
foreach (var sourceEnvironment in sourceAction.Environments)
{
targetAction.Environments.Add(sourceEnvironment);
}
}
foreach (var actionProperty in sourceAction.Properties)
{
if (targetAction.Properties.ContainsKey(actionProperty.Key))
{
targetAction.Properties[actionProperty.Key] = actionProperty.Value;
}
else
{
targetAction.Properties.Add(actionProperty.Key, actionProperty.Value);
}
}
}
}
The above code sample is available in the Octopus Client Api Samples

Overwrite filename in SetAttributesCallback in Azure Storage Data Movement Library

I need to lowercase all filenames during UploadDirectoryAsync - is this possible to control or set via 'SetAttributesCallback'??
I cannot control the local physical files or rename them locally before uploading them to azure via Azure Storage Data Movement Library.
The end result will be that source and destination filename always will be with lowercase.
Any solution out there??
I need to lowercase all filenames during UploadDirectoryAsync - is this possible to control or set via 'SetAttributesCallback'
Yes, we could do that in the SetAttributesCallback, currently there is no rename Azure blob API, so we could upload the required renamed file in the SetAttributesCallback and delete the UploadDirectoryAsync load file. I also test it on my side, it works correctly.
The following is my demo code.
using System;
using System.IO;
using System.Linq;
using System.Threading;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;
using Microsoft.WindowsAzure.Storage.DataMovement;
namespace DataMovementTest
{
class Program
{
static void Main(string[] args)
{
string storageConnectionString = "storage connection string";
CloudStorageAccount account = CloudStorageAccount.Parse(storageConnectionString);
CloudBlobClient blobClient = account.CreateCloudBlobClient();
CloudBlobContainer blobContainer = blobClient.GetContainerReference("container name");
blobContainer.CreateIfNotExists();
var destDir = blobContainer.GetDirectoryReference("directory name");
string sourceDirPath = #"local path";
var options = new UploadDirectoryOptions
{
Recursive = false,
BlobType = BlobType.BlockBlob
};
using (MemoryStream journalStream = new MemoryStream())
{
// Store the transfer context in a streamed journal.
DirectoryTransferContext context = new DirectoryTransferContext(journalStream)
{
SetAttributesCallback = (destination) =>
{
CloudBlob destBlob = destination as CloudBlob;
if (System.Text.RegularExpressions.Regex.IsMatch(destBlob.Uri.Segments.Last(), "[A-Z]")) //check whether blobName contains uppercase
{
var path = sourceDirPath + $"/{destBlob.Uri.Segments.Last()}";
Console.WriteLine(path);
var renameBlob = destDir.GetBlockBlobReference(destBlob.Uri.Segments.Last().ToLower());
using (var fileStream = File.OpenRead(path))
{
renameBlob.UploadFromStream(fileStream);
}
destBlob.DeleteIfExists();
}
},
ShouldTransferCallback = (source, destination) => true
};
CancellationTokenSource cancellationSource = new CancellationTokenSource();
try
{
// Start the upload
var uploadResult = TransferManager.UploadDirectoryAsync(sourceDirPath, destDir, options, context, cancellationSource.Token).Result;
}
catch (Exception e)
{
Console.WriteLine("The transfer is cancelled: {0}", e.Message);
}
Console.WriteLine("Files in directory {0} uploading to {1} is finished.", sourceDirPath, destDir.Uri.ToString());
}
}
}
}

How to select only string records with contains conditions using Linq

I'm trying iterate a sting[] and get only records which respects the condition "Contains".
I've been trying this:
var lines = File.ReadAllLines("C:\\temp\\" + ReportPushFile("fileToSend_1.txt")).Where(i => i.Contains("BILL"));
But this returns me all records again without no filtering. What's wrong?
class Program : Logger
{
public static FTPclient Ftp = new FTPclient("ftp://smsftp.mobyt.it/", "C17053_004", "i7z0dx5b");
static void Main(string[] args)
{
List<Sms> sms = new List<Sms>();
sms.Add(new Sms() { recipient = "+393664528452" });
var lines = File.ReadAllLines("C:\\temp\\" + ReportPushFile("fileToSend_1.txt")).Where(i => i.StartsWith("BILL"));
// ...
}
public static string ReportPushFile(string NomeFile)
{
try
{
String LocalFileName = Ftp.ListDirectory("/reports/").Where(f => f.Contains(NomeFile)).OrderBy(x => new FileInfo(x).CreationTime).Max();
Ftp.Download("/reports/" + LocalFileName, "c:\\temp\\" + LocalFileName, true);
return LocalFileName;
}
catch(Exception ex)
{
return string.Empty;
}
}
}
Your code should work. Let's split it up and make the types explicit:
string path = "C:\\temp\\" + ReportPushFile("fileToSend_1.txt");
string[] readLines = File.ReadAllLines(path);
IEnumerable<string> lines = readLines.Where(i => i.Contains("BILL"));
You can now set some debug points and verify that:
readLines contains a list of lines.
lines contains only those lines that have BILL in them.

Resources