I have a python 3.8 application deployed on a kubernetes cluster on azure that has to access a blob storage container in an account in a different resource group. I'm using a managed identity to authenticate and query the container:
from azure.storage.blob import BlobServiceClient
creds = ManagedIdentityCredential()
url_template = task_config["ACCOUNT_ADDRESS_TEMPLATE"]
account_name = task_config["BLOB_STORAGE_ACCOUNT"]
account_url = url_template.replace("*", account_name)
blob_service_client = BlobServiceClient(account_url=account_url, credential=creds)
if container not in [c.name for c in blob_service_client.list_containers()]:
raise BlobStorageContainerDoesNotExistError(
f"Container {container} does not exist"
)
self.client: ContainerClient = blob_service_client.get_container_client(
container=container
I have verified that the managed identity has been assigned the Storage Blob Data Contributor role in the storage account, and also at the level of the resource group. I have verified that the token generated when instantiating the ManagedIdentityCredential() object references the right managed identity, and I have whitelisted the outbound IP (and every other possible IP just in case) of my python application. Nevertheless, I keep getting this error when attempting to list the containers in the account:
Http ResponseError(response=response, model=error)\nazure.core.exceptions.HttpResponseError: Operation returned an invalid status 'This request is not authorized to perform this operation.'
Could anyone point me in the right direction?
Specs:
azure-identity = "1.5"
azure-storage-blob= "12.8.1"
python = "3.8"
platform: linux docker containers running on kubernetes cluster deployed on azure.
I have tested in my environment
It seems you are using Storage Account to allow access from Selected Networks.
Please make sure to allow access from your AKS VMSS virtual network :
Then you can use the below python script to list the blob containers in the Storage Account :
from azure.storage.blob import BlobServiceClient
from azure.identity import ManagedIdentityCredential
creds = ManagedIdentityCredential ()
blob_service_client = BlobServiceClient(account_url="https://StorageAccountName.blob.core.windows.net/", credential=creds)
test = blob_service_client.list_containers()
for container in test :
print(container.name)
Related
I am trying to access files stored in Azure blob storage and have followed the documentation linked below:
https://docs.databricks.com/external-data/azure-storage.html
I was successful in mounting the Azure blob storage on dbfs but it seems that the method is not recommended anymore. So, I tried to set up direct access using URI using SAS authentication.
spark.conf.set("fs.azure.account.auth.type.<storage-account>.dfs.core.windows.net", "SAS")
spark.conf.set("fs.azure.sas.token.provider.type.<storage-account>.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider")
spark.conf.set("fs.azure.sas.fixed.token.<storage-account>.dfs.core.windows.net", "<token>")
Now when I try to access any file using:
spark.read.load("abfss://<container-name>#<storage-account-name>.dfs.core.windows.net/<path-to-data>")
I get the following error:
Operation failed: "Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.", 403, HEAD,
I am able to mount the storage account using the same SAS token but this is not working.
What needs to be changed for this to work?
If you are using blob storage, then you have to use wasbs and not abfss. I have tried using using the same code as yours with my SAS token and got the same error with my blob storage.
spark.conf.set("fs.azure.account.auth.type.<storage_account>.dfs.core.windows.net", "SAS")
spark.conf.set("fs.azure.sas.token.provider.type.<storage_account>.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider")
spark.conf.set("fs.azure.sas.fixed.token.<storage_account>.dfs.core.windows.net", "<token>")
df = spark.read.load("abfss://<container>#<storage_account>.dfs.core.windows.net/input/sample1.csv")
When I used the following modified code, I was able to successfully read the data.
spark.conf.set("fs.azure.account.auth.type.<storage_account>.blob.core.windows.net", "SAS")
spark.conf.set("fs.azure.sas.token.provider.type.<storage_account>.blob.core.windows.net", "org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider")
spark.conf.set("fs.azure.sas.fixed.token.<storage_account>.blob.core.windows.net", "<token>")
df = spark.read.format("csv").load("wasbs://<container>#<storage_account>.blob.core.windows.net/input/sample1.csv")
UPDATE:
To access files from azure blob storage where the firewall settings are only from selected networks, you need to configure VNet for the Databricks workspace.
Now add the same virtual network to your storage account as well.
I have also selected service endpoints and subnet delegation as following:
Now when I run the same code again using the file path as wasbs://<container>#<storage_account>.blob.core.windows.net/<path>, the file is read successfully.
We have a web api(.NET 5) which access some secrets from the Azure KeyVault.
In local machine for development, since I am the owner the new vault created, my email has access privilege to keyvault.
Hence I selected my account though VS -->Tools> Options-->Azure Service Authentication-->Account Selection--> "myemail#.com"
I have the below code to fetch secrets from Keyvault and access through configuration like we access the appsettings value.
public static IHostBuilder CreateHostBuilder(string[] args) =>
Host.CreateDefaultBuilder(args)
.ConfigureAppConfiguration((context, config) =>
{
var appSettings = config.Build();
var credentialOptions = new DefaultAzureCredentialOptions();
var credential = new DefaultAzureCredential(credentialOptions);
config.AddAzureKeyVault(new Uri(appSettings["Url:KeyVault"]), credential);
})
.ConfigureWebHostDefaults(webBuilder =>
{
webBuilder.UseStartup<Startup>();
});
}
We access the secret value like _configuration["secret"] in service and controller layer.
My queries are
1, If I move deploy this code to on premise server how it will work (dev env is on-premise server)?
2, If I deploy this web API to Azure, how to use identity AD App to access the key vault without any code change. We have AD app registered which has read access to this particular Vault.
I want the code to seamlessly work for local and Azure.
DefaultAzureCredential is the new and unified way to connect and retrieve tokens from Azure Active Directory and can be used along with resources that need them
The DefaultAzureCredential gets the token based on the environment the application is running
The following credential types if enabled will be tried, in order - EnvironmentCredential, ManagedIdentityCredential, SharedTokenCacheCredential, InteractiveBrowserCredential
IF I move deploy this code to on premise server how it will work (dev env is on-premises server)
When executing this in a development machine (on-premises server), you need to first configure the environment setting the variables AZURE_CLIENT_ID, AZURE_TENANT_ID and AZURE_CLIENT_SECRET to the appropriate values for your service principal (app registered in Azure AD)
If I deploy this web app to Azure, how to use identity AD App to access the key vault without any code change. We have AD app
registered which have read access to this Vault
You can enable System assigned Managed Identity for your web app. Add access policy for this identity in your Azure Key Vault to read the secrets. Now without making any changes in your code, your web app would be able to read the key vault secrets
I am getting this below access issue while trying to connect from databricks to gen2 data lake using Service principal and OAuth 2.0
Steps performed: Reference article
created new service principal
provide necessary access to this service principal from azure storage account IAM with Contributor role access.
Firewalls and private end points connection has been enabled on databricks and storage account.
StatusCode=403
StatusDescription=This request is not authorized to perform this operation using this permission.
ErrorCode=AuthorizationPermissionMismatch
ErrorMessage=This request is not authorized to perform this operation using this permission.
However when I tried connecting via access keys it works well without any issue. Now I started suspecting if #3 from my steps is the reason for this access issue. If so, do I need to give any additional access to make it success? Any thoughts?
When performing the steps in the Assign the application to a role, make sure to assign the Storage Blob Data Contributor role to the service principal.
Repro: I have provided owner permission to the service principal and tried to run the “dbutils.fs.ls("mnt/azure/")”, returned same error message as above.
Solution: Now assigned the Storage Blob Data Contributor role to the service principal.
Finally, able to get the output without any error message after assigning Storage Blob Data Contributor role to the service principal.
For more details, refer “Tutorial: Azure Data Lake Storage Gen2, Azure Databricks & Spark”.
I am using Hyperledger Fabric SDK for node.js to enroll a user. I am using this code to deploy in fabric. It uses FileKeyValueStore (uses files to store the key values) to store client's user credential.
I want to use CouchDBKeyValueStore to store user key in CouchDB database instance. What changes do i need to make in client connection profile configuration file for credential store and in code to do so. Any link to sample code will also help.
There is no built-in support in the connection profile for using the CouchDBKeyValueStore, but you can still use the connection profile for the rest of the Fabric network configuration. You'll then need to use the Client APIs to configure the stores. Something like
const Client = require('fabric-client');
const CDBKVS = require('fabric-client/lib/impl/CouchDBKeyValueStore.js');
var client = Client.loadFromConfig('test/fixtures/network.yaml');
// Set the state store
let stateStore = await new CDBKVS({url: 'https://<USERNAME>:<PASSWORD>#<URL>', name: '<DB_NAME>'})
client.setStateStore(stateStore);
// Set the crypto store
const crypto = Client.newCryptoSuite();
let cryptoKS = Client.newCryptoKeyStore(
CDBKVS,
{
url: 'https://<USERNAME>:<PASSWORD>#<URL>.cloudant.com',
name: '<DB_NAME>'
}
);
crypto.setCryptoKeyStore(cryptoKS);
client.setCryptoSuite(crypto);
Official document Reference
Store Hyperledger Fabric certificates and keys in IBM Cloudant with Fabric Node SDK
I just started to explore Kubernetes and I deployed a service in a container on Kubernetes which is running on a cloud.
My service requires to make a call to a database which requires a certificate for authentication. I am wondering what would be the best practice to store/install the certificate on Kubernetes
I need to access the certificate from my code which I am using as follows
const (
serverCertificate = "./cert/api.cer"
serverPrivateKey = "./cert/api.key"
)
creds, err := credentials.NewServerTLSFromFile(serverCertificate, serverPrivateKey)
You could store the certificate in a Kubernetes Secret: https://kubernetes.io/docs/concepts/configuration/secret/
Here is an example on how to do so:
https://kubernetes.io/docs/concepts/configuration/secret/#creating-a-secret-using-kubectl-create-secret