I'm getting an error using AzCopy to copy an s3 bucket into an azure container, following the guide at https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-s3
I used azcopy login to authenticate, and added the below permissions to my azure account
Storage Blob Data Contributor
Storage Blob Data Owner
Storage Queue Data Contributor
Then trying to copy my bucket with
./azcopy copy 'https://s3.us-east-1.amazonaws.com/my-bucket' 'https://my-account.blob.core.windows.net/my-container' --recursive=true
I then receive an error that
AuthorizationPermissionMismatch
RESPONSE Status: 403 This request is not authorized to perform this operation using this permission.
What other permissions could I be missing or what else could it be?
Turns out I just had to wait a few hours for the permissions to fully propagate
Please check if below is missing:
To Authorize with AWS S3 ,you may need to gather your AWS access key and secret and then set the environment variables of that s3 source after getting hold of them.
Windows:
set AWS_ACCESS_KEY_ID=<access-key>
set AWS_SECRET_ACCESS_KEY=<secret-access-key>
(or)
Linux:
export AWS_ACCESS_KEY_ID=<access-key>
export AWS_SECRET_ACCESS_KEY=<secret-access-key>
Please make sure you've been granted the required permissions /actions for Amazon S3 object operations to copy data from Amazon S3,
for example> s3:GetObject and s3:GetObjectVersion.
References:
azcopy
Authorize with AWS S3
Related
I need to mount Azure Blob storage (where hierarchical namespace is disabled) from databricks. Mount command returns true but when I run fs.ls command, it returns error UnknownHostException. Please suggest
I got a similar kind of error. I tried and unmounted my blob storage account. Then, Remounted my storage account. Now, it's working fine.
Unmounting Storage account:
dbutils.fs.unmount("<mount_point>")
Mount Blob Storage:
dbutils.fs.mount(
source = "wasbs://<container>#<Storage_account_name>.blob.core.windows.net/",
mount_point = "<mount_point>",
extra_configs = {"fs.azure.account.key.vamblob.blob.core.windows.net":"Access_key"})
display(dbutils.fs.ls('/mnt/fgs'))
This command display(dbutils.fs.ls('/mnt/fgs')) returns all the files available in the mount point. You can perform all the required operations and then write to this DBFS, which will be reflected in your blob storage container also.
For more information refer this MS Document.
I am designing a storage using azure blob storage. In a container, how to do access control between different blobs?
For example, under container "images", there are 2 blobs: design1/logo.png and design2/logo.png. How to make the access to design1/ and design2/ are exclusively?
Have you tried to configure the access permissions from RBAC?
https://learn.microsoft.com/en-us/azure/storage/blobs/assign-azure-role-data-access?tabs=portal
Good day,
Is there any script or any aws cli command to know which IAM user created what resource in AWS. so that we just enter the IAM user name and it shows all the resources created by that particular IAM user.
thanks in advance.
The service that you're looking for is CloudTrail.
By default, it retains 90 days worth of events for the current account and region, and you can access it from either the Console or CLI. You can also configure it to write events to S3, where they're be preserved as long as you want to pay for the storage (this also lets you capture events across all regions, and for every account in an orgnanization).
CloudTrail events can be challenging to search. If you're just looking for events by a specific user, and know that user's access key (here I'm using my access key stored in an environment variable) you can use a query like this:
aws cloudtrail lookup-events --lookup-attributes "AttributeKey=AccessKeyId,AttributeValue=$AWS_ACCESS_KEY_ID" --query 'Events[].[EventTime,EventName,Username,EventId]' --output table
Or, by username:
aws cloudtrail lookup-events --lookup-attributes "AttributeKey=Username,AttributeValue=parsifal" --query 'Events[].[EventTime,EventName,Username,EventId]' --output table
You can then use grep to find the event(s) that interest you, and dig into the details of a specific event with:
aws cloudtrail lookup-events --lookup-attributes "AttributeKey=EventId,AttributeValue=8c5a5d8a-9999-9999-9999-a8e4b5213c3d"
I am trying to copy an object from Phoenix region to Ashburn . The admin for the tenant still unable to perform this action . Am I missing any privileges?
I am seeing an error in the Work Request The Service Cannot Access the Source Bucket
Do I need to add additional policy statements?
Yes, the service needs access too.
You can refer to the documentation here, specifically:
Service Permissions
To enable object copy, you must authorize the service to manage objects on your behalf. To do so, create the
following policy:
allow service objectstorage-<region_name> to manage object-family in
compartment <compartment_name>
Because Object Storage is a
regional service, you must authorize the Object Storage service for
each region that will be carrying out copy operations on your behalf.
For example, you might authorize the Object Storage service in region
us-ashburn-1 to manage objects on your behalf. Once you do this, you
will be able to initiate the copy of an object stored in a
us-ashburn-1 bucket to a bucket in any other region, assuming that
your user account has the required permissions to manage objects
within the source and destination buckets.
I would like to access external data from my aws ec2 instance.
In more detail: I would like to specify inside by user-data the name of a folder containing about 2M of binary data. When my aws instance starts up, I would like it to download the files in that folder and copy them to a specific location on the local disk. I only need to access the data once, at startup.
I don't want to store the data in S3 because, as I understand it, this would require storing my aws credentials on the instance itself, or passing them as userdata which is also a security risk. Please correct me if I am wrong here.
I am looking for a solution that is both secure and highly reliable.
which operating system do you run ?
you can use an elastic block storage. it's like a device you can mount at boot (without credentials) and you have permanent storage there.
You can also sync up instances using something like Gluster filesystem. See this thread on it.