How do I use server side encryption when uploading a file to s3 via the ruby sdk? - ruby

I'm running into an issue uploading to S3 with version 2 of the sdk.
When running:
Aws.config.update({
region: 'us-east-1',
credentials: Aws::Credentials.new(credentials['key'],credentials['secret'],
s3_server_side_encryption: :aes256)
})
s3 = Aws::S3::Resource.new
bucket = 'VandalayIndustriesAccountingData'
s3_file_path = "folder/filename.tar.gz"
s3_object = s3.bucket(bucket).object(s3_file_path)
s3_object.upload_file(artifact_location)
I get the following error:
Aws::S3::Errors::InvalidToken
-----------------------------
The provided token is malformed or otherwise invalid.
When I remove the s3_server_side_encryption setting it changes to an access denied error.
I've been trying to find documentation around doing this with v2 of the API, but everything online seems to rely on the bucket object having a write method which doesn't seem to exist in v2 of the API.
http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingRubySDK.html
I'm likely just not finding the correct document in the v2 api. I'd like to avoid using v1 and v2 of the api but may fall back to that.

upload_file takes arguments similar to write
Aws.config.update({
region: 'us-east-1',
credentials: Aws::Credentials.new(credentials['key'],credentials['secret'],
)
})
s3 = Aws::S3::Resource.new
bucket = 'VandalayIndustriesAccountingData'
s3_file_path = "folder/filename.tar.gz"
s3_object = s3.bucket(bucket).object(s3_file_path)
s3_object.upload_file(artifact_location, server_side_encryption: :AES256)

Related

Authenticating to Google Vault API -- getting 400 response with message: "The user does not belong to a G Suite customer."

I'm trying to do a simple query from the Google Vault API using JSON credentials provided from the Google API console for a service account. I'm getting a 400 response (Invalid Request / Invalid Argument) with the message:
The user does not belong to a G Suite customer.
Does anyone know what I might be doing wrong? Do I have to augment the JSON with anything to indicate our G Suite Account?
Thanks in advance.
The code's fairly straight forward:
matter_id = 'xxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx'
vault = Google::Apis::VaultV1::VaultService.new
scope = 'https://www.googleapis.com/auth/ediscovery.readonly'
vault.authorization = Google::Auth::ServiceAccountCredentials.make_creds(
json_key_io: File.open('./xxxxxxxxxx.json'),
scope: scope)
vault.authorization.fetch_access_token!
m = vault.get_matter(matter_id)
Update -- this has been resolved. You have to update sub after you create the credentials to impersonate a user.
matter_id = 'xxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx'
vault = Google::Apis::VaultV1::VaultService.new
scope = 'https://www.googleapis.com/auth/ediscovery.readonly'
credentials = Google::Auth::ServiceAccountCredentials.make_creds(
json_key_io: File.open('./xxxxxxxxxx.json'),
scope: scope)
credentials.update!(sub: 'user#domain.com')
vault.authorization = credentials
vault.authorization.fetch_access_token!
m = vault.get_matter(matter_id)

MIP SDK fails to protect files

I'm using MIP file sample command line interface to apply labelling.
When trying to apply a label that has protection set, i got "Label requires ad-hoc protection, but protection has not yet been set" error.
Therefore, I tried protecting the file using "--protect" option and got the following error message:
"Something bad happened: The service didn't accept the auth token. Challenge:['Bearer resource="https://aadrm.com", realm="", authorization="https://login.windows.net/common/oauth2/authorize"'], CorrelationId=ce732e4a-249a-47ec-a7c2-04f4d68357da, CorrelationId.Description=ProtectionEngine, CorrelationId=6ff992dc-91b3-4610-a24d-d57e13902114, CorrelationId.Description=FileHandler"
This is my auth.py file:
def main(argv):
client_id = str(argv[0])
tenant_id = str(argv[1])
secret = str(argv[2])
authority = "https://login.microsoftonline.com/{}".format(tenant_id)
app = msal.ConfidentialClientApplication(client_id, authority=authority, client_credential=secret)
result = None
scope = ["https://psor.o365syncservice.com/.default"]
result = app.acquire_token_silent(scope, account=None)
if not result:
logging.info("No suitable token exists in cache. Let's get a new one from AAD.")
result = app.acquire_token_for_client(scopes=scope)
if "access_token" in result:
sys.stdout.write(result['access_token'])
else:
print(result.get("error"))
print(result.get("error_description"))
print(result.get("correlation_id")) # You may need this when reporting a bug
if __name__ == '__main__':
main(sys.argv[1:])
I tried to change the scope to ["https://aadrm.com/.default"] and then I was able to protect the file, but when I try getting the file status or try applying label on it I get the same error message with bad auth token.
These are the permissions as configured in azure portal:
Thank you
I think that scope you have is incorrect: https://psor.o365syncservice.com/.default
It should be https://syncservice.o365syncservice.com/.default.
A good way to handle this is to just append .default to whatever resource the AcquireToken() call gets in the resource param. Something like this.

Bucketeer - Heroku's add-on s3 bucket configuration on Django

I am currently using S3 to serve static files on Heroku. The S3 bucket was created and is managed by me and its settings.py file is the following.
import os
AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
AWS_STORAGE_BUCKET_NAME = '<MY BUCKET NAME>'
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
STATIC_URL = 'http://' + AWS_STORAGE_BUCKET_NAME + '.s3.amazonaws.com/'
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
Which is the same as this answer and it works perfectly fine: Django + Heroku + S3
However I wanted to switch to Bucketeer which is a Heroku add-on that creates and manages a S3 bucket for you. But Bucketeer provides different parameters and the static URL looks different and I can't make it work. The URL has the following pattern: "bucketeer-heroku-shared.s3.amazonaws.com/UNIQUE_BUCKETEER_BUCKET_PREFIX/public/". So my updated code is the following.
#Bucketeer
AWS_ACCESS_KEY_ID = os.environ.get('BUCKETEER_AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('BUCKETEER_AWS_SECRET_ACCESS_KEY')
BUCKETEER_BUCKET_PREFIX = os.environ.get('BUCKETEER_BUCKET_PREFIX')
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
#Bucketeer Config
STATIC_URL = 'http://bucketeer-heroku-shared.s3.amazonaws.com/' +
BUCKETEER_BUCKET_PREFIX + '/public/'
#I also tried
#STATIC_URL = 'http://bucketeer-heroku-shared.s3.amazonaws.com/' +
# BUCKETEER_BUCKET_PREFIX + '/'
And this is the error I got.
Preparing static assets
Collectstatic configuration error. To debug, run:
$ heroku run python manage.py collectstatic --noinput
Needless to say no static files were present on the app, so when I ran the suggested command I got:
boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
Which means I'm not authorized to access said bucket. Could somebody shed some light on what is going on here and how to fix it.

AWS SDK v2 for s3

Can any one provide me a good documentation for uploading files to S3 using asw-sdk Version 2. I checked out the main doc and in v1 we used to do like
s3 = AWS::S3.new
obj = s3.buckets['my-bucket']
Now in v2 when I try as
s3 = Aws::S3::Client.new
am ending up with
Aws::Errors::MissingRegionError: missing region; use :region option or export region name to ENV['AWS_REGION']
Can anyone help me with this?
As per official documentation:
To use the Ruby SDK, you must configure a region and credentials.
Therefore,
s3 = Aws::S3::Client.new(region:'us-west-2')
Alternatively, a default region can be loaded from one of the following locations:
Aws.config[:region]
ENV['AWS_REGION']
Here's a complete S3 demo on aws v2 gem that worked for me:
Aws.config.update(
region: 'us-east-1',
credentials: Aws::Credentials.new(
Figaro.env.s3_access_key_id,
Figaro.env.s3_secret_access_key
)
)
s3 = Aws::S3::Client.new
resp = s3.list_buckets
puts resp.buckets.map(&:name)
Gist
Official list of AWS region IDs here.
If you're unsure of the region, the best guess would be US Standard, which has the ID us-east-1 for config purposes, as shown above.
If you were using a aws.yml file for your credentials in Rails, you might want to create a file config/initializers/aws.rb with the following content:
filename = File.expand_path(File.join(Rails.root, "config", "aws.yml"))
config = YAML.load_file(filename)
aws_config = config[Rails.env.to_s].symbolize_keys
Aws.config.update({
region: aws_config[:region],
credentials: Aws::Credentials.new(aws_config[:access_key_id], aws_config[:secret_access_key])
})
The config/aws.yml file would need to be adapter to include the region.
development: &development
region: 'your region'
access_key_id: 'your access key'
secret_access_key: 'your secret access key'
production:
<<: *development

unable to delete file from amazon s3 using ruby script

I am using aws-sdk-ruby for deleting a file saved in a bucket in my amazon s3 account, but i can't figure out why i am able to delete the desired file from S3 bucket using the following code.
this is my code
require 'aws-sdk-v1'
require 'aws-sdk'
ENV['AWS_ACCESS_KEY_ID'] = "XXXXXXX"
ENV["AWS_SECRET_ACCESS_KEY"] = '/ZZZZZZZZ'
ENV['AWS_REGION'] = 'us-east-1'
s3 = Aws::S3::Resource.new
bucket = s3.bucket('some-bucket')
obj = bucket.object('https://s3.amazonaws.com/some-bucket/38ac8226-fa72-4aee-8c3d-a34a1db77b91/some_image.jpg')
obj.delete
The documentation tells that it should look like this:
s3 = Aws::S3.new
bucket = s3.buckets['some-bucket']
object = bucket.objects['38ac8226-fa72-4aee-8c3d-a34a1db77b91/some_image.jpg']
object.delete
Please note:
the square brackets,
that the object's key doesn't include the domain and
instead of creating an instance of Aws::S3::Resource create an instance of AWS::S3
If you use API version 3 (aws-sdk-s3 (1.81.1)) you should do something like below:
s3 = Aws::S3::Client.new
s3.delete_object(bucket: 'bucket_name', key: 'bucket_folder/file.txt')
it should be:
objs = bucket.objects('https://s3.amazonaws.com/some-bucket/38ac8226-fa72-4aee-8c3d-a34a1db77b91/some_image.jpg')
objs.each {|obj| obj.delete}
With the aws-sdk v2: I had to do this: (see Doc)
$s3.bucket("my-bucket").objects(prefix: 'my_folder/').batch_delete!
(delete is deprecated in favor of batch_delete)
Useful post: https://ruby.awsblog.com/post/Tx1H87IVGVUMIB5/Using-Resources

Resources