I am trying to list and download folders from a bucket on path eg:"aaa/bbb/" using the aws-sdk gem v2. However I can't figure out how to do it.
This is what I tried:
require 'aws-sdk'
Aws.config.update({
region: 'us-west-2',
credentials: Aws::Credentials.new('akid', 'secret')
})
s3 = Aws::S3::Resource.new
# reference an existing bucket by name
bucket = s3.bucket('aaa')
bucket.objects(prefix: '/bbb/').each do |folder|
p folder
end
It says: Access Denied (Aws::S3::Errors::AccessDenied)
But, if I use the command line AWS CLI instead, and execute:
aws s3 ls aaa/bbb/
it works...
Any suggestion?
Many thanks.
The convention in S3 is that the "root" of a bucket's keyspace is a zero-length empty string... it is not / as some people naturally assume.
The prefix you are looking for would be expressed as bbb/ rather than /bbb/.
According to the documentation you've to put in the credentials slightly different:
require 'aws-sdk'
Aws.config.update({
region: 'us-west-2',
credentials: Aws::Credentials.new('akid', 'secret')
})
Maybe try this to list the content of the bucket:
s3.list_objects(bucket:'aaa').each do |response|
puts response.contents.map(&:key)
end
Related
I have a ec2 instance which has a profile attached. I can use awscli and it uploads to the bucket fine.
root#ocr-sa-test:/# aws s3 ls s3://company-ocr-east/
PRE 7_day_expiry/
root#ocr-sa-test:/# touch foo
root#ocr-sa-test:/# aws s3 cp foo s3://company-ocr-east/foo
upload: ./foo to s3://company-ocr-east/foo
root#ocr-sa-test:/# aws s3 rm s3://company-ocr-east/foo
delete: s3://company-ocr-east/foo
I can't get it to work with the aws-sdk in ruby though. I get access denied.
irb(main):001:0> require "aws-sdk"
=> true
irb(main):002:0>
irb(main):003:0> credentials = Aws::InstanceProfileCredentials.new
irb(main):004:1* client = Aws::S3::Client.new(
irb(main):005:1* region: "us-east-1",
irb(main):006:1* credentials: credentials,
irb(main):007:0> )
irb(main):008:0>
irb(main):009:0>
irb(main):010:0>
irb(main):011:1* begin
irb(main):012:2* client.put_object(
irb(main):013:2* key: 'hello.txt',
irb(main):014:2* body: 'Hello World!',
irb(main):015:2* bucket: 'company-ocr-east',
irb(main):016:2* content_type: 'text/plain'
irb(main):017:1* )
irb(main):018:1* rescue Exception => e
irb(main):019:1* puts "S3 Upload Error: #{e.class} : Message: #{e.message}"
irb(main):020:0> end
S3 Upload Error: Aws::S3::Errors::AccessDenied : Message: Access Denied
These commands aren't perfectly equivalent, so it'll be instructive to determine what exactly differs on the wire as a result. In particular, the SDK is being instructed to use a specific region and to obtain STS tokens from IMDS, whilst the CLI is left to work things out from either its own defaults or a profile config. Besides which, they don't behave exactly the same.
To find out what's actually happening, means re-running both with applicable debug flags, viz:
aws --debug s3 cp hello.txt s3://bucketname/hello.txt
and
credentials = Aws::InstanceProfileCredentials.new(http_debug_output: $stdout)
client = Aws::S3::Client.new(region: 'us-east-1', credentials: credentials, http_wire_trace: true)
client.put_object(key: 'hello.txt', body: 'Hello World!', bucket: 'bucketname', content_type: 'text/plain')
These will generate heaps of output but it's all relevant and, crucially, comparable once you look past the noise. The first thing to verify is that the CLI is definitely talking to IMDS (it'll have requests to http://169.254.169.254 that culminate with something like "found credentials from IAM Role". If not, then the instance isn't configured how you thought, and there'll be clues in the log to explain how it is getting credentials, e.g. unexpected profile file, or environment variables. You'll also want to check they are obtaining the same role.
The second thing to compare is the subsequent sequences of PUT they both attempt. At this point in the debugging, almost everything else is equal, so it's very likely you can adjust the settings of the Ruby SDK client to match whatever the CLI is succeeding with.
The third possibility is that of a system firewall, or some kind of process-level mandatory access controls, user permissions, cgroups/containers etc. However, debugging your OS kernel & configuration would be a deep, dark rabbit hole, and in any case you've said this is "an EC2 instance" so it is, presumably, a plain old EC2 instance. If in fact the Ruby commands above are running under a different user ID, or inside a container, then maybe there's your answer already, it could well be a networking issue due to user/container/security controls or similar OS-level configuration that needs fixing up.
Obligatory warning: if you choose to post any of the log data, be careful to overwrite any credentials! I don't believe these debug traces are particularly replayable, but you don't want to find out the hard way if I'm wrong.
The access denied error may be caused by the "very aggressive" default timeout in Aws::InstanceProfileCredentials.
Try initializing it with a longer timeout or additional retries:
credentials = Aws::InstanceProfileCredentials.new({
retries: 2, # Integer, default: 1
http_open_timeout: 2.5, # Float, default: 1
http_read_timeout: 2.5 # Float, default: 1
})
The docs did not make clear if the timeout options were given as seconds or another duration. 2.5 seemed conservative, given the default. Further tweaking may be needed.
The AWS docs for the v3 Ruby API discuss the aggressive timeout in the Aws::S3::Client docs and you can see options to configure Aws::InstanceProfileCredentials.
unable to upload a picture file into S3 bucket in my APP with fog gem, even though I could upload locally.
I still can log into AWS successfully with correct IAM user name, who has AdministratorAccess and AmazonS3FullAccess policy names. I can see my S3 Bucket right there with a picture file which I have manually uploaded there before.
I tried with a different new user and I got "The AWS Access Key Id you provided does not exist in our records." error message. I am trying 3rd new user while waiting for your answers.
In my carrier_wave.rb file,
...
if Rails.env.production?
CarrierWave.configure do |config|
config.fog_provider = 'fog/aws'
config.fog_credentials = {
# Configuration for Amazon S3
:provider => 'AWS',
:aws_access_key_id => 'correct aws_access_key_id',
:aws_secret_access_key => 'correct aws_secret_access_key',
:region => 'us-east-2',
:host => 's3.us-east-2.amazonaws.com'
}
config.fog_directory = 'correct bucket name'
end
end
...
Any suggests would help me a lot since I have already run out of ideas. Thank you!
Currently, I am sending GET requests to S3 using aws-sdk ruby as follow:
#!/usr/bin/ruby
#
require 'aws-sdk'
s3 = Aws::S3::Resource.new(region: 'test', endpoint:'http://10.0.23.45:8081')
my_bucket = s3.bucket('test.bucket-name')
my_bucket.objects.limit(50).each do |obj|
puts " #{obj.key} => #{obj.etag}"
end
But the request is trying to hit this url endpoint(virtual hosting):
http://test.bucket-name.10.0.23.45:8081
I would like to use path style addressing instead. This is what I want the request url endpoint to look like:
http://10.0.23.45:8081/test.bucket-name/
Any idea how to set path style addressing instead of virtual hosting address? Thanks.
I found the answer for my own question after looking at the source code of ruby aws-sdk Source Code
Aws.config[:s3] = { force_path_style: true }
Adding the above line forced to use path style addressing.
You need to set option :virtual_host to true according to documentation.
So in your case something like this should work:
s3.bucket('10.0.23.45:8081').object('test.bucket-name').public_url(virtual_host: true)
#=> "http://10.0.23.45:8081/test.bucket-name/"
This is the URL generated by the Ruby aws-sdk gem for put:
curl --upload-file "/Users/README.rdoc"
-H "x-amz-acl=public-read"
"http://videos.s3.amazonaws.com/6c06517c-64f1-45ed-b07f-8c4c4edec6e3?AWSAccessKeyId={key}&Expires=1384519899&Signature=MKtBESBklYXFT%2B48EKLSoBiQpNA%3D"
-H "x-amz-acl=public-read" is not present in the signature. The signature is OK (Amazon doesn't show any errors).
But the "public-read" permission is not applied, please advise me as to how I can generate a put signed URL which will be public-read after upload.
Thanks!
Updated:
s3 = AWS::S3.new
bucket = s3.buckets['some_videos']
id = SecureRandom.uuid
object = bucket.objects["#{id}"]
url = object.url_for(:put, expires_in: 30*60)
it looks like you can specify this with the acl method (documented here)
If you want to set your bucket to public read you can call:
s3.buckets['some-videos'].acl = :public_read
if you would like to apply this permission directly to an object you can call:
bucket.objects["#{id}"].acl= :public_read
Amazon team added this to their sdk. Thanks, guys!
https://github.com/aws/aws-sdk-ruby/issues/412
https://github.com/aws/aws-sdk-ruby/commit/15e900c0918a67e20bbb6dd9509c112aa01a95ee
I'm using a ruby script to generate an html page that I'm uploading to s3. My amazon bucket is configured to work as a static website, and the html page my script is generating is the index page.
I've created a couple of models to build a connection to s3 using the fog gem, generate the html page, and then push it up to s3.
When I use the script to push the file up to s3, I don't get any errors, but only half the file gets uploaded. It happens regardless of file size. I inevitably wind up with about 15 rows missing from the html table I generated.
The relevant code from my FogManager model is this:
def connect
#connection = Fog::Storage.new({
provider: 'AWS',
aws_access_key_id: AWS_ACCESS_KEY,
aws_secret_access_key: AWS_SECRET_KEY,
persistent: true
})
end
def locate_directory
#directory = #connection.directories.get(DIRECTORY)
end
def upload_report_page(index_file)
file = #directory.files.get(index_file)
file.body = File.open(index_file)
file.acl = 'public-read'
file.save
end
My script looks something like:
filename = "index.html"
s3 = FogManager.new
s3.connect
s3.locate_directory
# code for generating the html page
s3.upload_report_page(filename)
All of the code works, but the whole html file isn't getting uploaded to s3.
I'm using the latest version of fog (1.15.0) with ruby 2.0.0p0
I've been able to manually go into irb and establish an s3 connection, locate the file, and upload the new one. I use these commands:
require 'fog'
connection = Fog:Storage({provider: 'AWS',
aws_access_key_id: 'xxx',
aws_secret_access_key: 'xxx',
persistent: true})
directory = connection.directories.get('my_directory')
file = directory.files.get('index.html')
file.body = File.open('index.html')
file.acl = 'public-read'
file.save
When I upload the file this way, it works properly, and the whole file gets uploaded, but it defeats the purpose of having the script.
Any insight would be greatly appreciated!