Ruby: S3 access with AWS instance profile - ruby

Question

I have a ec2 instance which has a profile attached. I can use awscli and it uploads to the bucket fine.
root#ocr-sa-test:/# aws s3 ls s3://company-ocr-east/
PRE 7_day_expiry/
root#ocr-sa-test:/# touch foo
root#ocr-sa-test:/# aws s3 cp foo s3://company-ocr-east/foo
upload: ./foo to s3://company-ocr-east/foo
root#ocr-sa-test:/# aws s3 rm s3://company-ocr-east/foo
delete: s3://company-ocr-east/foo
I can't get it to work with the aws-sdk in ruby though. I get access denied.
irb(main):001:0> require "aws-sdk"
=> true
irb(main):002:0>
irb(main):003:0> credentials = Aws::InstanceProfileCredentials.new
irb(main):004:1* client = Aws::S3::Client.new(
irb(main):005:1* region: "us-east-1",
irb(main):006:1* credentials: credentials,
irb(main):007:0> )
irb(main):008:0>
irb(main):009:0>
irb(main):010:0>
irb(main):011:1* begin
irb(main):012:2* client.put_object(
irb(main):013:2* key: 'hello.txt',
irb(main):014:2* body: 'Hello World!',
irb(main):015:2* bucket: 'company-ocr-east',
irb(main):016:2* content_type: 'text/plain'
irb(main):017:1* )
irb(main):018:1* rescue Exception => e
irb(main):019:1* puts "S3 Upload Error: #{e.class} : Message: #{e.message}"
irb(main):020:0> end
S3 Upload Error: Aws::S3::Errors::AccessDenied : Message: Access Denied

Answer 1

These commands aren't perfectly equivalent, so it'll be instructive to determine what exactly differs on the wire as a result. In particular, the SDK is being instructed to use a specific region and to obtain STS tokens from IMDS, whilst the CLI is left to work things out from either its own defaults or a profile config. Besides which, they don't behave exactly the same.
To find out what's actually happening, means re-running both with applicable debug flags, viz:
aws --debug s3 cp hello.txt s3://bucketname/hello.txt
and
credentials = Aws::InstanceProfileCredentials.new(http_debug_output: $stdout)
client = Aws::S3::Client.new(region: 'us-east-1', credentials: credentials, http_wire_trace: true)
client.put_object(key: 'hello.txt', body: 'Hello World!', bucket: 'bucketname', content_type: 'text/plain')
These will generate heaps of output but it's all relevant and, crucially, comparable once you look past the noise. The first thing to verify is that the CLI is definitely talking to IMDS (it'll have requests to http://169.254.169.254 that culminate with something like "found credentials from IAM Role". If not, then the instance isn't configured how you thought, and there'll be clues in the log to explain how it is getting credentials, e.g. unexpected profile file, or environment variables. You'll also want to check they are obtaining the same role.
The second thing to compare is the subsequent sequences of PUT they both attempt. At this point in the debugging, almost everything else is equal, so it's very likely you can adjust the settings of the Ruby SDK client to match whatever the CLI is succeeding with.
The third possibility is that of a system firewall, or some kind of process-level mandatory access controls, user permissions, cgroups/containers etc. However, debugging your OS kernel & configuration would be a deep, dark rabbit hole, and in any case you've said this is "an EC2 instance" so it is, presumably, a plain old EC2 instance. If in fact the Ruby commands above are running under a different user ID, or inside a container, then maybe there's your answer already, it could well be a networking issue due to user/container/security controls or similar OS-level configuration that needs fixing up.
Obligatory warning: if you choose to post any of the log data, be careful to overwrite any credentials! I don't believe these debug traces are particularly replayable, but you don't want to find out the hard way if I'm wrong.

Answer 2

The access denied error may be caused by the "very aggressive" default timeout in Aws::InstanceProfileCredentials.
Try initializing it with a longer timeout or additional retries:
credentials = Aws::InstanceProfileCredentials.new({
retries: 2, # Integer, default: 1
http_open_timeout: 2.5, # Float, default: 1
http_read_timeout: 2.5 # Float, default: 1
})
The docs did not make clear if the timeout options were given as seconds or another duration. 2.5 seemed conservative, given the default. Further tweaking may be needed.
The AWS docs for the v3 Ruby API discuss the aggressive timeout in the Aws::S3::Client docs and you can see options to configure Aws::InstanceProfileCredentials.