Why is aws-sdk-ruby connecting to '169.254.169.254' on initialization? - ruby

Question

I installed the gem webmock, which blocks external connections during the test suite. After I installed it, I got an unexpected result when running the test suite.
I'm somewhat confused why the aws-sdk needs to connect to 169.254.169.254 on initialization?
rspec spec/models/concerns/posconcern_spec.rb
[Coveralls] Set up the SimpleCov formatter.
[Coveralls] Using SimpleCov's 'rails' settings.
An error occurred while loading ./spec/models/concerns/posconcern_spec.rb.
Failure/Error: require File.expand_path('../../config/environment', __FILE__)
WebMock::NetConnectNotAllowedError:
Real HTTP connections are disabled. Unregistered request: GET http://169.254.169.254/latest/meta-data/iam/security-credentials/ with headers {'Accept'=>'*/*', 'Accept-Encoding'=>'gzip;q=1.0,deflate;q=0.6,identity;q=0.3', 'User-Agent'=>'Ruby'}
You can stub this request with the following snippet:
stub_request(:get, "http://169.254.169.254/latest/meta-data/iam/security-credentials/").
with(headers: {'Accept'=>'*/*', 'Accept-Encoding'=>'gzip;q=1.0,deflate;q=0.6,identity;q=0.3', 'User-Agent'=>'Ruby'}).
to_return(status: 200, body: "", headers: {})
============================================================
# /home/andey/.rvm/gems/ruby-2.3.4/gems/webmock-3.1.0/lib/webmock/http_lib_adapters/net_http.rb:114:in `request'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:109:in `http_get'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:90:in `block (2 levels) in get_credentials'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:105:in `open_connection'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:88:in `block in get_credentials'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:121:in `retry_errors'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:87:in `get_credentials'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:73:in `block in refresh'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:121:in `retry_errors'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:72:in `refresh'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/refreshing_credentials.rb:20:in `initialize'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:51:in `initialize'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/credential_provider_chain.rb:90:in `new'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/credential_provider_chain.rb:90:in `instance_profile_credentials'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/credential_provider_chain.rb:12:in `block in resolve'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/credential_provider_chain.rb:11:in `each'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/credential_provider_chain.rb:11:in `resolve'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/plugins/request_signer.rb:37:in `block in <class:RequestSigner>'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:70:in `call'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:205:in `block in resolve_defaults'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:57:in `each'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:57:in `each'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:204:in `resolve_defaults'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:200:in `value_at'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:189:in `block in resolve'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:189:in `resolve'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:177:in `apply_defaults'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:150:in `build!'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/base.rb:68:in `build_config'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/base.rb:19:in `initialize'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/base.rb:105:in `new'
# ./config/initializers/aws.rb:1:in `<top (required)>'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/activesupport-4.2.9/lib/active_support/dependencies.rb:268:in `load'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/activesupport-4.2.9/lib/active_support/dependencies.rb:268:in `block in load'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/activesupport-4.2.9/lib/active_support/dependencies.rb:240:in `load_dependency'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/activesupport-4.2.9/lib/active_support/dependencies.rb:268:in `load'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/engine.rb:652:in `block in load_config_initializer'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/activesupport-4.2.9/lib/active_support/notifications.rb:166:in `instrument'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/engine.rb:651:in `load_config_initializer'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/engine.rb:616:in `block (2 levels) in <class:Engine>'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/engine.rb:615:in `each'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/engine.rb:615:in `block in <class:Engine>'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/initializable.rb:30:in `instance_exec'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/initializable.rb:30:in `run'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/initializable.rb:55:in `block in run_initializers'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/initializable.rb:44:in `each'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/initializable.rb:44:in `tsort_each_child'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/initializable.rb:54:in `run_initializers'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/application.rb:352:in `initialize!'
# ./config/environment.rb:5:in `<top (required)>'
# ./spec/rails_helper.rb:3:in `require'
# ./spec/rails_helper.rb:3:in `<top (required)>'
# ./spec/models/concerns/posconcern_spec.rb:1:in `require'
# ./spec/models/concerns/posconcern_spec.rb:1:in `<top (required)>'
Relevant link to repo:
https://github.com/aws/aws-sdk-ruby/blob/master/gems/aws-sdk-core/lib/aws-sdk-core/instance_profile_credentials.rb#L45

Answer 1

It is trying to connect to the AWS metadata server to get the AWS credentials. All AWS servers should be able to connect to the internal metadata server.
I installed the gem webmock, which blocks external connections during
the test suite
How are you blocking it? Firewall? Add a rule to allow traffic to 169.254.169.254 or stub it.
The script is trying to connect to AWS services. For that it needs credentials which can be provided in many ways. One way is to use AWS IAM role and fetch the credentials dynamically from the metadata server (169.254.169.254). Your script is connecting to 169.254.169.254 to fetch the credentials. It can be used to connect to AWS services later.

Answer 2

This is a follow up to #helloV 's answer.
Full documentation on
http://169.254.169.254/latest/meta-data/iam/security-credentials/
An application on the instance retrieves the security credentials
provided by the role from the instance metadata item
iam/security-credentials/role-name. The application is granted the
permissions for the actions and resources that you've defined for the
role through the security credentials associated with the role. These
security credentials are temporary and we rotate them automatically.
We make new credentials available at least five minutes prior to the
expiration of the old credentials.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#instance-metadata-security-credentials

Answer 3

Had a tough time researching on how to stub it. Will share my findings here hope it'll help other people.
Since AWS uses service name to fetch credentials for multiple services such as S3, any IAM Role, SQS etc.
But when dealing with IAM roles AWS hits http://169.254.169.254/latest/meta-data/iam/security-credentials/ which return the role name attached onto the machine and using the response of the same it hits the curl http://169.254.169.254/latest/meta-data/iam/security-credentials/#{role}.
So we need to stub both of the curls in order to successfully mock the IAM role. Below is the code for the same.
def self.auth(params = {})
status = params[:status].presence || 200
url = 'http://169.254.169.254/latest/meta-data/iam/security-credentials/'
WebMock.stub_request(:get, url)
.with({headers: request_headers}.compact)
.to_return(status: status, body: 'SampleRole', headers: {})
end
def self.role_auth(role = 'SampleRole')
url = "http://169.254.169.254/latest/meta-data/iam/security-credentials/#{role}"
WebMock.stub_request(:get, url)
.with({headers: request_headers}.compact)
.to_return(status: 200, body: sample_response, headers: {})
end
private
def self.request_headers
{
'Accept': '*/*',
'Accept-Encoding': 'gzip;q=1.0,deflate;q=0.6,identity;q=0.3',
'User-Agent': 'aws-sdk-ruby3/3.21.2'
}.with_indifferent_access
end
def self.sample_response
{
Code: 'Success',
LastUpdated: '2020-05-08T05:55:23Z',
Type: 'AWS-HMAC',
AccessKeyId: Faker::Lorem.characters(10),
SecretAccessKey: Faker::Lorem.characters(20),
Token: Faker::Lorem.characters(60),
Expiration: '2020-05-08T12:00:45Z'
}.to_json
end
So whenver any rspec uses InstanceCredentials provider you need to stub both of the above defined requests in before block of each of the rspec.
Example:
before do
ApiStub::IamCred.auth
ApiStub::IamCred.role_auth
end
Hope it helps.
Happy Coding

Answer 4

There are some good answers in this thread on how to stub this. Another alternative (a better one?) is to eliminate the need for thE SDK to check for credentials from the instance meta data service. You can do this by providing credentials. These can be fake. A simple way to do this would be to provide credential via the ENV (e.g. ENV['AWS_ACCESS_TOKEN'] and ENV['SECRET_ACCESS_TOKEN']).