I have a git lab runner whose job it is to:
a) Create IaC using Terraform. (7 ec2 instances with a defined keypair)
b) Run an ansible playbook that will need to SSH to all 7 instances and configure Kafka.
At the moment, I have automated part a. I then ssh to one of the instances using a private key. copy the ansible code and the private key to the instance and then execute the following cmd to run the ansible:
ansible-playbook --private-key=/home/ec2-user/keyname.pem hosts.yml all.yml
This all works fine but obviously, I want to automate the running of the ansible in a gitlab runner without having to store my private key on the docker container or in the git repo.
I have briefly investigated SSM but don't really understand how that all works.
Note: I need the key for two purposes.
ssh into the first instance
referenced in the host.yml so that the ansible playbook can connect to all other instances
Thanks in advance everyone.
Cheers
Adam
When ssh into a aws ec2 linux instance, the user is ec2-user by default. Then I need to set aws credentials by writing to ~/.aws/credentials, but got permission denied. I feel that if I use sudo then the credentials file would be owned by root user, as a result my api server can't read from it.
What's the correct approach to set up aws credentials there?
The 'correct' way to setup the credentials, is to assign a role to the ec2 instance when you create it (or assign them after you create it). That role can be created and assigned to the EC2 instance via the AWS console - there is no need to ssh in and create the credentials there.
See: Easily Replace or Attach an IAM Role to an Existing EC2 Instance by Using the EC2 Console | AWS Security Blog
You can create the credentials file locally, then upload to your ec2 instance.
create the credentials file locally
$ vim credentials
upload to your ec2 instance
$ scp /path/credentials username#servername:/path
I'm trying to configure my AWS account using Ansible and from what I know it needs to be on one line (unless theres a way to pres "ENTER" progomatically in the Windows command).
Is there a way to do this?
Follow this command
$aws configure set aws_access_key_id default_access_key
$ aws configure set aws_secret_access_key default_secret_key
$ aws configure set default.region us-west-2
or
aws configure set aws_access_key_id <key_id> && aws configure set aws_secret_access_key <key> && aws configure set default.region us-east-1
For more details use this link
https://awscli.amazonaws.com/v2/documentation/api/latest/reference/configure/set.html
With aws configure we can also set values interactively, but with aws configure set we can set the values directly.
SYNOPSIS
aws configure set varname value [--profile profile-name]
OPTIONS
varname (string) The name of the config value to set.
value (string) The value to set.
For example.
aws configure --profile myprofile set region us-east-1
aws configure --profile myprofile set aws_access_key_id XXXXXXXXXXX
aws configure --profile myprofile set aws_secret_access_key YYYYYYYY
Alternately, you may also use.
aws configure set profile.myprofile.region us-east-1
aws configure set profile.myprofile.aws_access_key_id XXXXXXXXXXX
aws configure set profile.myprofile.aws_secret_access_key YYYYYYYY
I am new to ansible - I am using ansible to add the instances created by ELB ( my AWS will create instances for ELB) to ansible hosts file and access the instances from ansible server. From a linux machine, i use jumpbox and .pem key to access the ec2instance. How will I do in ansible ?
You should be able to pass in the flag --private-key=. You will probably also want to use -u ec2user to instruct Ansible to login as the correct user.
Can anybody tell me how to automate the aws configure in bash with a one liner?
Example:
$ aws configure --profile user2
AWS Access Key ID [None]: AKIAI44QH8DHBEXAMPLE
AWS Secret Access Key [None]: je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
Default region name [None]: us-east-1
Default output format [None]: text
Application: I want to automate this inside a Docker Entrypoint!
If you run aws configure set help you will see that you can supply settings individually on the command line and they will be written to the relevant credentials or config file. For example:
aws configure set aws_access_key_id AKIAI44QH8DHBEXAMPLE
You can also run this interactively to modify the default credentials:
aws configure
Or run it interactively to create/modify a named profile:
aws configure --profile qa
Note: with the first technique above, whatever command you type will appear in your history and this is not a good thing for passwords, secret keys etc. So in that case, use an alternative that does not cause the secret parameter to be logged to history, or prevent the entire command being logged to history.
One liner
aws configure set aws_access_key_id "AKIAI44QH8DHBEXAMPLE" --profile user2 && aws configure set aws_secret_access_key "je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY" --profile user2 && aws configure set region "us-east-1" --profile user2 && aws configure set output "text" --profile user2
Note: setting region is optional (also never set it with an empty string if you don't have any region, or it will be buggy); as well as the user profile, if you don't set it it will go under default settings.
👍 Better practice with Secrets
Use secrets, then use associated environment variables:
aws configure set aws_access_key_id "$AWS_ACCESS_KEY_ID" --profile user2 && aws configure set aws_secret_access_key "$AWS_ACCESS_KEY_SECRET" --profile user2 && aws configure set region "$AWS_REGION" --profile user2 && aws configure set output "text" --profile user2
📖 To know more
Run aws configure set help to get command line options.
Documentation for aws configure set.
Documentation for secrets: Docker, Kubernetes, GitLab.
If you want to automate you should use files rather than CLI. Your CLI only write those files.
➜ cat ~/.aws/config
[profile_1]
output = json
region = eu-west-1
[profile_2]
output = json
region = eu-west-1
➜ cat ~/.aws/credentials
[profile_1]
aws_access_key_id =
aws_secret_access_key =
[profile_2]
aws_access_key_id =
aws_secret_access_key =
For those inclined to use bash, the following works quite well and keeps secrets out of your scripts. In addition, it will also save your input to a named profile in one go.
printf "%s\n%s\nus-east-1\njson" "$KEY_ID" "$SECRET_KEY" | aws configure --profile my-profile
I think this is the answer in one line
aws configure set aws_access_key_id $YOUR_ACCESS_KEY_ID; aws configure set aws_secret_access_key $YOUR_SECRET_ACCESS_KEY; aws configure set default.region $YOUR_AWS_DEFAULT_REGION
One liner
aws configure set aws_access_key_id "$AWS_ACCESS_KEY_ID" --profile profile_name_here && aws configure set aws_secret_access_key "$AWS_SECRET_ACCESS_KEY" --profile profile_name_here && aws configure set region "$AWS_REGION" --profile profile_name_here && aws configure set output "json" --profile profile_name_here
Setting individual configuration
profile_name_here is the aws profile name to be saved to your aws config. Replace it with your own.
ACCESS KEY
aws configure set aws_access_key_id "$AWS_ACCESS_KEY_ID" --profile profile_name_here
SECRET ACCESS KEY
aws configure set aws_secret_access_key "$AWS_SECRET_ACCESS_KEY" --profile profile_name_here
REGION
aws configure set region "$AWS_REGION" --profile profile_name_here
OUTPUT
aws configure set output "json" --profile profile_name_here
The value specified here is json but you can replace it from the list of supported output formats from aws docs.
json
yaml
yaml-stream
text
table
Note:
That $AWS_ACCESS_KEY_ID, $AWS_SECRET_ACCESS_KEY and $AWS_REGION are variables from your AWS credentials file or environment variables if you are using CI. You can also replace them using regular strings value but that is not safe.
Building upon the suggestion by Tom in jarmod's answer, to "configure your keys in a config file that you then share with your docker container instead".
I found that slightly confusing as I'm new to using Docker and awscli.
Also, I believe most who end up at this question are similarly trying to use Docker and awscli together.
So what you'd want to do, step by step is:
Create a credentials file containing
[default]
aws_access_key_id = default_access_key
aws_secret_access_key = default_secret_key
that you copy to ~/.aws/credentials, using a line in Dockerfile like
COPY credentials /root/.aws/credentials
and a config file containing
[default]
region = us-west-2
output = table
that you copy to ~/.aws/config, using a line in Dockerfile like
COPY config /root/.aws/config
Reference:
aws configure set help