Is it possible to initialize EC2 machine in CDK with needed packages?
Or only way is to create it first and then install needed packages?
Thank you for your answers an tips,
The UserData can run the cfn-init script which will pull down Metadata passed in from CloudFormation, In CDK, this can be done in one object with the CloudFormationInit class
From the CDK documentation - https://docs.aws.amazon.com/cdk/api/latest/docs/aws-ec2-readme.html#configuring-instances-using-cloudformation-init-cfn-init
adding the init param into your aws_ec2.Instance definition (for python similar to)
aws_ec2.Instance(self, scope, ...
init=aws_ec2.CloudFormationInit.from_config_sets(
config_sets={'default': ['init']},
configs={
'init': aws_ec2.InitConfig([
aws_ec2.InitPacakge.python(package_name='boto3'),
aws_ec2.InitFile.from_asset('/usr/local/myscript.sh', 'scripts/myscript.sh')
])
}
), ...
)
this will result in the EC2 instance having the appropriate cfn-init scripting in the UserData section automatically, and load the scripts/myscript.sh into the instance with the boto3 python package installed.
More information on AWS::CloudFormation::Init - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-init.html
I think you are looking for UserData: https://docs.aws.amazon.com/cdk/api/latest/docs/#aws-cdk_aws-ec2.Instance.html#userdata
With UserData you can add commands which your EC2 Instance executes at (by default only first) launch.
E.g. for installing apache:
const userData = ec2.UserData.forLinux();
userData.addCommands('yum install -y httpd');
new ec2.Instance(this, 'instance', {
userData: userData,
...
});
Another way would be to create a custom AMI which you use to start your EC2 instance. That AMI could have all packaged pre-installed.
Related
I'm new to serverless,
So far I was be able to deploy and use .env for the app.
then, under provider in stage property in serverless.yml file, I change it to different stage. I also made new.env.{stage}.
after re-deploy using sls deploy, It still reads the default .env file.
the documentation states:
The framework looks for .env and .env.{stage} files in service directory and then tries to load them using dotenv. If .env.{stage} is found, .env will not be loaded. If stage is not explicitly defined, it defaults to dev.
So, I still don't understand "If stage is not explicitly defined, it defaults to dev". How to explicitly define it?
The dotenv File is choosen based on your stage property configuration. You need to explicitly define the stage property in your serverless.yaml or set it within your deployment command.
This will use the .env.dev file
useDotenv: true
provider:
name: aws
stage: dev # dev [default], stage, prod
memorySize: 3008
timeout: 30
Or you set the stage property via deploy command.
This will use the .env.prod file
sls deploy --stage prod
In your serverless.yml you need to define the stage property inside the provider object.
Example:
provider:
name: aws
[...]
stage: prod
As Feb 2023 I'm going to attempt to give my solution. I'm using the Nx tootling for monorepo (this shouldn't matter but just in case) and I'm using the serverless.ts instead.
I see the purpose of this to be to enhance the developer experience in the sense that it is nice to just nx run users:serve --stage=test (in my case using Nx) or sls offline --stage=test and serverless to be able to load the appropriate variables for that specific environment.
Some people went the route of using several .env.<stage> per environment. I tried to go this route but because I'm not that good of a developer I couldn't make it work. The approach that worked for the was to concatenate variable names inside the serverless.ts. Let me explain...
I'm using just one .env file instead but changing variable names based on the --stage. The magic is happening in the serverless.ts
// .env
STAGE_development=test
DB_NAME_development=mycraftypal
DB_USER_development=postgres
DB_PASSWORD_development=abcde1234
DB_PORT_development=5432
READER_development=localhost // this could be aws rds uri per db instances
WRITER_development=localhost // this could be aws rds uri per db instances
# TEST
STAGE_test=test
DB_NAME_test=mycraftypal
DB_USER_test=postgres
DB_PASSWORD_test=abcde1234
DB_PORT_test=5433
READER_test=localhost // this could be aws rds uri per db instances
WRITER_test=localhost // this could be aws rds uri per db instances
// serverless.base.ts or serverless.ts based on your configuration
...
useDotenv: true, // this property is at the root level
...
provider: {
...
stage: '${opt:stage, "development"}', // get the --stage flag value or default to development
...,
environment: {
STAGE: '${env:STAGE_${self:provider.stage}}}',
DB_NAME: '${env:DB_NAME_${self:provider.stage}}',
DB_USER: '${env:DB_USER_${self:provider.stage}}',
DB_PASSWORD: '${env:DB_PASSWORD_${self:provider.stage}}',
READER: '${env:READER_${self:provider.stage}}',
WRITER: '${env:WRITER_${self:provider.stage}}',
DB_PORT: '${env:DB_PORT_${self:provider.stage}}',
AWS_NODEJS_CONNECTION_REUSE_ENABLED: '1',
}
...
}
When one is utilizing the useDotenv: true, serverless loads your variables from the .env and puts them in the env variable so you can access them env:STAGE.
Now I can access the variable with dynamic stage like so ${env:DB_PORT_${self:provider.stage}}. If you look at the .env file each variable has the ..._<stage> at the end. In this way I can retrieve dynamically each value.
I'm still figuring it out since I don't want to have the word production in my url but still get the values dynamically and since I'm concatenating this value ${env:DB_PORT_${self:provider.stage}}... then the actual variable becomes DB_PORT_ instead of DB_PORT.
I am new to the world of terraform. I am trying to use terraformer on a GCP project, but keep getting plugin not found:
terraformer import google --resources=gcs,forwardingRules,httpHealthChecks --connect=true --
regions=europe-west1,europe-west4 --projects=myproj
2021/02/12 08:17:03 google importing project myproj region europe-west1
2021/02/12 08:17:04 google importing... gcs
2021/02/12 08:17:05 open \.terraform.d/plugins/windows_amd64: The system cannot find the path specified.
Prior to the above, installed terraformer thus: choco install terraformer I downloaded the Google provider, terraform-provider-google_v3.56.0_x5.exe, placed it in my terraform dir, and did terraform init. I checked %AppData\Roaming\terraform.d\plugins\windows_amd64 and the plugins\windows_amd64 did not exist, so I created it, dropped in the Google exe and redid the init.
My main.tf was:
provider "google" {
project = "{myproj"
region = "europe-west2"
zone = "europe-west2-a"
}
Could it be the chocolatey install causing this issue? I have never used it before but the alternative install looks daunting!
If you install from Chocolatey, it seems it looks for the provider files at C:\.terraform.d\plugins\windows_amd64. If you create those directories and drop the terraform-provider-xxx.exe file in there it will be picked up.
I would also suggest running it with the --verbose flag.
Remember to execute "terraform init"
Once you install the terraformer, check this
Create the versions.tf file
versions.tf example
terraform {
required_providers {
google = {
source = "hashicorp/aws"
}
}
required_version = ">= 0.13"
}
Execute terraform init
Start the import process, something like:
terraformer import aws --profile [my-aws-profile] --resources="*" --regions=eu-west-1
Where --resources="*" will get everything from aws, but you could specify route53 or other resources
terraformer import aws --profile [my-aws-profile] --resources="route53" --regions=eu-west-1
The daunting instructions worked!
Run git clone <terraformer repo>
Run go mod download
Run go build -v for all providers OR build with one provider go run build/main.go {google,aws,azure,kubernetes and etc}
Run terraform init against an versions.tf file to install the plugins required for your platform. For example, if you need plugins for the google provider, versions.tf should contain:
terraform {
required_providers {
google = {
source = "hashicorp/google"
}
}
required_version = ">= 0.13"
}
making sure to run that version and not the chocolatey one.
go mod download the flag -x will give output while that command runs, otherwise you might think nothing is happening.
Old boto is used in ansible aws ec2 module, which is outdated. last commit 2018y. How do u provision instances in new regions?
my current version of ansible 2.9.6, but in 2.10 & 2.11 changelog there are nothing about chenge to boto3
region list in boto:
[eu-west-1, eu-west-2, cn-north-1, us-east-2, us-gov-west-1, ca-central-1, ap-southeast-2, us-west-2, ap-southeast-1, us-east-1, sa-east-1, us-west-1, ap-northeast-2, eu-central-1, ap-south-1, ap-northeast-1]
region list using boto3:
['af-south-1', 'ap-east-1', 'ap-northeast-1', 'ap-northeast-2', 'ap-south-1', 'ap-southeast-1', 'ap-southeast-2', 'ca-central-1', 'eu-central-1', 'eu-north-1', 'eu-south-1', 'eu-west-1', 'eu-west-2', 'eu-west-3', 'me-south-1', 'sa-east-1', 'us-east-1', 'us-east-2', 'us-west-1', 'us-west-2']
Here is a quote from https://docs.ansible.com/ansible/latest/collections/amazon/aws/ec2_module.html
Note: This module uses the older boto Python module to interact with the EC2 API. amazon.aws.ec2 will still receive bug fixes, but no new features. Consider using the amazon.aws.ec2_instance module instead.
I'm trying to run a kubeflow pipeline setup and I have several environements (dev, staging, prod).
In my pipeline I'm using kfp.components.func_to_container_op to get a pipeline task instance (ContainerOp), and then execute it with the appropriate arguments that allows it to integrate with my s3 bucket:
from utils.test import test
test_op = comp.func_to_container_op(test, base_image='my_image')
read_data_task = read_data_op(
bucket,
aws_key,
aws_pass,
)
arguments = {
'bucket': 's3',
'aws_key': 'key',
'aws_pass': 'pass',
}
kfp.Client().create_run_from_pipeline_func(pipeline, arguments=arguments)
Each one of the environments is using different credentials to connect to it and those credentials are being passed in the function:
def test(s3_bucket: str, aws_key: str, aws_pass: str):
....
s3_client = boto3.client('s3', aws_access_key_id=aws_key, aws_secret_access_key=aws_pass)
s3_client.upload_file(from_filename, bucket_name, to_filename)
so for each environment I need to update the arguments to contain the correct credentials and it makes it very hard to maintain since each time that I want to update from dev to stg to prod I can't simply copy the code.
My question is what is the best approach to pass those credentials?
Ideally you should push any env-specific configurations as close to the cluster as possible (as far away from components).
You can create Kubernetes secret in each environemnt with different creadentials. Then use that AWS secret in each task:
from kfp import aws
def my_pipeline():
...
conf = kfp.dsl.get_pipeline_conf()
conf.add_op_transformer(aws.use_aws_secret('aws-secret', 'AWS_ACCESS_KEY_ID', 'AWS_SECRET_ACCESS_KEY'))
Maybe boto3 can auto-load the credentials using the secret files and the environment variables.
At least all GCP libraries and utilities do that with GCP credentials.
P.S. It's better to create issues in the official repo: https://github.com/kubeflow/pipelines/issues
I followed the following tutorial to create a Lambda deploy pipeline using CDK. When I try to keep everything in the same account it works well.
https://docs.aws.amazon.com/cdk/latest/guide/codepipeline_example.html
But my scenario is slightly different from the example because it involves two AWS accounts instead one. I maintain application source code and pipeline
in the OPS account and this pipeline will deploy the Lambda application to the UAT account.
OPS Account (12345678) - CodeCommit repo & CodePipeline
UAT Account (87654321) - Lambda application
As per the aws following aws documentation (Cross-account actions section) I made the following changes to source code.
https://docs.aws.amazon.com/cdk/api/latest/docs/aws-codepipeline-actions-readme.html
Lambda stack expose deploy action role as follows
export class LambdaStack extends cdk.Stack {
public readonly deployActionRole: iam.Role;
constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
...
this.deployActionRole = new iam.Role(this, 'ActionRole', {
assumedBy: new iam.AccountPrincipal('12345678'), //pipeline account
// the role has to have a physical name set
roleName: 'DeployActionRole',
});
}
}
In the pipeline stack,
new codePipeline.Pipeline(this, 'MicroServicePipeline', {
pipelineName: 'MicroServicePipeline',
stages: [
{
stageName: 'Deploy',
actions: [
new codePipelineAction.CloudFormationCreateUpdateStackAction({
role: props.deployActionRole,
....
})
]
}
]
});
Following is how I initiate stacks
const app = new cdk.App();
const opsEnv: cdk.Environment = {account: '12345678', region: 'ap-southeast-2'};
const uatEnv: cdk.Environment = {account: '87654321', region: 'ap-southeast-2'};
const lambdaStack = new LambdaStack(app, 'LambdaStack', {env: uatEnv});
const lambdaCode = lambdaStack.lambdaCode;
const deployActionRole = lambdaStack.deployActionRole;
new MicroServicePipelineStack(app, 'MicroServicePipelineStack', {
env: opsEnv,
stackName: 'MicroServicePipelineStack',
lambdaCode,
deployActionRole
});
app.synth();
AWS credentials profiles looks liks
[profile uatadmin]
role_arn=arn:aws:iam::87654321:role/PigletUatAdminRole
source_profile=opsadmin
region=ap-southeast-2
when I run cdk diff or deploy I get an error saying,
➜ infra git:(master) ✗ cdk diff MicroServicePipelineStack --profile uatadmin
Including dependency stacks: LambdaStack
Stack LambdaStack
Need to perform AWS calls for account 87654321, but no credentials have been configured.
What have I done wrong here? Is it my CDK code or is it the way I have configured my AWS profile?
Thanks,
Kasun
The problem is with your AWS CLI configuration. You cannot use the CDK CLI natively to deploy resources in two separate accounts with one CLI command. There is a recent blog post on how to tell CDK which credentials to use, depending on the stack environment parameter:
https://aws.amazon.com/blogs/devops/cdk-credential-plugin/
The way we use it is to deploy stacks into separate accounts with multiple CLI commands specifying the required profile. All parameters that need to be exchanged (such as the location of your lambdaCode) is passed via e.g. environment variables.
Just try to use using the environment variables:
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html
Or
~/.aws/credentials
[default]
aws_access_key_id=****
aws_secret_access_key=****
~/.aws/config
[default]
region=us-west-2
output=json
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html
It works for me.
I'm using cdk version 1.57.0
The issue is in the fact that you have resources that exist in multiple accounts and hence there are different credentials required to create those resources. However, CDK does not understand natively how to get credentials for those different accounts or when to swap between the different credentials. One way to fix this is to use cdk-assume-role-credential-plugin, which will allow you to use a single CDK deploy command to deploy to many different accounts.
I wrote a detailed tutorial here: https://johntipper.org/aws-cdk-cross-account-deployments-with-cdk-pipelines-and-cdk-assume-role-credential-plugin/