I was learning terraform and was asked to provision it for CI/CD pipeline at gitlab.
My doubt is that ,
Let's say a lambda function is already running/live.
How can I provision it using terraform ?
Should I use data block to consume the running aws lambda?
Or this isn't how it works ! I am not sure how can we do this.
I searched the docs which isn't supporting this use case.
So with the Lambda function that is already running, basically here you have two use cases:
whether you want to add further changes/updates to that Lambda later on using Terraform. In this case, you need to import it to your terraform code, and all the changes you add to that Lambda can be deployed via your CI/CD pipeline, e.g.:
terraform import aws_lambda_function.my_lambda existing_lambda_function_name
Note: Please note that the my_lambda function is your terraform block of code that is defining the exact Lambda that is already running, this is to match the existing resource with your code Terraform, to then be added to the state. I hope that it is clear
or you simply just need some outputs of that Lambda to be used as inputs to other services, here you can simply just keep the Lambda up and running and use Terraform data source, e.g.:
data "aws_lambda_function" "existing_lambda" {
function_name = var.function_name
}
And somewhere else in your code you can use it as follows:
function_name = data.aws_lambda_function.existing_lambda
I hope this was helpful
Related
I'm using CDK to instantiate a Queue and a Lambda Function.
Lambda function requires QueueURL in order to push messages into it.
QueueURL is not fixed, it changes when the stack is re-created.
I have two options:
Pass QueueURL as an env variable to Lambda in CDK.
Create a cfnOutput with QueueURL and read it from the Lambda.
If I use option 2, Lambda will have to make an API call every time it runs to get the URL.
Are these the only options?
What is the recommended approach for this?
Thanks!
Option 1 is recommended. If the value changes for any reason, the lambda will also be updated accordingly automatically. It also ensures that the lambda will be created after the queue, as it creates an implicit dependency.
Don't forget to grant your lambda access to the queue with myQueue.grantSendMessages(myLambda);
I want to deploy my infrastructure in different AWS environments (dev, prod, qa).
That deployment creates a few EC2 instances from a custom AMI. When deployed, instances are in the "running" state. I understand this seems to be related to some constraint in the EC2 API. However, I don't necessarily want my instances started, depending on context. Sometimes, I just want the instances to be created, and they will be started later on. I guess this is a quite common scenario.
Reading the few related issues/requests on Hashicorp's github, makes me think so:
Terraform aws instance state change
Stop instances
aws_instance should allow to specify the instance state
There must be some TerraForm based solution which doesn't require to rely on AWS CLI / CDK or lambda, right? Something in the TerraForm script that, for example, would stop the instance right after its creation.
My google foo didn't help me much here. Any help / suggestion for dealing with that scenario is welcome.
Provisioning a new instance automatically puts it in a 'started' state.
As Marcin suggested, you can use user data scripts, here's some psuedo user data script. For you to figure out the actual implementation ;)
#!/bin/bash
get instance id, pass it to the subsequent line
aws ec2 stop-instances --instance-ids i-1234567890abcdef0
You can read about running scripts as part of the bootstrapping here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html
Basically its all up to your use case. We don't do this generally. Still, if you want to provision your EC2 instances and need to put them in stopped state, as bschaatsbergen suggested, you can use the user_data in Terraform. Make sure to attach the role with relevant permission.
#!/bin/bash
INSTANCE_ID=`curl -s http://169.254.169.254/latest/meta-data/instance-id/`
REGION=`curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone | sed 's/.$//'`
aws ec2 stop-instances --instance-ids $INSTANCE_ID --region $REGION
As already stated by others, you cannot just "create" instances, they will be in "started" state.
Rather I would ask what is the exact use case here :
Sometimes, I just want the instances to be created, and they will be started later on.
Why you have to create instances now and use them later? Can't they be created exactly when they are required? Any specific requirement to keep them initialized before they are used? Or the instances take time to start?
I have a CodePipeline with GitHub as a source, set up. I am trying to, without a success, pass a single secret parameter( in this case a Stripe secret key, currently defined in an .env file -> explaination down below ) to a specific Lambda during a Deployment stage in CodePipeline's execution.
Deployment stage in my case is basically a CodeBuild project that runs the deployment.sh script:
#! /bin/bash
npm install -g serverless#1.60.4
serverless deploy --stage $env -v -r eu-central-1
Explanation:
I've tried doing this with serverless-dotenv-plugin, which serves the purpose when the deployment is done locally, but when it's done trough CodePipeline, it returns an error on lambda's execution, and with a reason:
Since CodePipeline's Source is set to GitHub (.env file is not commited), whenever a change is commited to a git repository, CodePipeline's execution is triggered. By the time it reaches deployment stage, all node modules are installed (serverless-dotenv-plugin along with them) and when serverless deploy --stage $env -v -r eu-central-1 command executes serverless-dotenv-plugin will search for .env file in which my secret is stored, won't find it since there's no .env file because we are out of "local" scope, and when lambda requiring this secret triggers it will throw an error looking like this:
So my question is, is it possible to do it with dotenv/serverless-dotenv-plugin, or should that approach be discarded? Should I maybe use SSM Parameter Store or Secrets Manager? If yes, could someone explain how? :)
So, upon further investigation of this topic I think I have the solution.
SSM Parameter Store vs Secrets Manager is an entirely different topic, but for my purpose, SSM Paremeter Store is a choice that I chose to go along with for this problem. And basically it can be done in 2 ways.
1. Use AWS Parameter Store
Simply by adding a secret in your AWS Parameter Store Console, then referencing the value in your serverless.yml as a Lambda environement variable. Serverless Framework is able to fetch the value from your AWS Parameter Store account on deploy.
provider:
environement:
stripeSecretKey: ${ssm:stripeSecretKey}
Finally, you can reference it in your code just as before:
const stripe = Stripe(process.env.stripeSecretKey);
PROS: This can be used along with a local .env file for both local and remote usage while keeping your Lambda code the same, ie. process.env.stripeSecretKey
CONS: Since the secrets are decrypted and then set as Lambda environment variables on deploy, if you go to your Lambda console, you'll be able to see the secret values in plain text. (Which kinda indicates some security issues)
That brings me to the second way of doing this, which I find more secure and which I ultimately choose:
2. Store in AWS Parameter Store, and decrypt at runtime
To avoid exposing the secrets in plain text in your AWS Lambda Console, you can decrypt them at runtime instead. Here is how:
Add the secrets in your AWS Parameter Store Console just as in the above step.
Change your Lambda code to call the Parameter Store directly and decrypt the value at runtime:
import stripePackage from 'stripe';
const aws = require('aws-sdk');
const ssm = new aws.SSM();
const stripeSecretKey = ssm.getParameter(
{Name: 'stripeSecretKey', WithDecryption: true}
).promise();
const stripe = stripePackage(stripeSecretKey.Parameter.Value);
(Small tip: If your Lambda is defined as async function, make sure to use await keyword before ssm.getParameter(...).promise(); )
PROS: Your secrets are not exposed in plain text at any point.
CONS: Your Lambda code does get a bit more complicated, and there is an added latency since it needs to fetch the value from the store. (But considering it's only one parameter and it's free, it's a good trade-off I guess)
For the conclusion I just want to mention that all this in order to work will require you to tweak your lambda's policy so it can access Systems Manager and your secret that's stored in Parameter Store, but that's easily inspected trough CloudWatch.
Hopefully this helps someone out, happy coding :)
When I am creating lambda via pycharm AWS Toolkit, it appends a name to the lambda function I create. Therefore name of a function is something like "hello-world-12MJU0DB7Y99B". While it is OK for custom functions, it is not something that I can easily use to automate multi-account AWS environment. I need the name of the function to be "hello-world".
Is there anyway to specify exact function name?
The solution was indeed as Milan cermak suggested, unlike in serverless aws toolkit requires name variable to be set. Once that is done the name is as expected!
I have been trying to create my first Lambda function but every time I run it I get "Unable to import module 'app'".
I have been following the tutorial written by Amazon to create a function that connects to a Amazon RDS db so my Lambda function must include the PyMySQL library.
https://docs.aws.amazon.com/lambda/latest/dg/vpc-rds-create-rds-mysql.html
I am pretty sure my issue is I am not zipping the contents of the directories correctly therefore the Lambda cannot find my app.py file in order to find the handler and execute.
I have followed the steps to create a deployment package, I have looked at online, create new linux machines, zip in virtual env etc but nothing is working and I get the same error back each time.
Can someone please write down step by step with full commands as what I need to do in order to create the deployment package correctly, or what they do in order to do get their python functions to run in Lambda
Thanks in advance!!