I'm running aws serverless using aws cdk and aws-sdk.
I wrote my code and then ran the following commands:
cdk synth
cdk deploy
Now I update the code locally on my machine and want to push the changes.
Which command/s should I run now?
Should I run cdk destroy in between?
Thanks
Running cdk deploy will first synthesize the stacks and then deploy the changes. No need to run synth prior.
Deploying will apply the current stack and destroy any recourses that are no longer in the code, so no need to run destroy first.
Use cdk watch. Cdk will observe the files you specified in your cdk.json file and automatically deploy your changes, which also makes it much faster.
Here is the cdk watch documentation:
https://cdkworkshop.com/20-typescript/30-hello-cdk/300-cdk-watch.html#cdk-watch (start here)
https://aws.amazon.com/blogs/developer/increasing-development-speed-with-cdk-watch/
Just run the same commands again
cdk synth
cdk deploy
Related
Host Machine: Mac M1
SAM version: 1.70.0
CDK version: 2.60.0 (build 2d40d77)
In our api_stack.py file we have a simple setup of
API Gateway REST endpoints
Lambda functions triggered by API Gateway
The entire codebase was initially setup with cdk init
When we run cdk synth, we get a files generated in the folder cdk.out including the api.template.json which is the CloudFormation template file.
Following commands all work:
cdk synth
cdk deploy
However when we run sam local invoke or sam local start-api, I get an error that looks like below.
% sam local invoke GetUserTemporary -t cdk.out/api.template.json -e events/get_user/prod-user.json
Invoking get_user_temporary.handler.handle (python3.9)
DependenciesLayerDF300E31 is a local Layer in the template
Local image was not found.
Building image.......................................................................................................................................................................................................................................................................................................................
Failed to build Docker Image
NoneType: None
Error: Error building docker image: The command '/bin/sh -c mv /var/rapid/aws-lambda-rie-x86_64 /var/rapid/aws-lambda-rie && chmod +x /var/rapid/aws-lambda-rie' returned a non-zero code: 1
This used to work just about 1 week ago and now it is showing this error all of sudden. Exact same codebase is git cloned in my co-workers' machines.
Windows
Mac M1
And this all works for them. I try to do this in my M1 laptop and it fails now.
It turns out you have to specifically specify what architecture you are using when defining your lambda functions in your CDK Python source code. You can achieve this by adding the architecture= argument like below.
fn = _lambda.Function(
self,
func_name,
runtime=_lambda.Runtime.PYTHON_3_9,
code=_lambda.Code.from_asset('api/lambda_fns'),
handler=f'{func_path}.handle',
layers=[self.layer],
role=role,
architecture=_lambda.Architecture.X86_64,
)
I am tring to run my go project using amazon lambda, this is my current main.go
https://gist.github.com/krakiun/61e4e4dc5ab91f557e481f0230ed3ba0
I tried several methods, but none worked
How i can make run this project in lambda, in this moment if i run using router.Run(cfg.HTTP.ListenAddr) is working without any error,
with this log.Fatal(gateway.ListenAndServe(cfg.HTTP.ListenAddr, router)) is die with this error :
expected AWS Lambda environment variables [_LAMBDA_SERVER_PORT AWS_LAMBDA_RUNTIME_API] are not defined
exit status 1
How i can fix my code to run in aws lambda ?
You have to separate the environment
the below code for run at local
router.Run(cfg.HTTP.ListenAddr)
the lambda can only run on the AWS Lambda Function. You must deploy it to lambda
lambda.Start(router)
You can check the example at https://maxrohde.com/2021/05/01/lambda-go-starter-project/. And the source code is here: https://github.com/mxro/go-lambda-starter-project/tree/main/packages/lambda-go-gin
I'm developing an application with micronaut using SAM CLI to deploy it on AWS Lambda. As I was including dependencies and developing new features, the function packages got bigger an bigger (now they are around 250MB). This makes deployment take a while.
On top of that every time I edit template.yaml and then run sam build && sam deploy to try a new configuration on S3, RDS, etc... I have to wait for gradle to build the function again (even though it's unchanged since the last deployment) and upload the whole package to S3.
As I'm trying to configure this application with many trials and errors on SAM, waiting for this process to complete just to get an error because of some misconfiguration is getting quite counterproductive.
Also my SAM s3 bcuket is at 10GB size after just a single day of work. This may get expensive on the long run.
Is there a way to avoid those gradle rebuilds and reuploads when teh function code is unchanged?
If you are only updating the template.yml file, you could copy the new version to ./.aws-sam/build folder and then run sam deploy
$ cp template.yml ./.aws-sam/build/template.yml
$ sam deploy
If you are editing a lambda you could try to update the function code by itself (after you create it in the template and deploy of course). That can be done via the AWS CLI update-function-code command:
rm index.zip
cd lambda
zip –X –r ../index.zip *
cd ..
aws lambda update-function-code --function-name MyLambdaFunction --zip-file fileb://index.zip
more info can be found here:
Alexa Blogs - Publishing Your Skill Code to Lambda via the Command Line Interface
AWS CLI Command Reference - lambda - update-function-code
my SAM s3 bcuket is at 10GB size
Heh. Yea start deleting stuff. Maybe you can write a script using aws s3?
I have a rails application that is deployed on AWS EC2 instance with CodePipeline. I have added the Build stage in the pipeline using AWS CodeBuild to build test my code.
I have no idea about where to add below rails command to execute whenever code auto-deploy using the pipeline.
bundle install
rake db:migrate, create, assets compile
Restart sidekiq
You need to use CodeDeploy service as part of your CodePipeline. The pipeline will have two stages, one source stage (taking source from GitHub or CodeCommit etc) and second deploy stage (deploy to EC2 using CodeDeploy).
The CodeDeploy agent will be running on the EC2 instance and will take deployment command from the service. CodeDeploy deployments need an AppSpec file that provides the details of where to copy the source file on the EC2 instance and then run some scripts on the instance ("hooks") where you will do the commands like 'bundle install' or 'restart sidekik' etc.
Instead of me trying to list every step, I found a few resources that may get you started. Try the first tutorial which will help you understand the complete picture (CodeDeploy + CoedPipeline):
https://docs.aws.amazon.com/codepipeline/latest/userguide/tutorials-simple-codecommit.html
https://dev.to/mknycha/deploying-ruby-app-using-aws-codedeploy-to-ec2-how-to-do-it-with-bundler-49gg
How to write appspec.yml for Ruby on Rails on AWS CodeDeploy
Maybe I misunderstand the purpose of packaging but it doesn't seem to helpful in creating an artifact for production deployment because it only packages code. It leaves out the conf, data, and other directories that make the kedro project reproducible.
I understand that I can use docker or airflow plugins for deployment but what about deploying to databricks. Do you have any advice here?
I was thinking about making a wheel that could be installed on the cluster but I would need to package the conf first. Another option is to just sync a git workspace to the cluster and run kedro via a notebook.
Any thoughts on a best practice?
If you are not using docker and just using kedro to deploy directly on a databricks cluster. This is how we have been deploying kedro to databricks.
CI/CD pipeline builds using kedro package. Creates a wheel file.
Upload dist and conf to dbfs or AzureBlob file copy (if using Azure Databricks)
This will upload everything to databricks on every git push
Then you can have a notebook with the following:
You can have an init script in databricks something like:
from cargoai import run
from cargoai.pipeline import create_pipeline
branch = dbutils.widgets.get("branch")
conf = run.get_config(
project_path=f"/dbfs/project_name/build/cicd/{branch}"
)
catalog = run.create_catalog(config=conf)
pipeline = create_pipeline()
Here conf, catalog, and pipeline will be available
Call this init script when you want to run a branch or a master branch in production like: %run "/Projects/InitialSetup/load_pipeline" $branch="master"
For development and testing, you can run specific nodespipeline = pipeline.only_nodes_with_tags(*tags)
Then run a full or a partial pipeline with just SequentialRunner().run(pipeline, catalog)
In production, this notebook can be scheduled by databricks. If you are on Azure Databricks, you can use Azure Data Factory to schedule and run this.
I found the best option was to just use another tool for packaging, deploying, and running the job. Using mlflow with kedro seems like a good fit. I do most everything in Kedro but use MLFlow for the packaging and job execution: https://medium.com/#QuantumBlack/deploying-and-versioning-data-pipelines-at-scale-942b1d81b5f5
name: My Project
conda_env: conda.yaml
entry_points:
main:
command: "kedro install && kedro run"
Then running it with:
mlflow run -b databricks -c cluster.json . -P env="staging" --experiment-name /test/exp
So there is a section of the documentation that deals with Databricks:
https://kedro.readthedocs.io/en/latest/04_user_guide/12_working_with_databricks.html
The easiest way to get started will probably be to sync with git and run via a Databricks notebook. However, as mentioned, there are other ways using the ".whl" and referencing the "conf" folder.