Endpoint URL for DynamoDB inside localstack's Lambda function - aws-lambda

I'm using localstack for local development. I have a DynamoDB table named readings and I'd like to insert items from a lambda function.
I have deployed simple lambda function in python runtime:
import os
import boto3
def lambda_handler(events, context):
DYNAMODB_ENDPOINT_URL = os.environ.get("DYNAMODB_ENDPOINT_URL")
dynamodb = boto3.resource("dynamodb", endpoint_url=DYNAMODB_ENDPOINT_URL)
readings_table = dynamodb.Table(DYNAMODB_READINGS_TABLE_NAME)
readings_table.put_item(Item={"reading_id": "10", "other": "test"})
But I'm getting error: [ERROR] EndpointConnectionError: Could not connect to the endpoint URL: "http://localstack:4569/"
I've tried combinations of localhost and localstack along with ports: 4566 and 4569. All of them fail.
Here's my docker-compse service that I use to start localstack
localstack:
image: localstack/localstack:0.11.2
ports:
- 4566:4566
- 8080:8080
environment:
SERVICES: "dynamodb,sqs,lambda,iam"
DATA_DIR: "/tmp/localstack/data"
PORT_WEB_UI: "8080"
LOCALSTACK_HOSTNAME: localstack
LAMBDA_EXECUTOR: docker
AWS_ACCESS_KEY_ID: "test"
AWS_SECRET_ACCESS_KEY: "test"
AWS_DEFAULT_REGION: "us-east-1"
volumes:
- localstack_volume:/tmp/localstack/data
- /var/run/docker.sock:/var/run/docker.sock
# When a container is started for the first time, it will execute files with extensions .sh that are found in /docker-entrypoint-initaws.d.
# Files will be executed in alphabetical order. You can easily create aws resources on localstack using `awslocal` (or `aws`) cli tool in the initialization scripts.
# source: https://github.com/localstack/localstack/pull/1018/files#diff-04c6e90faac2675aa89e2176d2eec7d8R185
- ./localstack-startup-scripts/:/docker-entrypoint-initaws.d/
What would be the correct endpoint url that I have to set inside in my lambda so that I can send requests to localstack's DynamoDB?

According to the docs LOCALSTACK_HOSTNAME is a read-only env var:
LOCALSTACK_HOSTNAME: Name of the host where LocalStack services are available. Use this hostname as endpoint (e.g., http://${LOCALSTACK_HOSTNAME}:4566) in order to access the services from within your Lambda functions (e.g., to store an item to DynamoDB or S3 from a Lambda).

try with
ports:
- "0.0.0.0:4566-4599:4566-4599"
Hope it helps

By following Robert Taylor's answer, I was able to get the following working in my Java Lambda (this should really be a comment in that answer, but code formatting for big snippet is not adequate in comment, so I'm sharing this as a separate answer to make people's life easier):
var url = System.getenv("LOCALSTACK_HOSTNAME");
var credentials = new BasicAWSCredentials("mock_access_key", "mock_secret_key");
var ep = new AwsClientBuilder.EndpointConfiguration(String.format("http://%s:4566",url), "us-east-1");
s3 = AmazonS3ClientBuilder.standard()
.withEndpointConfiguration(ep)
.withPathStyleAccessEnabled(true)
.withCredentials(new AWSStaticCredentialsProvider(credentials))
.build();
dynamoDBMapper = new DynamoDBMapper(
AmazonDynamoDBClientBuilder.standard()
.withEndpointConfiguration(ep)
.withCredentials(new AWSStaticCredentialsProvider(credentials))
.build()
);

Related

Accessing AWS Keyspaces service from an AWS Lambda through boto3

I am trying to access the keyspaces service through boto3 since it is referenced in the official doc ... However when I try from a sample lambda:
def lambda_handler(event: Dict[str, Any], context: Any):
client = boto3.client('keyspaces')
print(client.get_table(keyspaceName='mykeyspace', tableName='mytable'))
return {
'status': 'completed'
}
I get the following error message, which basically says that "keyspaces" are not available.
{
"errorMessage": "Unknown service: 'keyspaces'. Valid service names are: accessanalyzer, account, acm, acm-pca, alexaforbusiness, amp, amplify, amplifybackend, amplifyuibuilder, apigateway, apigatewaymanagementapi, apigatewayv2, appconfig, appconfigdata, appflow, appintegrations, application-autoscaling, application-insights, applicationcostprofiler, appmesh, apprunner, appstream, appsync, athena, auditmanager, autoscaling, autoscaling-plans, backup, backup-gateway, batch, braket, budgets, ce, chime, chime-sdk-identity, chime-sdk-meetings, chime-sdk-messaging, cloud9, cloudcontrol, clouddirectory, cloudformation, cloudfront, cloudhsm, cloudhsmv2, cloudsearch, cloudsearchdomain, cloudtrail, cloudwatch, codeartifact, codebuild, codecommit, codedeploy, codeguru-reviewer, codeguruprofiler, codepipeline, codestar, codestar-connections, codestar-notifications, cognito-identity, cognito-idp, cognito-sync, comprehend, comprehendmedical, compute-optimizer, config, connect, connect-contact-lens, connectparticipant, cur, customer-profiles, databrew, dataexchange, datapipeline, datasync, dax, detective, devicefarm, devops-guru, directconnect, discovery, dlm, dms, docdb, drs, ds, dynamodb, dynamodbstreams, ebs, ec2, ec2-instance-connect, ecr, ecr-public, ecs, efs, eks, elastic-inference, elasticache, elasticbeanstalk, elastictranscoder, elb, elbv2, emr, emr-containers, es, events, evidently, finspace, finspace-data, firehose, fis, fms, forecast, forecastquery, frauddetector, fsx, gamelift, glacier, globalaccelerator, glue, grafana, greengrass, greengrassv2, groundstation, guardduty, health, healthlake, honeycode, iam, identitystore, imagebuilder, importexport, inspector, inspector2, iot, iot-data, iot-jobs-data, iot1click-devices, iot1click-projects, iotanalytics, iotdeviceadvisor, iotevents, iotevents-data, iotfleethub, iotsecuretunneling, iotsitewise, iotthingsgraph, iottwinmaker, iotwireless, ivs, kafka, kafkaconnect, kendra, kinesis, kinesis-video-archived-media, kinesis-video-media, kinesis-video-signaling, kinesisanalytics, kinesisanalyticsv2, kinesisvideo, kms, lakeformation, lambda, lex-models, lex-runtime, lexv2-models, lexv2-runtime, license-manager, lightsail, location, logs, lookoutequipment, lookoutmetrics, lookoutvision, machinelearning, macie, macie2, managedblockchain, marketplace-catalog, marketplace-entitlement, marketplacecommerceanalytics, mediaconnect, mediaconvert, medialive, mediapackage, mediapackage-vod, mediastore, mediastore-data, mediatailor, memorydb, meteringmarketplace, mgh, mgn, migration-hub-refactor-spaces, migrationhub-config, migrationhubstrategy, mobile, mq, mturk, mwaa, neptune, network-firewall, networkmanager, nimble, opensearch, opsworks, opsworkscm, organizations, outposts, panorama, personalize, personalize-events, personalize-runtime, pi, pinpoint, pinpoint-email, pinpoint-sms-voice, polly, pricing, proton, qldb, qldb-session, quicksight, ram, rbin, rds, rds-data, redshift, redshift-data, rekognition, resiliencehub, resource-groups, resourcegroupstaggingapi, robomaker, route53, route53-recovery-cluster, route53-recovery-control-config, route53-recovery-readiness, route53domains, route53resolver, rum, s3, s3control, s3outposts, sagemaker, sagemaker-a2i-runtime, sagemaker-edge, sagemaker-featurestore-runtime, sagemaker-runtime, savingsplans, schemas, sdb, secretsmanager, securityhub, serverlessrepo, service-quotas, servicecatalog, servicecatalog-appregistry, servicediscovery, ses, sesv2, shield, signer, sms, sms-voice, snow-device-management, snowball, sns, sqs, ssm, ssm-contacts, ssm-incidents, sso, sso-admin, sso-oidc, stepfunctions, storagegateway, sts, support, swf, synthetics, textract, timestream-query, timestream-write, transcribe, transfer, translate, voice-id, waf, waf-regional, wafv2, wellarchitected, wisdom, workdocs, worklink, workmail, workmailmessageflow, workspaces, workspaces-web, xray",
"errorType": "UnknownServiceError",
"stackTrace": [
" File \"/var/task/lambda_function.py\", line 39, in lambda_handler\n client = boto3.client(\"keyspaces\")\n",
" File \"/var/runtime/boto3/__init__.py\", line 93, in client\n return _get_default_session().client(*args, **kwargs)\n",
" File \"/var/runtime/boto3/session.py\", line 270, in client\n return self._session.create_client(\n",
" File \"/var/runtime/botocore/session.py\", line 841, in create_client\n client = client_creator.create_client(\n",
" File \"/var/runtime/botocore/client.py\", line 83, in create_client\n service_model = self._load_service_model(service_name, api_version)\n",
" File \"/var/runtime/botocore/client.py\", line 150, in _load_service_model\n json_model = self._loader.load_service_model(service_name, 'service-2',\n",
" File \"/var/runtime/botocore/loaders.py\", line 132, in _wrapper\n data = func(self, *args, **kwargs)\n",
" File \"/var/runtime/botocore/loaders.py\", line 377, in load_service_model\n raise UnknownServiceError(\n"
]
}
Does anybody know if keyspaces are actually available in AWS Lambda through boto3?
The error indicates that keyspaces is not available with current boto3 version available with lambda runtime. As per below link it is boto3-1.20.32
https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html
I have also gone through boto3 changelog and unable to find when keyspaces support added to boto3, but I am able to see some changes/fixes done to boto3 version 1.21.x
https://github.com/boto/boto3/blob/develop/CHANGELOG.rst#12120
How ever you can still try creating lambda layers with latest boto3 versions as explained below and use keyspaces with it.
https://aws.amazon.com/premiumsupport/knowledge-center/lambda-python-runtime-errors/
Below link talks about one another way to connect to keyspaces via lambda:
https://github.com/aws-samples/aws-keyspaces-lambda-python
secret_client = boto3.client('secretsmanager')
secret_response = secret_client.get_secret_value(SecretId=CASSANDRA_CREDS)
secret = json.loads(secret_response.get('SecretString'))
cassandra_user = secret['ServiceSpecificCredential']['ServiceUserName']
cassandra_password = secret['ServiceSpecificCredential']['ServicePassword']
auth_provider = PlainTextAuthProvider(username=cassandra_user, password=cassandra_password)
ssl_context = SSLContext(PROTOCOL_TLSv1)
ssl_context.load_verify_locations('AmazonRootCA1.pem')
ssl_context.verify_mode = CERT_REQUIRED
cluster = Cluster(['cassandra.{}.amazonaws.com'.format(AWS_DEFAULT_REGION)], port=9142, ssl_context=ssl_context, auth_provider=auth_provider)
session = cluster.connect()

How do I access the Cognito UserPoolClient Secret in Lambda function?

I have created Cognito UserPool and UserpoolClient via Resources in serverless.yml file like this -
CognitoUserPool:
Type: AWS::Cognito::UserPool
Properties:
AccountRecoverySetting:
RecoveryMechanisms:
- Name: verified_email
Priority: 2
UserPoolName: ${self:provider.stage}-user-pool
UsernameAttributes:
- email
MfaConfiguration: OFF
Policies:
PasswordPolicy:
MinimumLength: 8
RequireLowercase: True
RequireNumbers: True
RequireSymbols: True
RequireUppercase: True
CognitoUserPoolClient:
Type: AWS::Cognito::UserPoolClient
Properties:
ClientName: ${self:provider.stage}-user-pool-client
UserPoolId:
Ref: CognitoUserPool
ExplicitAuthFlows:
- ALLOW_USER_PASSWORD_AUTH
- ALLOW_REFRESH_TOKEN_AUTH
GenerateSecret: true
Now I can pass the Userpool and UserpoolClient as environment variables to the lambda functions like this -
my_function:
package: {}
handler:
events:
- http:
path:<path>
method: post
cors: true
environment:
USER_POOL_ID: !Ref CognitoUserPool
USER_POOL_CLIENT_ID: !Ref CognitoUserPoolClient
I can access these IDs in my code as -
USER_POOL_ID = os.environ['USER_POOL_ID']
USER_POOL_CLIENT_ID = os.environ['USER_POOL_CLIENT_ID']
I have printed the values and they are being printed correctly. However, UserpoolClient also generates one AppClient secret which I need to use while generating secret hash. How shall I access app client secret (UserpoolClient's secret) in my lambda?
Probably now what you hoped for, but you cannot export client secret in CloudFormation explicitly. Take a look at the return values from AWS::Cognito::UserPoolClient. There you can only get the client ID.
What you could do is to create the client in another CF template and either create there a custom resource to read the secret and output it, or have an intermediate step where you get this value with CLI and then pass it into serverless.
There is currently no other option.

Cross-stack reference within a lambda function

Going round in circles and tearing my hair out on this one, any pointers are greatly appreciated.
I have a CloudFormation stack which creates an EC2 instance. I am outputting its ID as follows:
Outputs:
DevBoxInstanceId:
Description: The instance ID of the EC2 Dev_Box.
Value: !Ref TestInstance
Export:
Name: DevBoxId
Now, in the console I can see that this outputs the ID as I'd like. My problem is that I can't work out how to reference this in my second stack. I haven't used Fn::ImportValues much but for obvious reasons I'd like to. My second stack creates a lambda function which will stop an instance. I want to reference the DevBoxId within the function - have I misunderstood something here? I've tried several variations on the following:
LambdaFunctionStop:
Type: AWS::Lambda::Function
Properties:
Runtime: python3.7
Timeout: 30
Code:
ZipFile: |
import boto3
region = 'eu-west-2'
instances = ['!ImportValue DevBoxId']
ec2 = boto3.client('ec2', region_name=region)
def handler(event, context):
ec2.stop_instances(InstanceIds=instances)
print('stopped your instances: ' + str(instances))
Description: Automatically stop Dev_Env instances based on specified schedule.
Handler: index.handler
Role: !GetAtt 'IAMRole.Arn'
The relevant part which I'm seeking help on is:
instances = ['!ImportValue DevBoxId']
How do I write this correctly?
Your ZipFile component is meant to be a string. So you will need to refer to Export Values outside that string and then substitute the value within that string. Try the below
LambdaFunctionStop:
Type: AWS::Lambda::Function
Properties:
Runtime: python3.7
Timeout: 30
Code:
ZipFile:
!Sub
- |
import boto3
region = 'eu-west-2'
instances = '${devboxvalue}'
ec2 = boto3.client('ec2', region_name=region)
def handler(event, context):
ec2.stop_instances(InstanceIds=instances)
print('stopped your instances: ' + str(instances))
- { devboxvalue : !ImportValue "DevBoxId" }
Description: Automatically stop Dev_Env instances based on specified schedule.
Handler: index.handler
Role: !GetAtt 'IAMRole.Arn'
You can pass the instance id to tha lambda's environment variables this way:
and you can then access if from you code depending on the way you read env variables

Listening to remote AWS SQS from local using serverless

I want to execute the lambda function locally , on SQS event which is on my AWS account. I have defined the required event but this not getting triggered.
How can this be achieved?
I am able to send the messages to the same queue using cron event from my local.
Here are few things I tried... but didnt work for me .
functions:
account-data-delta-test:
handler: functions/test/data/dataDeltatestGenerator.handler
name: ${self:provider.stage}-account-data-delta-test
description: account delta update - ${self:provider.stage}-account-data-delta-test
tags:
Name: ${self:provider.stage}-account-data-delta-test
# keeping 5 minute function timeout just in case large volume of data.
timeout: 300
events:
- sqs:
arn:
Fn::GetAtt: [ testGenerationQueue, Arn ]
batchSize: 10
----------
Policies:
- PolicyName: ${self:provider.stage}-test-sqs-policy
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- sqs:ReceiveMessage
- sqs:DeleteMessage
- sqs:GetQueueAttributes
- sqs:ChangeMessageVisibility
- sqs:SendMessage
- sqs:GetQueueUrl
- sqs:ListQueues
Resource: "*"
---------------
---
Resources:
testGenerationQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: ${self:provider.stage}-account-test-queue
VisibilityTimeout: 60
Tags:
-
Key: Name
Value: ${self:provider.stage}-account-test-queue
-------------
const sqs = new AWS.SQS({
region: process.env.REGION,
});
exports.handler = async (event) => {
console.error('------------ >>>>CRON:START: Test delta Job run.', event);
log.error('------------ >>>>CRON:START: Test delta Job run.', event);
});
You can't trigger your local Lambda function from your remote context because they haven't nothin in common.
I suppose your goal is to test the logic of Lambda function, if so you have two options.
Option 1
A faster way could be invoke function locally using sam local invoke. In this way, you could provide this command some argument, one of those arguments is the event source (i.e. the event information that SQS will send to the Lambda as soon this is triggered).
sam local invoke -e sqs.input.json account-data-delta-test
and your sqs.input.json would look like this (generate using sam local generate-event sqs receive-message)
so you will actually test your Lambda locally.
Pros: is fast
Cons: You still have to test the trigger when you will deploy on AWS
Option 2
In a second scenario you will sacrifice the bind between a queue and Lambda. You have to trigger your function at fix interval and explicitly use the ReceiveMessage in your code.
Pro: you can read a real message from a real queue.
Con: you have to invoke function at regular interval and this is not handy.

Hide or encrypt credentials information in AWS Data pipeline

I am creating an AWS data-pipeline to copy data from mysql to S3. I have written a shell script which accepts credentials as arguments and creates the pipeline so that my credentials are not exposed in script.
used below bash shell script to create pipeline.
unique_id="$(date +'%s')"
profile="${4}"
startDate="${1}"
echo "{\"values\":{\"myS3CopyStartDate\":\"$startDate\",\"myRdsUsername\":\"$2\",\"myRdsPassword\":\"$3\"}}" > mysqlToS3values.json
sqlpipelineId=`aws datapipeline create-pipeline --name mysqlToS3 --unique-id mysqlToS3_$unique_id --profile $profile --query '{ID:pipelineId}' --output text`
validationErrors=`aws datapipeline put-pipeline-definition --pipeline-id $sqlpipelineId --pipeline-definition file://mysqlToS3.json --parameter-objects file://mysqlToS3Parameters.json --parameter-values-uri file://mysqlToS3values.json --query 'validationErrors' --profile $profile`
aws datapipeline activate-pipeline --pipeline-id $sqlpipelineId --profile $profile
However when I fetch pipeline definition through aws cli using
aws datapipeline get-pipeline-definition --pipeline-id 27163782,
I get my credentials in plain text in json output.
{ "parameters": [...], "objects": [...], "values": { "myS3CopyStartDate": "2018-04-05T10:00:00", "myRdsPassword": "sbc", "myRdsUsername": "ksnck" } }
Is there any way to encrypt or hide the credentials information?
I don't think there is a way to mask the data in the pipeline definition.
The strategy I have used is to store my secrets in S3 (encrypted with a specific KMS key and using appropriate IAM/bucket permisions). Then, inside my datapipeline step, I use the AWS CLI to read the secret from S3 and pass it to the mysql command or whatever.
So instead of having a pipeline parameter like myRdsPassword I have:
"myRdsPasswordFile": "s3://mybucket/secrets/rdspassword"
Then inside my step I read it with something like:
PWD=$(aws s3 cp ${myRdsPasswordFile} -)
You could also have a similar workflow that retrieves the password from AWS Parameter Store instead of S3.
There is actually a way that's built into data pipelines:
You prepend the field with an * and it will encrypt the field and hide it visibly like a password form field.
If you're using parameters, then prepend the * on both the object field and the corresponding parameter field like so (note - there are three * with a parameterized setup; the example below is just a sample - missing required fields just to simplify and illustrate how to handle the encryption through parameters):
...{
"*password": "#{*myDbPassword}",
"name": "DBName",
"id": "DB",
},
],
"parameters": [
{
"id": "*myDbPassword",
"description": "Database password",
"type": "String"
}...
See more below:
https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-pipeline-characters.html
You can store RDS Credentials in AWS Secret Manager. You can then retrieve the credentials from SecretManager in the data-pipeline using cloudformation template as described below:
Mappings:
RegionToDatabaseConfig:
us-west-2:
CredentialsSecretKey: us-west-2-SECRET_NAME
# ...
us-east-1:
CredentialsSecretKey: us-east-1-SECRET_NAME
# ...
eu-west-1:
CredentialsSecretKey: eu-west-1-SECRET_NAME
# ...
Resources:
OurProjectDataPipeline:
Type: AWS::DataPipeline::Pipeline
Properties:
# ...
PipelineObjects:
# ...
# RDS resources
- Id: PostgresqlDatabase
Name: Source database to sync data from
Fields:
- Key: type
StringValue: RdsDatabase
- Key: username
StringValue:
!Join
- ''
- - '{{resolve:secretsmanager:'
- !FindInMap
- RegionToDatabaseConfig
- {Ref: 'AWS::Region'}
- CredentialsSecretKey
- ':SecretString:username}}'
- Key: "*password"
StringValue:
!Join
- ''
- - '{{resolve:secretsmanager:'
- !FindInMap
- RegionToDatabaseConfig
- {Ref: 'AWS::Region'}
- CredentialsSecretKey
- ':SecretString:password}}'
- Key: jdbcProperties
StringValue: 'allowMultiQueries=true'
- Key: rdsInstanceId
StringValue:
!FindInMap
- RegionToDatabaseConfig
- {Ref: 'AWS::Region'}
- RDSInstanceId

Resources