statemachine logs for a specific execution can be easily queried in cloudwatch via the execution_arn field in the log events.
However how can i find out which logs/logstreams of the LAMBDA functions correspond to which statemachine execution? I haven't found any connection so far. Ideally i would have expected to find an execution_arn field also in the lambda logs when called from stepfunctions.
There is no connection. You can, however, manually associate your Lambda logs with a State Machine execution by including the execution ARN in your input payload and logging it in Lambda. The execution ARN is available on the State Machine context object:
"Execution.$": "$$.Execution.Id"
Related
I'm using CDK to instantiate a Queue and a Lambda Function.
Lambda function requires QueueURL in order to push messages into it.
QueueURL is not fixed, it changes when the stack is re-created.
I have two options:
Pass QueueURL as an env variable to Lambda in CDK.
Create a cfnOutput with QueueURL and read it from the Lambda.
If I use option 2, Lambda will have to make an API call every time it runs to get the URL.
Are these the only options?
What is the recommended approach for this?
Thanks!
Option 1 is recommended. If the value changes for any reason, the lambda will also be updated accordingly automatically. It also ensures that the lambda will be created after the queue, as it creates an implicit dependency.
Don't forget to grant your lambda access to the queue with myQueue.grantSendMessages(myLambda);
An AWS Lambda Function(lambda_fns1) get triggered whenever a file (sample_file) is uploaded to an S3 bucket (raw_data_bucket)
Say path is s3://raw_data_bucket/client1/data/sample_file
This Lambda Function (lambda_fns1) then triggers an AWS Glue Job(glue_job) for file processing.
Note: Glue Job is part of the entire Glue Workflow. Half a dozen jobs get completed before the CloudWatch Event to start.
Once this Glue is completed, using a Cloud Watch Event Rule (cloud_event_rule) triggering another Lambda function(lambda_fns2).
[lambda_fns1] -> [[Glue Workflow(glue_job)]] -> [cloud_event_rule] -> [lambda_fns2]
https://www.youtube.com/watch?v=HYydZ5JLtQo
Initially, there had only 1 client we hardcoded these values but there are many clients. Now I need to pass the following details to lambda_fns2. How to do this?
{
'tenant': 'client1',
'path': 's3://raw_data_bucket/client1/data/sample_file'
}
Where do you see the console.log() calls made inside of AWS Lambda functions? I looked at AWS Cloud Watch event log and didn't see them there. Is there a CLI way to see them?
console.log() should definitely end up in the CloudWatch logs for your function. You should be able to find the correct log group in the web console interface for your function under the Monitoring tab - Jump to Logs. Note that you will have a different log stream for each invocation of your function, and there may be a delay between logs being written and logs showing up in a stream, so be patient.
It's possible you do not have the IAM permissions to create log groups or write to log streams. Ashan has provided links on how to fix that.
Additionally, you can use the awslogs tool to list groups/streams, as well as to download or tail groups/streams:
To list available groups: awslogs groups
To list available streams in group app/foo: awslogs streams app/foo
To "tail -f" all streams from a log group app/foo: awslogs get app/foo ALL --watch
Make sure you the IAM role assigned to the AWS Lambda function has permission to write to CloudWatch Logs. For more information regarding the policy refer Using Identity-Based Policies (IAM Policies)for CloudWatch Logs.
In addition, you should be able to view the CloudWatch log group by clicking on the CloudWatch Logs under Add Triggers in Lambda Console.
I want to view AWS lambda last hour errors of two types:
Lambda function that finished with an error
Lambda function returned http 500
How should I do that?
If you have many lambdas, in can be difficult to identify exactly which lambda caused an error. Here is how to find it out, even if you have hundreds of lambdas.
In CloudWatch, go to the Metrics page, then go to the Graph Metrics tab, then navigate to the dropdown menu item “Math expression > Search > Lambda Throttles or Errors.”
This will give you error counts per lambda in a graph, mouse over to get the name of the offending lambda.
Once you launched an AWS Lambda project, automatically that is watched by CloudWatch.
Lambda function that finished with an error
You can see Lambda function errors from monitoring tab on Lambda default view.
Lambda function returned http 500
I guess your Lambda function is WEB API. If your WEB API created by Lambda function, you need to output logs with standard output in order to see logging on CloudWatch.
Please find documents from Accessing Amazon CloudWatch Logs for AWS Lambda
NOTE: only if you use serverless:
Alternatively, you can monitor your lambda function logs using serverless cli.
For example, to get log in the past 1 hours:
sls logs -f functionName --startTime 1h
You also can filter based on the string 'error' in the past 1 hours:
sls logs -f functionName --startTime 1h --filter error
Please check on the doc.
You could enable X-Ray traces from the lambda dashboard
Lambda Console Enable X-Ray Tracing
The X-Ray service displays trace mappings for lambda execution results. The service is great for checking the results of errors within lambda functions, but if you are looking for detailed error result logs, CloudWatch is your best bet.
You could also try something like Logbird that processes CloudWatch streams for all errors in AWS Lambda, API Gateway and other cloud services and can trigger notifications.
Is it possible to see where a lambda is called from? Like an API Gateway or another lambda? Something like an event list.
When Lambda is called, you will require to pass identification from where it is being called (API or another Lambda) and then log it as source of lambda execution using:
LambdaLogger.Log(string message)
All logs you will be able to see in AWS » Couldwatch » Logs
You can log it in your lambda, using the context object.
More info in the aws documentation