A lambda can have a result that is either a success or an error.
I want to see the logs of lambda that errored. I am trying to do that via a CloudWatch Insights query.
How can I do this?
If someone comes here looking for a solution, here's what I use:
filter #message like /(?i)(Exception|error|fail)/| fields #timestamp, #message | sort #timestamp desc | limit 20
I use the below query to get those errors which are not covered by the query mentioned in answer and I can only see failure in monitoring dashboard.
fields #timestamp, #message
| sort #timestamp desc
| filter #message not like 'INFO'
| filter #message not like 'REPORT'
| filter #message not like 'END'
| filter #message not like 'START'
| limit 20
Here is some example that cover by this query
timeout
#ingestionTime
1600997135683
#log
060558051165:/aws/lambda/prod-
#logStream
2020/09/25/[$LATEST]abc
#message
2020-09-25T01:25:35.623Z d0801056-abc-595a-b67d-47b14d3e9a20 Task timed out after 30.03 seconds
#requestId
d0801056-abc-595a-b67d-47b14d3e9a20
#timestamp
1600997135623
innovation error
#ingestionTime
1600996797947
#log
060558051165:/aws/lambda/prod-****
#logStream
2020/09/25/[$LATEST]123
#message
2020-09-25T01:19:48.940Z 7af13cdc-74fb-5986-ad6b-6b3b33266425 ERROR Invoke Error {"errorType":"Error","errorMessage":"QueueProcessor 4 messages failed processing","stack":["Error:QueueProcessor 4 messages failed processing"," at Runtime.handler (/var/task/lambda/abc.js:25986:11)"," at process._tickCallback (internal/process/next_tick.js:68:7)"]}
#requestId
7af13cdc-74fb-5986-ad6b-6b3b33266425
#timestamp
1600996788940
errorMessage
QueueProcessor 4 messages failed processing
errorType
Error
stack.0
Error: QueueProcessor 4 messages failed processing
stack.1
at Runtime.handler (/var/task/lambda/abcBroadcast.js:25986:11)
stack.2
at process._tickCallback (internal/process/next_tick.js:68:7)
another example with node run time
Value
#ingestionTime
1600996891752
#log
060558051165:/aws/lambda/prod-
#logStream
2020/09/24/[$LATEST]abc
#message
2020-09-25T01:21:31.213Z 32879c8c-abcd-5223-98f9-cb6b3a192f7c ERROR (node:6) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
#requestId
32879c8c-7242-5223-abcd-cb6b3a192f7c
#timestamp
1600996891214
If anyone looking how to search an error or a log in AWS Log insights, can use this query to search:
fields #timestamp, #message
| filter #message like /text to search/
| sort #timestamp desc
| limit 20
Actually just selecting log group(s) and adding a new line as | filter #message like /text to search/ into the query editor is enough. The rest comes by default.
Also, keep in mind to configure the time span for the search history in case if you cannot find the relevant results. By default, it only searches for the last 1h.
In your console, navigate to your lambda's configuration page. In the top left, click Monitoring, then View logs in CloudWatch on the right.
you can run the following query in the CloudWatch Logs Insights.
filter #type = "REPORT"
| stats max(#memorySize / 1000 / 1000) as provisonedMemoryMB,
min(#maxMemoryUsed / 1000 / 1000) as smallestMemoryRequestMB,
avg(#maxMemoryUsed / 1000 / 1000) as avgMemoryUsedMB,
max(#maxMemoryUsed / 1000 / 1000) as maxMemoryUsedMB,
provisonedMemoryMB - maxMemoryUsedMB as overProvisionedMB
Related
I have created Sumologic dashboard to show some errors in the application. What I want is show the error per date. It show the error but it doesn't aggregate the same error messages as the messages have some GUID.
This is the sample part of the query:
_sourceCategory="playGames/web-app-us"
and ERR
| timeslice 1d
| count _timeslice, message
enter image description here
I believe you need to format the message and remove the GUID. So all the messages with GUID will be aggregate to single message.
You can use regex to format the messages and remove the GUID. The sample query look like this and use as needed.
The sample error message is like this
Error occurred. Exception: System.Exception: my custom error message: 1121fd05-065b-499f-b174-2a13efdaf8b5
And the Sumologic query
_sourceCategory="dev/test-app"
and "[Error]"
and "Error occurred"
// | timeslice 1d
| formatDate(_receiptTime, "yyyy-MM-dd") as date
| replace(_raw,/my custom error message: ([0-9A-Fa-f\-]{36})/,"my custom error message") as finalMessage
| count date, finalMessage
| transpose row date column finalMessage
This video shows step by step guidance. https://youtu.be/Nxzp7G-rUh8
I am trying to send data from fluent_bit to the Elastic search
Here is my fluent-bit parser:
[PARSER]
Name escape_utf8_log
Format json
# Command | Decoder | Field | Optional Action
# =============|=====================|=================
Decode_Field_As escaped_utf8 log
Decode_Field json log [PARSER]
Name escape_message
Format json
# Command | Decoder | Field | Optional Action
# =============|=================|=================
Decode_Field_As escaped_utf8 message
Decode_Field json message
Here is my fluent-bit config:
[FILTER]
Name parser
Match docker_logs
Key_Name message
Parser escape_message
Reserve_Data True
In some cases, other people would put the log data to the fluent-bit in the wrong format so that we can get "mapper_parsing_exception" (example: filed to parse field [id] of type long in document).
I am trying to skip parsing a log and then send it to ES anyway if the fluent can not parse that log. so that we would not get the parser error even if someone sends the wrong format to fluent_bit. Is this possible to do that?
Here are two example messages of the lambda:
WARNING:
Field Value
#ingestionTime 1653987507053
#log XXXXXXX:/aws/lambda/lambda-name
#logStream 2022/05/31/[$LATEST]059106a15343448486b43f8b1168ec64
#message 2022-05-31T08:58:18.293Z b1266ad9-95aa-4c4e-9416-e86409f6455e WARN error catched and errorHandler configured, handling the error: Error: Error while executing handler: TypeError: Cannot read property 'replace' of undefined
#requestId b1266ad9-95aa-4c4e-9416-e86409f6455e
#timestamp 1653987498296
ERROR:
Field Value
#ingestionTime 1653917638480
#log XXXXXXXX:/aws/lambda/lambda-name
#logStream 2022/05/30/[$LATEST]bf8ba722ecd442dbafeaeeb3e7251024
#message 2022-05-30T13:33:57.406Z 8b5ec77c-fb30-4eb3-bd38-04a10abae403 ERROR Invoke Error {"errorType":"Error","errorMessage":"Error while executing configured error handler: Error: No body found in handler event","stack":["Error: Error while executing configured error handler: Error: No body found in handler event"," at Runtime.<anonymous> (/var/task/index.js:3180:15)"]}
#requestId 8b5ec77c-fb30-4eb3-bd38-04a10abae403
#timestamp 1653917637407
errorMessage
Error while executing configured error handler: Error: No body found in handler event
errorType
Error
stack.0 Error: Error while executing configured error handler: Error: No body found in handler event
stack.1 at Runtime.<anonymous> (/var/task/index.js:3180:15)
Can you help me understand how to set up the query in order to have a table with the following columns and their values:
from #message extract timestamp, requestID, type (WARN or ERROR), errorMessage and if feasible also the name of the lambda from #log and the #logStream.
If we'd look at the documentation on AWS Insights parse method
We can use asterisks * to capture details which for you would be:
fields #timestamp, #message, #log, #logStream, #requestId
| parse #message "* * * *" as timestamp, requestId, type, body
| display #timestamp, #requestId, #log, #logStream, body
If you'd like to also capture the error message try to now parse the body as well:
fields #timestamp, #message, #log, #logStream, #requestId
| parse #message "* * * *" as timestamp, requestId, type, body
| parse body "*,\"errorMessage\":\"*\"*" as startBody, errorMessage, endBody
| display #timestamp, #requestId, #log, #logStream, body, errorMessage
Should work but please feel free to look up any additional information in the AWS documentation, they've made it very thorough👌🏽
Appreciate your help in advance.
In my scenario - Cloudwatch multiline logs needs to be shipped to elasticsearch service.
ECS--awslog->Cloudwatch---using lambda--> ES Domain
(Basic flow though very open to change how data is shipped from CW to ES )
I was able to solve multi-line issue using multi_line_start_pattern BUT
The main issue I am experiencing now - is my logs have ODL format (following format)
[yyyy-mm-ddThh:mm:ss.SSS-Z][ProductName-Version][Log Level]
[Message ID][LoggerName][Key Value Pairs][[
Message]]
AND I will like to parse and tokenize log events before storing in ES (vs the complete log line ).
For example:
[2018-05-31T11:08:49.148-0400] [glassfish 4.1] [INFO] [] [] [tid: _ThreadID=43 _ThreadName=Thread-8] [timeMillis: 1527692929148] [levelValue: 800] [[
[] INFO : (DummyApplicationFunctionJPADAO) EntityManagerFactory located under resource lookup name [null], resource name=AuthorizationPU]]
Needs to be parsed and tokenize using format
timestamp 2018-05-31T11:08:49.148-0400
ProductName-Version glassfish 4.1
LogLevel INFO
MessageID
LoggerName
KeyValuePairs tid: _ThreadID=43 _ThreadName=Thread-8
Message [] INFO : (DummyApplicationFunctionJPADAO)
EntityManagerFactorylocated under resource lookup name
[null], resource name=AuthorizationPU
In above Key Value pairs repeat and are variable - for simplicity I can store all as one long string.
As far as what I gathered about Cloudwatch - It seems Subscription Filter Pattern reg ex support is very limited really not sure how to fit the above pattern. For lambda function that pushes the data to ES have not seen AWS doc or examples that support lambda as means to parse and push for ES.
Will appreciate if someone can please guide what/where will be best option to parse CW logs before it gets into ES => Subscription Filter -Pattern vs in lambda function or any other way.
Thank you .
From what I can see your best bet is what you're suggesting, a CloudWatch log triggered lambda that reformats the logged data into your ES prefered format and then posts it into ES.
You'll need to subscribe this lambda to your CloudWatch logs. You can do this on the lambda console, or the cloudwatch console (https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Subscriptions.html).
The lambda's event payload will be: { "awslogs": { "data": "encoded-logs" } }. Where encoded-logs is a Base64 encoding of a gzipped JSON.
For example, the sample event (https://docs.aws.amazon.com/lambda/latest/dg/eventsources.html#eventsources-cloudwatch-logs) can be decoded in node, for example, using:
const zlib = require('zlib');
const data = event.awslogs.data;
const gzipped = Buffer.from(data, 'base64');
const json = zlib.gunzipSync(gzipped);
const logs = JSON.parse(json);
console.log(logs);
/*
{ messageType: 'DATA_MESSAGE',
owner: '123456789123',
logGroup: 'testLogGroup',
logStream: 'testLogStream',
subscriptionFilters: [ 'testFilter' ],
logEvents:
[ { id: 'eventId1',
timestamp: 1440442987000,
message: '[ERROR] First test message' },
{ id: 'eventId2',
timestamp: 1440442987001,
message: '[ERROR] Second test message' } ] }
*/
From what you've outlined, you'll want to extract the logEvents array, and parse this into an array of strings. I'm happy to give some help on this too if you need it (but I'll need to know what language you're writing your lambda in- there are libraries for tokenizing ODL- so hopefully it's not too hard).
At this point you can then POST these new records directly into your AWS ES Domain. Somewhat crypitcally the S3-to-ES guide gives a good outline of how to do this in python: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-aws-integrations.html#es-aws-integrations-s3-lambda-es
You can find a full example for a lambda that does all this (by someone else) here: https://github.com/blueimp/aws-lambda/tree/master/cloudwatch-logs-to-elastic-cloud
This step in my Ruby feature file to find and access a customer record:
When I search with the following details: "<recordReference>", "<secondIdentifier>", "<postcode>":
| recordReference | secondIdentifier| postcode |
| REFABCDE12345678| IDcode1234 | PC11 1PC |
It has this step definition:
When(/^I search with the following details: "(.*?)", "(.*?)", "(.*?)":$/) do |recordReference, secondIdentifier, postcode|
find(:css, "#dln").native.send_keys recordReference
find(:css, "#nino").native.send_keys secondIdentifier
find(:css, "#postcode").native.send_keys postcode
check 'confirmation'
click_button 'submit'
end
When it's run, I get the following error:
Cucumber::ArityMismatchError: Your block takes 3 arguments, but the Regexp matched 4 arguments.
features/step_definitions/refactored_commands.rb:207:in `/^I search with the following details: "(.*?)", "(.*?)", "(.*?)":$/'
What have I done wrong and how can it be fixed?
For info - I get the same error message if the parenthases are take out of the step definition:
When /^I search with the following details: "(.*?)", "(.*?)", "(.*?)":$/ do |recordReference, secondIdentifier, postcode|
The fourth argument is the DataTable. Remove the first 3 parameters and put in just the DataTable option, you will get all the data from the DataTable. Suggest you use dryRun=true option to let Cucumber create the proper step definition matcher, this is from my Java knowledge dont know how this dryRun option is in ruby.
Plus you will have to change your step in the feature file to remove the mentioned 3 parameters
It looks like you're mixing up scenario outlines with passing data tables to steps
From the format of your table it looks like what you're going for should actually be
Scenario Outline: xxx
...
When I search with the following details: "<recordReference>", "<secondIdentifier>", "<postcode>"
...
Examples:
| recordReference | secondIdentifier| postcode |
| REFABCDE12345678| IDcode1234 | PC11 1PC |
and then each the outline will be called once for each row of Examples with the values filled in - so your step would be
When(/^I search with the following details: "(.*?)", "(.*?)", "(.*?)"$/) do |recordReference, secondIdentifier, postcode|
...
end
On a side note - is there any reason you're calling .native.send_keys - I believe every driver now supports the normal Capybara send_keys API so it would just be find(:css, '#dln').send_keys recordReference (or of course just fill_in('dln', with: recordReference) )