Parse message in CloudWatch Logs Insights - aws-lambda

Here are two example messages of the lambda:
WARNING:
Field Value
#ingestionTime 1653987507053
#log XXXXXXX:/aws/lambda/lambda-name
#logStream 2022/05/31/[$LATEST]059106a15343448486b43f8b1168ec64
#message 2022-05-31T08:58:18.293Z b1266ad9-95aa-4c4e-9416-e86409f6455e WARN error catched and errorHandler configured, handling the error: Error: Error while executing handler: TypeError: Cannot read property 'replace' of undefined
#requestId b1266ad9-95aa-4c4e-9416-e86409f6455e
#timestamp 1653987498296
ERROR:
Field Value
#ingestionTime 1653917638480
#log XXXXXXXX:/aws/lambda/lambda-name
#logStream 2022/05/30/[$LATEST]bf8ba722ecd442dbafeaeeb3e7251024
#message 2022-05-30T13:33:57.406Z 8b5ec77c-fb30-4eb3-bd38-04a10abae403 ERROR Invoke Error {"errorType":"Error","errorMessage":"Error while executing configured error handler: Error: No body found in handler event","stack":["Error: Error while executing configured error handler: Error: No body found in handler event"," at Runtime.<anonymous> (/var/task/index.js:3180:15)"]}
#requestId 8b5ec77c-fb30-4eb3-bd38-04a10abae403
#timestamp 1653917637407
errorMessage
Error while executing configured error handler: Error: No body found in handler event
errorType
Error
stack.0 Error: Error while executing configured error handler: Error: No body found in handler event
stack.1 at Runtime.<anonymous> (/var/task/index.js:3180:15)
Can you help me understand how to set up the query in order to have a table with the following columns and their values:
from #message extract timestamp, requestID, type (WARN or ERROR), errorMessage and if feasible also the name of the lambda from #log and the #logStream.

If we'd look at the documentation on AWS Insights parse method
We can use asterisks * to capture details which for you would be:
fields #timestamp, #message, #log, #logStream, #requestId
| parse #message "* * * *" as timestamp, requestId, type, body
| display #timestamp, #requestId, #log, #logStream, body
If you'd like to also capture the error message try to now parse the body as well:
fields #timestamp, #message, #log, #logStream, #requestId
| parse #message "* * * *" as timestamp, requestId, type, body
| parse body "*,\"errorMessage\":\"*\"*" as startBody, errorMessage, endBody
| display #timestamp, #requestId, #log, #logStream, body, errorMessage
Should work but please feel free to look up any additional information in the AWS documentation, they've made it very thorough👌🏽

Related

Sumologic query to show data by date

I have created Sumologic dashboard to show some errors in the application. What I want is show the error per date. It show the error but it doesn't aggregate the same error messages as the messages have some GUID.
This is the sample part of the query:
_sourceCategory="playGames/web-app-us"
and ERR
| timeslice 1d
| count _timeslice, message
enter image description here
I believe you need to format the message and remove the GUID. So all the messages with GUID will be aggregate to single message.
You can use regex to format the messages and remove the GUID. The sample query look like this and use as needed.
The sample error message is like this
Error occurred. Exception: System.Exception: my custom error message: 1121fd05-065b-499f-b174-2a13efdaf8b5
And the Sumologic query
_sourceCategory="dev/test-app"
and "[Error]"
and "Error occurred"
// | timeslice 1d
| formatDate(_receiptTime, "yyyy-MM-dd") as date
| replace(_raw,/my custom error message: ([0-9A-Fa-f\-]{36})/,"my custom error message") as finalMessage
| count date, finalMessage
| transpose row date column finalMessage
This video shows step by step guidance. https://youtu.be/Nxzp7G-rUh8

The "error_marshaling_enabled" setting seems to be always enabled

When I run this code:
$client->evaluate('
box.session.settings.error_marshaling_enabled = false
box.error{code = 42, reason = "Foobar", type = "MyError"}
');
regardless of the value of error_marshaling_enabled I always get a response with a new (extended) error format:
[
49 => 'Foobar',
82 => [
0 => [
0 => [
0 => 'CustomError',
2 => 3,
1 => 'eval',
3 => 'Foobar',
4 => 0,
5 => 42,
6 => [
'custom_type' => 'MyError',
],
],
],
],
],
Why is that?
Short answer.
error_marshaling_enabled option affects only how error objects are encoded in response body (48, IPROTO_DATA). It does not affect how they are returned as exceptions, in the response header (82, IPROTO_ERROR).
Long answer.
In Tarantool an error object can be returned in 2 ways: as an exception and as an object. For example, this is how to throw an error as exception:
function throw_error()
box.error({code = 1000, reason = "Error message"})
-- Or
error('Some error string')
end
This is how to return it as an object:
function return_error()
return box.error.new({code = 1000, reason = "Error message"})
end
If the function was called remotely, using IPROTO protocol via a connector like netbox, or PHP connector, or any other one, the error return way affects how it is encoded into MessagePack response packet. When the function throws, and the error reaches the top stack frame without being caught, it is encoded as IPROTO_ERROR (82) and IPROTO_ERROR_24 (49).
When the error object is returned as a regular value, not as an exception, it is encoded also as a regular value, inside IPROTO_DATA (48). Just like a string, a number, a tuple, etc.
With encoding as IPROTO_ERROR/IPROTO_ERROR_24 there is no much of a configuration space. Format of these values can't be changed. IPROTO_ERROR is always returned as a MessagePack map, with a stack of errors in it. IPROTO_ERROR_24 is always an error message. The IPROTO_ERROR_24 field is kept for compatibility with connectors to Tarantool versions < 2.4.1.
With encoding as a part of IPROTO_DATA you can choose serialization way using error_marshaling_enabled option. When it is true, errors are encoded as MessagePack extension type MP_EXT, and contain the whole error stack, encoded exactly like IPROTO_ERROR value. When the option is false (default behaviour in 2.4.1), the error is encoded as a string, MP_STR, which is the error's message. If there is a stack of errors, only the newest error is encoded.
error_marshaling_enabled option exists for backward compatibility, in case your application on Tarantool wants to be compatible with old connectors, which don't support MP_EXT encoded errors.
In Tarantool < 2.4.1 errors were encoded into result MessagePack as a string with error message, and error stacks didn't exist at all. So when the new format and the error stacks feature were introduced, making the new format default would be a too radical change breaking the old connectors.
Consider these examples of how error marshaling affects results. I use Tarantool 2.4.1 console here, and built-in netbox connector. The code below can be copy pasted into the console.
First instance:
box.cfg{listen = 3313}
box.schema.user.grant('guest', 'super')
function throw_error()
box.error({code = 1000, reason = "Error message"})
end
function return_error()
return box.error.new({code = 1000, reason = "Error message"})
end
Second instance:
netbox = require('net.box')
c = netbox.connect(3313)
Now I try to call the function on the second instance:
tarantool> c:call('throw_error')
---
- error: Error message
...
The c:call('throw_error') threw an exception. If I catch it using pcall() Lua function, I will see the error object.
tarantool> ok, err = pcall(c.call, c, 'throw_error')
tarantool> err:unpack()
---
- code: 1000
base_type: ClientError
type: ClientError
message: Error message
trace:
- file: '[string "function throw_error()..."]'
line: 2
...
As you can see, I didn't set error_marshaling_enabled, but got the full error. Now I will call the other function, without exceptions. But the error object won't be full.
tarantool> err = c:call('return_error')
tarantool> err
---
- Error message
...
tarantool> err:unpack()
---
- error: '[string "return err:unpack()"]:1: attempt to call method ''unpack'' (a nil
value)'
...
The error was returned as a mere string, error message. Not as an error object. Now I will turn on the marshaling:
tarantool> c:eval('box.session.settings.error_marshaling_enabled = true')
---
...
tarantool> err = c:call('return_error')
---
...
tarantool> err:unpack()
---
- code: 1000
base_type: ClientError
type: ClientError
message: Error message
trace:
- file: '[C]'
line: 4294967295
...
Now the same function returned the error in the new format, more featured.
On the summary: error_marshaling_enabled affects only returned errors. Not thrown errors.

CloudWatch Insights: get logs of errored lambdas

A lambda can have a result that is either a success or an error.
I want to see the logs of lambda that errored. I am trying to do that via a CloudWatch Insights query.
How can I do this?
If someone comes here looking for a solution, here's what I use:
filter #message like /(?i)(Exception|error|fail)/| fields #timestamp, #message | sort #timestamp desc | limit 20
I use the below query to get those errors which are not covered by the query mentioned in answer and I can only see failure in monitoring dashboard.
fields #timestamp, #message
| sort #timestamp desc
| filter #message not like 'INFO'
| filter #message not like 'REPORT'
| filter #message not like 'END'
| filter #message not like 'START'
| limit 20
Here is some example that cover by this query
timeout
#ingestionTime
1600997135683
#log
060558051165:/aws/lambda/prod-
#logStream
2020/09/25/[$LATEST]abc
#message
2020-09-25T01:25:35.623Z d0801056-abc-595a-b67d-47b14d3e9a20 Task timed out after 30.03 seconds
#requestId
d0801056-abc-595a-b67d-47b14d3e9a20
#timestamp
1600997135623
innovation error
#ingestionTime
1600996797947
#log
060558051165:/aws/lambda/prod-****
#logStream
2020/09/25/[$LATEST]123
#message
2020-09-25T01:19:48.940Z 7af13cdc-74fb-5986-ad6b-6b3b33266425 ERROR Invoke Error {"errorType":"Error","errorMessage":"QueueProcessor 4 messages failed processing","stack":["Error:QueueProcessor 4 messages failed processing"," at Runtime.handler (/var/task/lambda/abc.js:25986:11)"," at process._tickCallback (internal/process/next_tick.js:68:7)"]}
#requestId
7af13cdc-74fb-5986-ad6b-6b3b33266425
#timestamp
1600996788940
errorMessage
QueueProcessor 4 messages failed processing
errorType
Error
stack.0
Error: QueueProcessor 4 messages failed processing
stack.1
at Runtime.handler (/var/task/lambda/abcBroadcast.js:25986:11)
stack.2
at process._tickCallback (internal/process/next_tick.js:68:7)
another example with node run time
Value
#ingestionTime
1600996891752
#log
060558051165:/aws/lambda/prod-
#logStream
2020/09/24/[$LATEST]abc
#message
2020-09-25T01:21:31.213Z 32879c8c-abcd-5223-98f9-cb6b3a192f7c ERROR (node:6) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
#requestId
32879c8c-7242-5223-abcd-cb6b3a192f7c
#timestamp
1600996891214
If anyone looking how to search an error or a log in AWS Log insights, can use this query to search:
fields #timestamp, #message
| filter #message like /text to search/
| sort #timestamp desc
| limit 20
Actually just selecting log group(s) and adding a new line as | filter #message like /text to search/ into the query editor is enough. The rest comes by default.
Also, keep in mind to configure the time span for the search history in case if you cannot find the relevant results. By default, it only searches for the last 1h.
In your console, navigate to your lambda's configuration page. In the top left, click Monitoring, then View logs in CloudWatch on the right.
you can run the following query in the CloudWatch Logs Insights.
filter #type = "REPORT"
| stats max(#memorySize / 1000 / 1000) as provisonedMemoryMB,
min(#maxMemoryUsed / 1000 / 1000) as smallestMemoryRequestMB,
avg(#maxMemoryUsed / 1000 / 1000) as avgMemoryUsedMB,
max(#maxMemoryUsed / 1000 / 1000) as maxMemoryUsedMB,
provisonedMemoryMB - maxMemoryUsedMB as overProvisionedMB

I am getting exception on a post request logged in stackify but not able to find cause of the error

Following is the exception stackify shows me on dashboard:
ERROR The string was not recognized as a valid DateTime. There is an unknown word starting at index 0.
System.FormatException: The string was not recognized as a valid DateTime. There is an unknown word starting at index 0. ---> System.FormatException: null is not a valid value for DateTime.
at System.DateTimeParse.Parse
at System.ComponentModel.DateTimeConverter.ConvertFrom
--- End of inner exception stack trace ---
at System.ComponentModel.DateTimeConverter.ConvertFrom
at System.ComponentModel.NullableConverter.ConvertFrom
at Microsoft.AspNetCore.Mvc.ModelBinding.Binders.SimpleTypeModelBinder.BindModelAsync
I am not able to understand which field of the viewmodel causes the issue like this. How can I catch this exception on my local code?

fluent-plugin-elasticsearch: "Could not push log to Elasticsearch" error with "error"=>{"type"=>"mapper_parsing_exception"}

When I am injecting data collected by Fluentd to Elasticsearch using fluent-plugin-elasticsearch, some data caused the following error:
2017-04-09 23:47:37 +0900 [error]: Could not push log to Elasticsearch: {"took"=>3, "errors"=>true, "items"=>[{"index"=>{"_index"=>"logstash-201704", "_type"=>"ruby", "_id"=>"AVtTLz_cUzkwT9CQCxrH", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [message]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:27"}}}}, .....]}
It seems that elasticsearch banned the data for error failed to parse [message] and Can't get text on a START_OBJECT at 1:27. but I cannot see what data is sent to Elasticsearch and what's wrong.
Any ideas?
fluent-plugin-elasticsearch uses _bulk API to sending data. I put the request-dumping code on /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-5.0.4/lib/elasticsearch/api/actions/bulk.rb as following:
def bulk(arguments={})
...
payload = body
end
$log.info([method, path, params, payload].inspect) # <=== here ($log is global logger of fluentd)
perform_request(method, path, params, payload).body
And I found the request sent to Elasticsearch was as following:
POST /_bulk
{"index":{"_index":"logstash-201704","_type":"ruby"}}
{"level":"INFO","message":{"status":200,"time":{"total":46.26,"db":33.88,"view":12.38},"method":"PUT","path":"filtered","params":{"time":3815.904,"chapter_index":0},"response":[{}]},"node":"main","time":"2017-04-09T14:39:06UTC","tag":"filtered.console","#timestamp":"2017-04-09T23:39:06+09:00"}
The problem is message field contains JSON object, although this field is mapped as analyzed string on Elasticsearch.

Resources