I am working on elastalert. I am able to send the email alerts whenever my condition matches.
Now my use case is I want to send email whenever any error is encountered.
So if its error : email content body should be "ERROR OCCURED"
else
if it's FATAL : email content body should be "FATAL ERROR"
I have following example_frequency.yaml :
name: Error Occurred in your Application
type: frequency
index: logstash-*
num_events: 1
timframe:
hours:1
filter:
-query:
query_string:
query: "message: *ERROR* OR message: *FATAL*"
alert:
- "email"
alert_text_type: "alert_text_only"
alert_text: |
Error occured at {0}
Host Machine id: {1}
Error Message: {2}
Log File Location: {3}
alert_text_args:
- "#timestamp"
- "beat.hostname"
- "message"
- "source"
email:
- "test#gmail.com"
With this configuration even its FATAL/ERROR I am getting the same email content. I want to change some text if its FATAL.
Is there any wa to do it in ElastAlert?
Please help guys!
Related
Here are two example messages of the lambda:
WARNING:
Field Value
#ingestionTime 1653987507053
#log XXXXXXX:/aws/lambda/lambda-name
#logStream 2022/05/31/[$LATEST]059106a15343448486b43f8b1168ec64
#message 2022-05-31T08:58:18.293Z b1266ad9-95aa-4c4e-9416-e86409f6455e WARN error catched and errorHandler configured, handling the error: Error: Error while executing handler: TypeError: Cannot read property 'replace' of undefined
#requestId b1266ad9-95aa-4c4e-9416-e86409f6455e
#timestamp 1653987498296
ERROR:
Field Value
#ingestionTime 1653917638480
#log XXXXXXXX:/aws/lambda/lambda-name
#logStream 2022/05/30/[$LATEST]bf8ba722ecd442dbafeaeeb3e7251024
#message 2022-05-30T13:33:57.406Z 8b5ec77c-fb30-4eb3-bd38-04a10abae403 ERROR Invoke Error {"errorType":"Error","errorMessage":"Error while executing configured error handler: Error: No body found in handler event","stack":["Error: Error while executing configured error handler: Error: No body found in handler event"," at Runtime.<anonymous> (/var/task/index.js:3180:15)"]}
#requestId 8b5ec77c-fb30-4eb3-bd38-04a10abae403
#timestamp 1653917637407
errorMessage
Error while executing configured error handler: Error: No body found in handler event
errorType
Error
stack.0 Error: Error while executing configured error handler: Error: No body found in handler event
stack.1 at Runtime.<anonymous> (/var/task/index.js:3180:15)
Can you help me understand how to set up the query in order to have a table with the following columns and their values:
from #message extract timestamp, requestID, type (WARN or ERROR), errorMessage and if feasible also the name of the lambda from #log and the #logStream.
If we'd look at the documentation on AWS Insights parse method
We can use asterisks * to capture details which for you would be:
fields #timestamp, #message, #log, #logStream, #requestId
| parse #message "* * * *" as timestamp, requestId, type, body
| display #timestamp, #requestId, #log, #logStream, body
If you'd like to also capture the error message try to now parse the body as well:
fields #timestamp, #message, #log, #logStream, #requestId
| parse #message "* * * *" as timestamp, requestId, type, body
| parse body "*,\"errorMessage\":\"*\"*" as startBody, errorMessage, endBody
| display #timestamp, #requestId, #log, #logStream, body, errorMessage
Should work but please feel free to look up any additional information in the AWS documentation, they've made it very thorough👌🏽
i am using serverless framework with serverless-step-functions plugin. I want to check any errors in my stepfunction workflow and send this error to sqs queue.
Currently I want to pass all input as message to the queue(MessageBody: $). But if I get the data from the queue, message is $ (dollar sign) and not actual input. How can I send to queue the error message from the previous step?
States:
state1:
Type: Task
Resource:
Fn::GetAtt: [function1, Arn]
Next: state2
Catch:
- ErrorEquals: [States.ALL]
Next: sendErrorToDLQ
ResultPath: $.error
state2:
Type: Task
Resource:
Fn::GetAtt: [function2, Arn]
Next: done
Catch:
- ErrorEquals: [ States.ALL ]
Next: sendErrorToDLQ
ResultPath: $.error
sendErrorToDLQ:
Type: Task
Resource: arn:aws:states:::sqs:sendMessage
Parameters:
QueueUrl:
Ref: ServiceDeadLetterQueue
MessageBody: $ # <== how to pass input to sqs message
Next: fail
fail:
Type: Fail
done:
Type: Succeed
I have got the same when connecting SNS. As per the AWS doc, we have to follow the below structure to send the parameters
"MessageBody.$": "$"
Reference: https://docs.aws.amazon.com/step-functions/latest/dg/connect-sqs.html
Recently switched to using Cypress parallel for our Angular project in our pipeline. We run on a Codebuild on AWS and run 5 threads of the Cypress runner. About a quarter of the time, the first test on one of the threads fails with this error:
An uncaught error was detected outside of a test
Invalid or unexpected token
This error originated from your test code, not from Cypress.
When Cypress detects uncaught errors originating from your test code it will automatically fail the current test.
Cypress could not associate this error to any specific test. We dynamically generated a new test to display this failure.
Tried many things to try to fix this, including setting modifyObtrusiveCode to false, chromeWebSecurity to false, upgrading Cypress. We are already catching uncaught exceptions so that doesn't seem like it should be the issue. I turned on some extra logs for this and here is the output
[3] 2020-03-06T19:57:20.369Z cypress:server:project onMocha start
[3] 2020-03-06T19:57:20.369Z cypress:server:reporter got mocha event 'start' with args: [ { start: '2020-03-06T19:57:20.366Z' } ]
[3] 2020-03-06T19:57:20.374Z cypress:server:project onMocha suite
[3] 2020-03-06T19:57:20.374Z cypress:server:reporter got mocha event 'suite' with args: [ { id: 'r1', title: '', root: true, type: 'suite', file: 'cypress/integration/ci-tests/content-acquisition/channels/channel-manual-upload-run-acquired-items-tab.spec.ts' } ]
[3]
[3] 2020-03-06T19:57:20.390Z cypress:server:project onMocha test
[3] 2020-03-06T19:57:20.391Z cypress:server:reporter got mocha event 'test' with args: [ { id: 'r2', title: 'An uncaught error was detected outside of a test', body: 'function throwErr() {\n throw err;\n }', type: 'test' } ]
[3] 2020-03-06T19:57:20.555Z cypress:server:reporter got mocha event 'fail' with args: [ { id: 'r2', title: 'An uncaught error was detected outside of a test', err: { message: 'Unexpected end of input\n' + '\n' + 'This error originated from your test code, not from Cypress.\n' + '\n' + 'When Cypress detects uncaught errors originating from your test code it will automatically fail the current test.\n' + '\n' + 'Cypress could not associate this error to any specific test.\n' + '\n' + 'We dynamically generated a new test to display this failure.', name: 'Uncaught SyntaxError', stack: 'Uncaught SyntaxError: Unexpected end of input\n' + '\n' + 'This error originated from your test code, not from Cypress.\n' + '\n' + 'When Cypress detects uncaught errors originating from your test code it will automatically fail the current test.\n' + '\n' + 'Cypress could not associate this error to any specific test.\n' + '\n' + 'We dynamically generated a new test to display this failure.' }, state: 'failed', body: 'function throwErr() {\n throw err;\n }', type: 'test', duration: 179, wallClockStartedAt: '2020-03-06T19:57:20.374Z', timings: { lifecycle: 26, test: [Object] } } ]
I couldn't really make anything of these errors, but maybe someone else can. I'm kind of out of ideas on what to try (I've tried more things today than I've listed but can't recall them all). Any ideas?
as setting modifyObtrusiveCode to false didn't help you as the folks in https://github.com/cypress-io/cypress/issues/6132 .. I can give my debug procedure when I encountered a similar flakey "unexpected .." error with Cypress:
cypress run has a burn= param, able to repeatedly run. Enable .har output recording for those runs with the cypress-har-generator plugin.
When you have two groups of successful and failing example .har files for the same request, open them in a Browser to compare if anything stands out.
I used diff + jq queries on the .har files to compare between the groups content per request path, but already opening a failing .har in the browser inspector network tab showed a 30s processing time for a .js path that was ultimately incomplete, and thus violated js syntax, causing an unexpected end of input error, similar to your "unexpected token".
Interestingly this occured to the same file at the same code line, hinting at a parsing problem in Cypress.
We exchanged that dependency (or specifically - updated it and changed how it was webpacked) and Cypress stopped to hiccup on the ressource, the flakiness disappeared.
My impression is, running parallel threads of Cypress contributes to the problem occuring.
I have configured aggregation option in config.yaml to send summary of alerts after every 1 hour. But it is throwing following error when I try to run it.
File "elastalert.py", line 863, in run_rule
self.add_aggregated_alert(match, rule)
File "elastalert.py", line 1614, in add_aggregated_alert
alert_time = ts_now() + rule['aggregation']
TypeError: unsupported operand type(s) for +: 'datetime.datetime' and 'dict'
ERROR:root:Uncaught exception running rule Test Alert : unsupported operand type(s) for +: 'datetime.datetime' and 'dict'
INFO:elastalert:Rule Test Alert disabled
The config parameters are:
rules_folder: test_rules
run_every:
minutes: 15
buffer_time:
minutes: 30
es_host: 100.38.46.3
es_port: 9200
aggregation:
hours: 1
writeback_index: elastalert_status
alert_time_limit:
days: 2
Test Alert rule configuration:
name: Test Alert
type: metric_aggregation
index: logstash-*
buffer_time:
minutes: 30
metric_agg_key: count
metric_agg_type: sum
query_key: "name.keyword"
doc_type: counter
max_threshold: 1
min_threshold: 0
filter:
- query:
query_string:
query: "name.keyword: *timedout_count"
alert:
- "email"
email:
- "admin#abc.com"
I have followed the ElastAlert docs but not able to figure out what is causing this issue.
Thanks
From the error:
TypeError: unsupported operand type(s) for +: 'datetime.datetime' and 'dict'
And from your config.yaml
metric_agg_type: sum
It's trying (and faililng) to perform a sum aggregation on datetime and dict values that do not support summation. You'll need to choose an aggregation like count or unique count, probably, and adjust the logic of your alerts accordingly.
My validation.yml is given:
task:
- Email:
message: The email "{{ value }}" is not a valid email.
- MinLength: { limit: 50, message: You must be 50 or under to enter. }
My issue is that if I give "wrong-email" in the task-field it gives two error messages:
The email "wrong-email" is not a valid email.
You must be 50 or under to enter.
Actually, I want to show only one error-message at a time.
That means it should check for the validation "MinLength" only if it is a valid email.
Validation sequencing can be done using group sequences. I fixed group sequences for the YAML driver only today, so you might need to wait for the next release of the 2.0 or master branch.
MyEntity:
group_sequence: [MyEntity, Extra]
properties:
task:
- Email: { message: ... }
- MinLength { limit: 50, message: ..., groups: Extra }
Now the constraints in group "Extra" will only be validated if all constraints in group "MyEntity" (i.e. the default group) succeed.