Push log file to elastic using logstash - elasticsearch

Question

We have a error log file from Jmeter , that we need to be pushing to elastic using logstash
Here is the log file , we wanna be pushing the data from line 5 ,
2021-05-14 20:32:49,822 INFO o.a.j.u.JMeterUtils: Setting Locale to en_EN
2021-05-14 20:32:49,842 INFO o.a.j.JMeter: Loading user properties from: user.properties
2021-05-14 20:32:49,843 INFO o.a.j.JMeter: Loading system properties from: system.properties
2021-05-14 20:32:49,844 WARN o.a.j.JMeter: LogLevel: ERROR
2021-05-14 20:32:51,824 ERROR o.a.j.e.J.JSR223 PostProcessor: Http URL/API Test 1-4: ; response received: {"timestamp":"2021-05-14T14:32:51.688+0000","status":404,"error":"Not Found","message":"No message available","path":"/healths"}
2021-05-14 20:32:51,824 ERROR o.a.j.e.J.JSR223 PostProcessor: Http URL/API Test 1-7: ; response received: {"timestamp":"2021-05-14T14:32:51.689+0000","status":404,"error":"Not Found","message":"No message available","path":"/healths"}
2021-05-14 20:32:51,824 ERROR o.a.j.e.J.JSR223 PostProcessor: Http URL/API Test 1-20: ; response received: {"timestamp":"2021-05-14T14:32:51.850+0000","status":404,"error":"Not Found","message":"No message available","path":"/healths"}
2021-05-14 20:32:51,824 ERROR o.a.j.e.J.JSR223 PostProcessor: LEX 2-1: ; response received: <SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Header><ns3:systemContext xmlns:ns3="urn:abc.mon.utils.v1.context"><systemId>TREX</systemId><correlationId>hgfhjghj-c62dsfsdfdb-4aff-a871-8f32088943b3</correlationId></ns3:systemContext></SOAP-ENV:Header><SOAP-ENV:Body><ns3:retrieveStopPayResponse xmlns:ns3="urn:abc.tailhead.readyforcheck.processStopPay"><retrieveGagaPayResponse recordFound="false"><checkNumber/><stopPayDate/></retrieveGagaPayResponse></ns3:retrieveStopPayResponse></SOAP-ENV:Body></SOAP-ENV:Envelope>
2021-05-14 20:32:51,824 ERROR o.a.j.e.J.JSR223 PostProcessor: Http URL/API Test 1-5: ; response received: {"timestamp":"2021-05-14T14:32:51.687+0000","status":404,"error":"Not Found","message":"No message available","path":"/healths"}
We want to be pushing data from line 4 specifically PostProcessor: , response received: to elastic. The below conf is just for response received:
here is the logstash conf
if [type] == "errors"{​​​​​
grok {​​​​​
match => {​​​​​"message" => "%{​​​​​GREEDYDATA}​​​​​#*response received: %{​​​​​GREEDYDATA:error_message}​​​​​"}​​​​​
}​​​​​
}​​​​​
We get an error Grok parse failure _grokparsefailure when we try to push
Not sure what am I missing here

Answer 1

Something like this should work for you:
%{TIMESTAMP_ISO8601:timestamp} %{WORD:LogType} %{GREEDYDATA:data1} PostProcessor: %{GREEDYDATA:data2} ; response received: %{GREEDYDATA:responseRecd}
Example 1:
2021-05-14 20:32:51,824 ERROR o.a.j.e.J.JSR223 PostProcessor: Http URL/API Test 1-4: ; response received: {"timestamp":"2021-05-14T14:32:51.688+0000","status":404,"error":"Not Found","message":"No message available","path":"/healths"}
Output :
{
"timestamp": [
[
"2021-05-14 20:32:51,824"
]
],
"LogType": [
[
"ERROR"
]
],
"data1": [
[
"o.a.j.e.J.JSR223"
]
],
"data2": [
[
"Http URL/API Test 1-4:"
]
],
"responseRecd": [
[
"{"timestamp":"2021-05-14T14:32:51.688+0000","status":404,"error":"Not Found","message":"No message available","path":"/healths"}"
]
]
}
Example 2 :
2021-05-14 20:32:51,824 ERROR o.a.j.e.J.JSR223 PostProcessor: LEX 2-1: ; response received: <SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Header><ns3:systemContext xmlns:ns3="urn:abc.mon.utils.v1.context"><systemId>TREX</systemId><correlationId>hgfhjghj-c62dsfsdfdb-4aff-a871-8f32088943b3</correlationId></ns3:systemContext></SOAP-ENV:Header><SOAP-ENV:Body><ns3:retrieveStopPayResponse xmlns:ns3="urn:abc.tailhead.readyforcheck.processStopPay"><retrieveGagaPayResponse recordFound="false"><checkNumber/><stopPayDate/></retrieveGagaPayResponse></ns3:retrieveStopPayResponse></SOAP-ENV:Body></SOAP-ENV:Envelope>
Output :
{
"timestamp": [
[
"2021-05-14 20:32:51,824"
]
],
"LogType": [
[
"ERROR"
]
],
"data1": [
[
"o.a.j.e.J.JSR223"
]
],
"data2": [
[
"LEX 2-1:"
]
],
"responseRecd": [
[
"<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Header><ns3:systemContext xmlns:ns3="urn:abc.mon.utils.v1.context"><systemId>TREX</systemId><correlationId>hgfhjghj-c62dsfsdfdb-4aff-a871-8f32088943b3</correlationId></ns3:systemContext></SOAP-ENV:Header><SOAP-ENV:Body><ns3:retrieveStopPayResponse xmlns:ns3="urn:abc.tailhead.readyforcheck.processStopPay"><retrieveGagaPayResponse recordFound="false"><checkNumber/><stopPayDate/></retrieveGagaPayResponse></ns3:retrieveStopPayResponse></SOAP-ENV:Body></SOAP-ENV:Envelope>"
]
]
}
I have given dummy names as data1 and data2. You can separate them out further as per your needs.
Please use Grok Debugger for debugging.

Answer 2

^%{GREEDYDATA}ERROR%{GREEDYDATA}response received: %{GREEDYDATA:error_message}