How to add a custom key-value pair to a grok pattern? - filter

How can I add a custom key-value pair to a grok pattern?
For example, I would like to add a key-value pair of "city": [["New York]] to the data result, even though it doesn't exist in the log line.
How do I do this? Tyvm, Keith :^)
Complete, Minimal, and Verifiable Example
Data:
WARN 10/11/2017 kmiklas
Grok:
%{WORD:logLevel}\s%{DATE:date}\s%{USER:user}
{
"logLevel": [
[
"WARN"
]
],
"date": [
[
"10/11/2017"
]
],
"DATE_US": [
[
"10/11/2017"
]
],
"MONTHNUM": [
[
"10",
null
]
],
"MONTHDAY": [
[
"11",
null
]
],
"YEAR": [
[
"2017",
null
]
],
"DATE_EU": [
[
null
]
],
"user": [
[
"kmiklas"
]
],
"USERNAME": [
[
"kmiklas"
]
]
}


I understand that will be a fixed field, so you need to use the mutate method, like this:
mutate { add_field => { "city" => [["New York"]] } }
if you want the new field to be only in some logs you need to include if
if "some_test" in [message]{mutate.....}

Related

Setup a Kibana dashboard for data about last Jenkins build

I use Kibana to show data about automated test cases stored in a single elastic search index.
These tests can be repeated multiple times during the day and right now are identified by a build number that comes from Jenkins. So, if I want to see the latest results, I need to add a filter in my dashboards where I set the last known value of the build number.
Is there a way to automatically show in a dashboard the values about the last build?
Thank you.
EDIT: Here's a data sample:
{
"_index": "data",
"_type": "_doc",
"_id": "33rugH0B0CwJH7IcV11v",
"_score": 1,
"_source": {
"market": "FRA",
"price_code": "DIS22FREH1003",
"test_case_id": "NPM_14",
"environment": "PROD",
"cruise_id": "DI20220707CPVCP1",
"jenkins_job_name": "MonitoringNPM_14",
"#timestamp": "2021-12-03T16:34:03.360+0100",
"jenkins_job_number": 8,
"agency": "FR900000",
"fail_code": "IncorrectGuarantee",
"build_number": 8,
"category": "IR2"
},
"fields": {
"environment.keyword": [
"PROD"
],
"test_case_id": [
"NPM_14"
],
"category.keyword": [
"IR2"
],
"price_code": [
"DIS22FREH1003"
],
"cruise_id": [
"DI20220707CPVCP1"
],
"price_code.keyword": [
"DIS22FREH1003"
],
"agency": [
"FR900000"
],
"jenkins_job_number": [
"8"
],
"agency.keyword": [
"FR900000"
],
"jenkins_job_number.keyword": [
"8"
],
"market": [
"FRA"
],
"jenkins_job_name.keyword": [
"MonitoringNPM_14"
],
"test_case_id.keyword": [
"NPM_14"
],
"environment": [
"PROD"
],
"#timestamp": [
"2021-12-03T15:34:03.360Z"
],
"jenkins_job_name": [
"MonitoringNPM_14"
],
"fail_code.keyword": [
"IncorrectGuarantee"
],
"fail_code": [
"IncorrectGuarantee"
],
"build_number": [
8
],
"market.keyword": [
"FRA"
],
"cruise_id.keyword": [
"DI20220707CPVCP1"
],
"category": [
"IR2"
]
}
}

To index geojson data to elasticsearch using curl

I'd like to index a geojson data to elasticsearch using curl
The geojson data looks like this:
{
"type": "FeatureCollection",
"name": "telco_new_development",
"crs": { "type": "name", "properties": { "name": "urn:ogc:def:crs:OGC:1.3:CRS84" } },
"features": [
{ "type": "Feature", "properties": { "ogc_fid": 1, "name": "Yarrabilba", "carrier_name": "OptiComm", "uid": "35", "development_name": "Yarrabilba", "stage": "None", "developer_name": "Refer to Carrier", "development_nature": "Residential", "development_type": "Sub-division", "estimated_number_of_lots_or_units": "18500", "status": "Ready for service", "developer_application_date": "Check with carrier", "contract_date": "TBC", "estimated_service_date": "30 Jul 2013", "technology_type": "FTTP", "last_modified_date": "8 Jul 2020" }, "geometry": { "type": "MultiPolygon", "coordinates": [ [ [ [ 153.101112, -27.797998 ], [ 153.09786, -27.807122 ], [ 153.097715, -27.816313 ], [ 153.100598, -27.821068 ], [ 153.103789, -27.825047 ], [ 153.106079, -27.830225 ], [ 153.108248, -27.836107 ], [ 153.110692, -27.837864 ], [ 153.116288, -27.840656 ], [ 153.119923, -27.844818 ], [ 153.122317, -27.853523 ], [ 153.127785, -27.851777 ], [ 153.131234, -27.85115 ], [ 153.135634, -27.849741 ], [ 153.138236, -27.848668 ], [ 153.141703, -27.847075 ], [ 153.152205, -27.84496 ], [ 153.155489, -27.843381 ], [ 153.158613, -27.841546 ], [ 153.161937, -27.84059 ], [ 153.156361, -27.838492 ], [ 153.157097, -27.83451 ], [ 153.15036, -27.832705 ], [ 153.151126, -27.827536 ], [ 153.15169, -27.822564 ], [ 153.148492, -27.820801 ], [ 153.148375, -27.817969 ], [ 153.139019, -27.815804 ], [ 153.139814, -27.808556 ], [ 153.126486, -27.80576 ], [ 153.124679, -27.803584 ], [ 153.120764, -27.802953 ], [ 153.121397, -27.797353 ], [ 153.100469, -27.79362 ], [ 153.099828, -27.793327 ], [ 153.101112, -27.797998 ] ] ] ] } },
{ "type": "Feature", "properties": { "ogc_fid": 2, "name": "Elliot Springs", "carrier_name": "OptiComm", "uid": "63", "development_name": "Elliot Springs", "stage": "None", "developer_name": "Refer to Carrier", "development_nature": "Residential", "development_type": "Sub-division", "estimated_number_of_lots_or_units": "11674", "status": "Ready for service", "developer_application_date": "Check with carrier", "contract_date": "TBC", "estimated_service_date": "29 Nov 2018", "technology_type": "FTTP", "last_modified_date": "8 Jul 2020" }, "geometry": { "type": "MultiPolygon", "coordinates": [ [ [ [ 146.862725, -19.401424 ], [ 146.865987, -19.370253 ], [ 146.872767, -19.370901 ], [ 146.874484, -19.354706 ], [ 146.874913, -19.354301 ], [ 146.877059, -19.356811 ], [ 146.87972, -19.35835 ], [ 146.889161, -19.359321 ], [ 146.900062, -19.367581 ], [ 146.884955, -19.38507 ], [ 146.88341, -19.402558 ], [ 146.862725, -19.401424 ] ] ] ] } },
...
However, my curl is returns an error called The bulk request must be terminated by a newline [\\n]
curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/geo/building/_bulk?pretty' --data-binary #building.geojson
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "The bulk request must be terminated by a newline [\\n]"
}
],
"type" : "illegal_argument_exception",
"reason" : "The bulk request must be terminated by a newline [\\n]"
},
"status" : 400
}
Any suggestion?

your format is not suitable for _bulk like that, as it's missing the structure it expects. https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html goes into that
you need;
to update your json file to have something like { "index" : { "_index" : "INDEX-NAME-HERE" } } before each of the documents
each document also needs to be on a single line
each line needs a \n at the end of it so that the bulk API knows when the action/record ends

what is the json mapping to insert geo data into elasticsearch?

what would be the json mapping to insert geo data into elasticsearch ??
if the sample json data as follows:
{ "type": "Feature", "properties": { "ID": 631861455.000000, "address": "1206 UPPER", "city": "la vegas", "state": "AL", "zip_code": "15656", "OGR_GEOMETRY": "POLYGON" }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -86.477551331, 32.490605650000099 ], [ -86.477637350999899, 32.4903921820001 ], [ -86.478257247, 32.490565591000099 ], [ -86.478250466, 32.490580239000103 ], [ -86.478243988, 32.490593680000096 ], [ -86.47823751, 32.490607122 ], [ -86.478231749, 32.490619100000096 ], [ -86.478224637, 32.490634065000101 ], [ -86.47821823699999, 32.490647540000097 ], [ -86.478211847999901, 32.490661035000095 ], [ -86.478205478999897, 32.490674526000099 ], [ -86.478202107999891, 32.490681666000093 ], [ -86.478199132, 32.4906880240001 ], [ -86.478192825999898, 32.490701523 ], [ -86.478186533, 32.490715047 ], [ -86.47818320899999, 32.490722209000097 ], [ -86.47818027999989, 32.490728569000098 ], [ -86.478174063, 32.490742125000097 ], [ -86.47816785099999, 32.490755654000097 ], [ -86.47816255799999, 32.490767236000096 ], [ -86.478159053999889, 32.490774513000105 ], [ -86.477551331, 32.490605650000099 ] ] ] } }

Look at Geo point mapping.
You need to define mapping.

How to use the logstash mutate or ruby filter

I have the following Json syntax
{"result": {
"entities": {
"SERVICE-CCC89FB0A922657A": "service1",
"SERVICE-D279F46CD751424F": "service2",
"SERVICE-7AB760E70FCDCA18": "service3",
},
"dataPoints": {
"SERVICE-CCC89FB0A922657A": [
[
1489734240000,
1101.0
],
[
1489734300000,
null
]
],
"SERVICE-7AB760E70FCDCA18": [
[
1489734240000,
4080800.5470588235
],
[
1489734300000,
null
]
],
"SERVICE-D279F46CD751424F": [
[
1489734240000,
26677.695652173912
],
[
1489734300000,
null
]
]
}
},
"#timestamp": "2017-03-17T07:05:37.531Z",
"data": "data",
"#version": "1"
}
I want to change the following and input it in elasticsearch.
{"#timestamp": "2017-03-17T07:05:37.531Z",
"data": "data",
"#version": "1",
"data" : {
"service1",: [
[
1489734240000,
1101.0
],
[
1489734300000,
null
]
],
"service3" : [
[
1489734240000,
4080800.5470588235
],
[
1489734300000,
null
]
],
"service2": [
[
1489734240000,
26677.695652173912
],
[
1489734300000,
null
]
]
}
}
This is the contents of the current logstash conf file.
input {
http_poller {
urls => {
test => {
method => get
url => "https://xxxx.com"
headers => {
Accept => "application/json"
}
}
}
request_timeout => 60
schedule => { every => "60s" }
codec => "plain"
}
}
filter {
json{
source => "message"
remove_field => ["[result][aggregationType]","message"]
}
# translate{
# }
# mutate{
# }
# ruby{
# }
}
output {
stdout {
codec => rubydebug {
#metadata => true
}
}
elasticsearch {
hosts => ["http://192.168.0.36:9200"]
}
}
I have just used elasticsearch and I do not know how to implement what filter to use.
I wonder if it is possible to implement the contents of the mutate filter rename.
Or should I implement code with ruby ​​filters?
It is likely that the entities will be arrayed with the ruby ​​filter to match the SERVICE- * s of the dataPoints.
However, it is difficult to cope with Ruby code.
I want you to help me.
Thank you.

Here are couple of filters are used for logstash...
https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html

Logstash Grok Filter Apache Access Log

I have been looking around here and there, but could not find the working resolution. I try to use Grok Filter inside the Logstash config file to filter Apache-Access log file. The log message looks like this: {"message":"00.00.0.000 - - [dd/mm/YYYY:hh:mm:ii +0000] \"GET /index.html HTTP/1.1\" 200 00"}.
On this moment I could only filter the client ip by using grok { match => [ "message", "%{IP:client_ip}" ] }.
I want to filter:
- The GET method,
- requested page (index.html),
- HTTP/1.1\,
- server response 200
- the last number 00 after 200 inside the message body
Please note that none of these does not work for me :
grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
or
grok { match => [ "message", "%{COMBINEDAPACHELOG}" ] }

Use the Grok Debugger to get an exact match on your log format. Its the only way.
http://grokdebug.herokuapp.com/

grok {
match => [ "message", "%{IP:client_ip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:apache_timestamp}\] \"%{WORD:method} /%{NOTSPACE:request_page} HTTP/%{NUMBER:http_version}\" %{NUMBER:server_response} " ]
}

Use the following:
filter {
grok {
match => { "message" => "%{COMMONAPACHELOG}" }
}
}
As you can see from your pattern COMBINEDAPACHELOG would fail because there are some missing components:
COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}
https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns

You can use COMBINEDAPACHELOG pattern for this,
%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}
for instance, consider this sample apache log
111.222.333.123 HOME - [01/Feb/1998:01:08:46 -0800] "GET /bannerad/ad.htm HTTP/1.0" 200 28083
"http://www.referrer.com/bannerad/ba_intro.htm" "Mozilla/4.01
(Macintosh; I; PPC)"
above filter will produce,
{
"clientip": [
[
"111.222.333.123"
]
],
"HOSTNAME": [
[
"111.222.333.123"
]
],
"IP": [
[
null
]
],
"IPV6": [
[
null
]
],
"IPV4": [
[
null
]
],
"ident": [
[
"HOME"
]
],
"USERNAME": [
[
"HOME",
"-"
]
],
"auth": [
[
"-"
]
],
"timestamp": [
[
"01/Feb/1998:01:08:46 -0800"
]
],
"MONTHDAY": [
[
"01"
]
],
"MONTH": [
[
"Feb"
]
],
"YEAR": [
[
"1998"
]
],
"TIME": [
[
"01:08:46"
]
],
"HOUR": [
[
"01"
]
],
"MINUTE": [
[
"08"
]
],
"SECOND": [
[
"46"
]
],
"INT": [
[
"-0800"
]
],
"verb": [
[
"GET"
]
],
"request": [
[
"/bannerad/ad.htm"
]
],
"httpversion": [
[
"1.0"
]
],
"BASE10NUM": [
[
"1.0",
"200",
"28083"
]
],
"rawrequest": [
[
null
]
],
"response": [
[
"200"
]
],
"bytes": [
[
"28083"
]
],
"referrer": [
[
""http://www.referrer.com/bannerad/ba_intro.htm""
]
],
"QUOTEDSTRING": [
[
""http://www.referrer.com/bannerad/ba_intro.htm"",
""Mozilla/4.01 (Macintosh; I; PPC)""
]
],
"agent": [
[
""Mozilla/4.01 (Macintosh; I; PPC)""
]
]
}
can be tested here,
https://grokdebug.herokuapp.com/

Resources