I'm trying to configure filter mutates properly for a logastash configuration file, but I fail.
I have a tags array which comes from SQL as a string like this:
"lunch | mellanmål | middag"
I configure mutate to split it into collection like this:
mutate { split => ["meal_type_tags", " | "] }
I'm getting an index with an expected collection:
"meal_type_tags" : ["lunch", "mellanmål","middag"]
I'm trying rename to nest it inside a new tags object by adding another mutate:
mutate { rename => "meal_type_tags" => "[tags][meal_type]"}
My expected index is:
"tags": {
"meal_type" :
[
"lunch",
"mellanmål"
"middag"
]
}
...but I get this instead:
"tags" : [
[
"meal_type",
[
"middag",
"lunch"
]
]
]
I don't know why but it just wouldn't work if I tried rename to tags. I changed to recipe_tags and it worked I expected. Maybe #baudsp know more, as the solution was inspired by his comment.
Related
I am receiving a JSON object with an array property. I would like to search the array and save only the element that matches my criteria. My input looks like this:
{
"identifier": [
{ "system" : "Source1", "value" : "TheValueIDontWant"},
{ "system" : "Source2", "value" : "TheValueIWant"}
]
}
and I would like my output to look like this:
{
"SourceID": "TheValueIWant"
}
So in this case, I want to search the identifier array for the element which has Source2 as the system and save its corresponding value to my new property.
Is there a way to do this in Logstash?
Thanks
Got this answer from someone on the elastic forum. Using ruby was indeed the answer and this is how:
ruby {
code => '
ids = event.get("identifier")
if ids.is_a? Array
ids.each { |x|
if x["system"] == "Source2"
event.set("SourceID", x["value"])
end
}
end
'
}
I have logs in new line separated JSON like following
{
"httpRequest": {
"requestMethod": "GET",
"requestUrl": "/foo/submit?proj=56"
}
}
Now I need the url without the dynamic parts in the i.e. 1st resource (someTenant) and the query parameters to be added as a field in elasticsearch ie. the expected normalised url is
"requestUrl": "/{{someTenant}}/submit?{{someParams}}"
I already have the following filter in logstash config but not sure how to do sequence of regex operation on a specific field and add it as a new one.
json{
source => "message"
}
This way I could aggregate the unique endpoints although the urls are different in logs due to variable path params and query params.
Since this question tagged with grok, i will go ahead and assume you can use grok filters.
use grok filter and create a new field from requestUrl field, you can then use URIPATHPARAM grok pattern to separate various components from requestUrl as follows,
grok {
match => {"requestUrl" => "%{URIPATHPARAM:request_data}"}
}
this will produce following output,
{
"request_data": [
[
"/foo/submit?proj=56"
]
],
"URIPATH": [
[
"/foo/submit"
]
],
"URIPARAM": [
[
"?proj=56"
]
]
}
Can be tested on Grok Online Debugger
thanks
I'm searching on internet a way to put a variable in logstash and use or modify the value if a term is corresponding to a pattern.
Here, the is an example of my data source:
2017-04-12 15:49:57,641|OK|file1|98|||
2017-04-12 15:49:58,929|OK|file2|1387|null|msg_fils|
2017-04-12 15:49:58,931|OK|file3|2|msg_pere|msg_fils|
2017-04-12 15:50:17,666|OK|file1|25|||
2017-04-12 15:50:17,929|OK|file2|1387|null|msg_fils|
I'm using this grok code to parse my source.
grok {
match => {"message" => '%{TIMESTAMP_ISO8601:msgdates:date}\|%{WORD:verb}\|%{DATA:component}\|%{NUMBER:temps:int}\|%{DATA:msg_pere}\|%{DATA:msg_fils}\|'}
}
But in fact I want to modify the first field by the previous value of the line which contains file1
Can you tell me if it's possible or not?
Thanks
I have found a solution to my issue. I'm sharing you the solution to my problem.
I'm using a plugin named logstash-filter-memorize, it can be install by the command :
logstash-plugin install logstash-filter-memorize
So my filter is like this :
grok {
match => {"message" => '%{TIMESTAMP_ISO8601:msgdates:date}\|%{WORD:verb}\|%{DATA:component}\|%{NUMBER:temps:int}\|%{DATA:msg_pere}\|%{DATA:msg_fils}\|'}
}
if [component] =~ "file1" {
mutate {
add_field => [ "msg_id", "%{msgdates}" ]
}
memorize {
fields => [ "msg_id" ]
default => { "msg_id" => "NOTFOUND" }
} }
memorize {
fields => [ "msg_id9" ]
}
I hope that it can be useful for others.
I am using logstash to get data from a sql database. There is a field called "code" in which the content has
this structure:
PO0000001209
ST0000000909
And what I would like to do is to remove the 6 zeros after the letters to get the following result:
PO1209
ST0909
I will put the result in another field called "code_short" and use it for my query in elasticsearch. I have configured the input
and the output in logstash but I am not sure how to do it using grok or maybe mutate filter
I have read some examples but I am quite new on this and I am a bit stuck.
Any help would be appreciated. Thanks.
You could use a mutate/gsub filter for this but that will replace the value of the code field:
filter {
mutate {
gsub => [
"code", "000000", "",
]
}
}
Another option is to use a grok filter like this:
filter {
grok {
match => { "code" => "(?<prefix>[a-zA-Z]+)000000%{INT:suffix}" }
add_field => { "code_short" => "%{prefix}%{suffix}"}
}
}
I am trying to use logstash/elasticsearch.
First, I have tried to put an xml (table) into logstash but, it seemed that xml was unreadable, so I converted it into a json array looking like this:
[
["bla","blieb"],
["things",more"],
]
my config looks like this:
input {
file {
path => "C:\Users\mipmip\Downloads\noch.json"
start_position => "beginning"
}
}
filter {
json {source => message
}
}
output {
elasticsearch{
hosts => "localhost"
index => "datensatz"
}
stdout { }
}
But it still doesn't work, all I get are a lot of _jsonparsefailures in elasticsearch :(
But whyyyy D:
[
["bla","blieb"],
["things",more"],
]
This is not a JSON object.
First, you are missing a double quote near "more". Second, you have an extra comma after the second object. I recommend checking with jsonlint.com if you have a valid JSON.
You should also surround the "message" with double quotes, in the filter part.