I have the following ruby hash.
config = {
'output' => {
'elasticsearch' => {
'hosts' => ['localhost:9200']
}
}
}
Which I'm trying to represent as a logstash configuration file (https://www.elastic.co/guide/en/logstash/current/configuration.html). In this case, something that looks similar to this.
output {
elasticsearch { hosts => ["localhost:9200"] }
}
I've tried using map which is close, but "elasticsearch" should not have "=>" and "elasticsearch" and "hosts" should not be quoted.
puts config.map{|k, v| "#{k} #{v}"}.join('&')
output {"elasticsearch"=>{"hosts"=>["localhost:9200"]}}
I've also tried converting to json and using gsub, but in this case I need to unindent the string and "output" and "elasticsearch" should not be quoted.
puts JSON.pretty_generate(config).gsub(/^[{}]$/, "")
.gsub(": {", " {")
.gsub(": ", " => ")[1..-2]
"output" {
"elasticsearch" {
"hosts" => [
"localhost:9200"
]
}
}
While each implementation is close, it's still off by a bit. Is there a simple way to achieve this?
The Logstash config format isn't standard JSON or anything. It may be best to just write a serializer for it. I took a quick stab at it:
def serialize_config(config, tabs = 0)
clauses = []
config.each do |key, val|
case val
when Hash
clauses << format("%s {\n%s%s}", key, serialize_config(val, tabs + 1), "\t" * tabs)
else
clauses << format("%s => %s", key, val.inspect)
end
end
clauses.map {|c| format("%s%s\n", "\t" * tabs, c) }.join
end
config = {
'output' => {
'elasticsearch' => {
'hosts' => ['localhost:9200']
},
'ruby' => {
"code" => "event.cancel if rand <= 0.90"
}
}
}
puts serialize_config(config)
When gives output:
output {
elasticsearch {
hosts => ["localhost:9200"]
}
ruby {
code => "event.cancel if rand <= 0.90"
}
}
You'd want to check it against more complex Logstash configs, though.
Related
I have two questions;
parsing xml data & adding it to an array in a record in an index
checking for an existing record in an index and if it exists add the new data of that record to the array of the existing record
I have an jdbc input that has an xml column,
input {
jdbc {
....
statement => "SELECT event_xml....
}
}
then an xml filter to parse the data,
How do i make the the last 3 xpaths to be an array? Do i need a mutate or ruby filter? I cant seem to figure it out
filter {
xml {
source => "event_xml"
remove_namespaces => true
store_xml => false
force_array => false
xpath => [ "/CaseNumber/text()", "case_number" ]
xpath => [ "/FormName/text()", "[conversations][form_name]" ]
xpath => [ "/EventDate/text()", "[conversations][event_date]" ]
xpath => [ "/CaseNote/text()", "[conversations][case_note]" ]
}
}
so it would something like this look like this in the Elastic search.
{
"case_number" : "12345",
"conversations" :
[
{
"form_name" : "form1",
"event_date" : "2019-01-09T00:00:00Z",
"case_note" : "this is a case note"
}
]
}
So second question is, if there is already a unique case_number of "12345" instead of creating a new record for this add the new xml values to the conversations array. so it would look like this
{
"case_number" : "12345",
"conversations" : [
{
"form_name" : "form1",
"event_date" : "2019-01-09T00:00:00Z",
"case_note" : "this is a case note"
},
{
"form_name" : "form2",
"event_date" : "2019-05-09T00:00:00Z",
"case_note" : "this is another case note"
}
]
}
my output filter
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "cases"
manage_template => false
}
}
Is this possible? thanks
this ruby filter created the array
ruby {
code => '
event.set("conversations", [Hash[
"publish_event_id", event.get("publish_event_id"),
"form_name", event.get("form_name"),
"event_date", event.get("event_date"),
"case_note", event.get("case_note")
]])
'
}
for the output was resolved by
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "cases"
document_id => "%{case_number}"
action => "update"
doc_as_upsert => true
script => "
boolean recordExists = false;
for (int i = 0; i < ctx._source.conversations.length; i++)
{
if(ctx._source.conversations[i].publish_event_id == params.event.get('conversations')[0].publish_event_id)
{
recordExists = true;
}
}
if(!recordExists){
ctx._source.conversations.add(params.event.get('conversations')[0]);
}
"
manage_template => false
}
}
The only thing that is certain about [url][queryString] is that it begins with 404; or that the key is long.I need to remove such keys.
If I use the ruby code below it gives cannot convert linked hashmap to string exception.
input {
file {
# Wildcards work, here :)
path => ["C:\Users\ppurush\Desktop\test\*.log"]
start_position => "beginning"
}
}
filter {
ruby {
code=>
"
require json
my_hash = JSON.parse([url][queryString])
my_hash.delete_if { |key,value| key.to_s.match(/^404;/) }
"
}
}
output {
stdout{}
elasticsearch {
host => localhost
}
}
You get a ruby exception because your ruby code is invalid. Try this instead:
filter {
ruby {
init => "require 'json'"
code => "
my_hash = JSON.parse( event['url']['queryString'] )
my_hash.delete_if { |key,value| key.to_s.match(/^404;/) }
"
}
}
This works if your event has a 'url' => 'queryString' field which contains valid json. You might already have some kind of filter to achieve this (e.g. grok). You might also consider using logstash's built-in json filter and maybe drop to delete certain events.
EDIT:
Suppose your input is plain json (I had to tidy this up):
{"id":"val1","host":"val2","app":"val3","#timestamp":"2015-08-04T19:00:03.6429322Z","#timestampEnd":"2015-08-04T19:00:03.6429322Z","vid":"val4","vidNew":"val5","sessionId":"val6","url":{"rawUrl":"val7","path":"val8","queryString":{"404;dfdgfdgfghfhjghhhhhhhhhhhhh":""}},"net":{"method":"GET","status":"200","size":"0","timeTakenMillis":"0"},"context":{"SearchType":""}}
You can use codec => "json" in your file input.
input {
file {
path => ["C:\Users\ppurush\Desktop\test\*.log"]
start_position => "beginning"
codec => "json"
}
}
You will get a field:
"url" => {
"rawUrl" => "val7",
"path" => "val8",
"queryString" => {
"404;dfdgfdgfghfhjghhhhhhhhhhhhh" => ""
}
}
So 404;dfdgfdgfghfhjghhhhhhhhhhhhh is a variable, too. To check for it and delete the event you could do something like this:
if [url][queryString][404;dfdgfdgfghfhjghhhhhhhhhhhhh] {
drop {}
}
I have a log with a format similar to:
name=johnny amount=30 uuid=2039248934
The problem is I am using this parser on multiple log files with each basically containing numerous kv pairs.
Is there a way to recognize when values are integers and cast them as such without having to use mutate on every single key value pair?(Rather than a string)
I found this link but it was very vague in where the template json file was suppose to go and how I was to go about using it.
Can kv be told to auto-detect numeric values and emit them as numeric JSON values?
You can use ruby plugin to do it.
input {
stdin {}
}
filter {
ruby {
code => "
fieldArray = event['message'].split(' ');
for field in fieldArray
name = field.split('=')[0];
value = field.split('=')[1];
if value =~ /\A\d+\Z/
event[name] = value.to_i
else
event[name] = value
end
end
"
}
}
output {
stdout { codec => rubydebug }
}
First, split the message to an array by SPACE.
Then, for each k,v mapping, check whether the value is numberic, if YES, convert it to Integer.
Here is the sample output for your input:
{
"message" => "name=johnny amount=30 uuid=2039248934",
"#version" => "1",
"#timestamp" => "2015-06-25T08:24:39.755Z",
"host" => "BEN_LIM",
"name" => "johnny",
"amount" => 30,
"uuid" => 2039248934
}
Update Solution for Logstash 5:
input {
stdin {}
}
filter {
ruby {
code => "
fieldArray = event['message'].split(' ');
for field in fieldArray
name = field.split('=')[0];
value = field.split('=')[1];
if value =~ /\A\d+\Z/
event.set(name, value.to_i)
else
event.set(name, value)
end
end
"
}
}
output {
stdout { codec => rubydebug }
}
Note, if you decide to upgrade to Logstash 5, there are some breaking changes:
https://www.elastic.co/guide/en/logstash/5.0/breaking-changes.html
In particular, it is the event that needs to be modified to use either event.get or event.set. Here is what I used to get it working (based on Ben Lim's example):
input {
stdin {}
}
filter {
ruby {
code => "
fieldArray = event.get('message').split(' ');
for field in fieldArray
name = field.split('=')[0];
value = field.split('=')[1];
if value =~ /\A\d+\Z/
event.set(name, value.to_i)
else
event.set(name, value)
end
end
"
}
}
output {
stdout { codec => rubydebug }
}
I'm trying display some Mongo data that I've been collecting using logstash using the Mongostat tool. It displays things with a suffix like "b", "k", "g" to signify byte, kilobyte, gigabyte, which is fine if I'm just reading the output, but I want to throw this into kibana and display it in a graphical format to see trends.
I've done this with several other log files and everything is fine. When I use a grok filter everything is fine but I've added a Ruby filter and now data seems to be duplicated in all fields other than the logstash generated fields and my new field created in my Ruby filter.
Here is the relevant parts of my conf file:
input {
file {
path => "/var/log/mongodb/mongostat.log"
type => "mongostat"
start_position => "end"
}
}
filter {
if [type] == "mongostat" {
grok {
patterns_dir => "/opt/logstash/patterns"
match => ["message","###a bunch of filtering that i know works###"]
add_tag => "mongostat"
}
if [mongoMappedQualifier] == 'b' {
ruby {
code => "event['mongoMappedKB'] = event['mongoMapped'].to_f / 1024"
}
}
if [mongoMappedQualifier] == 'k' {
ruby {
code => "event['mongoMappedKB'] = event['mongoMapped'].to_f * 1"
}
}
if [mongoMappedQualifier] == 'm' {
ruby {
code => "event['mongoMappedKB'] = event['mongoMapped'].to_f * 1024"
}
}
if [mongoMappedQualifier] == 'g' {
ruby {
code => "event['mongoMappedKB'] = event['mongoMapped'].to_f * 1048576"
}
}
}
}
output {
if [type] == "mongostat" {
redis {
host => "redis"
data_type => "list"
key => "logstash-mongostat"
}
}
}
Any idea why or how this can be fixed?
Hello I'm exporting logs from RabbitMQ. In order to debug, I would like to print the content of the message before applying some regex and base64 decode. I'm trying to do something like this:
input {
rabbitmq {
host => "***host***"
vhost => "/"
exchange => "Exchange"
key => "#"
}
}
filter {
ruby {
code => "print event['message']"
}
}
output {
elasticsearch {
host => "localhost"
}
}
But I only have nil values for my messages.
Use this to print the message.
input {
stdin{}
}
filter {
ruby {
code => "
puts event['message']
"
}
}
output {
stdout {
codec => "rubydebug"
}
}
FYI.
filter {
ruby {
code =>'hash=Hash[event]
puts hash
'
}
}