Hello I'm exporting logs from RabbitMQ. In order to debug, I would like to print the content of the message before applying some regex and base64 decode. I'm trying to do something like this:
input {
rabbitmq {
host => "***host***"
vhost => "/"
exchange => "Exchange"
key => "#"
}
}
filter {
ruby {
code => "print event['message']"
}
}
output {
elasticsearch {
host => "localhost"
}
}
But I only have nil values for my messages.
Use this to print the message.
input {
stdin{}
}
filter {
ruby {
code => "
puts event['message']
"
}
}
output {
stdout {
codec => "rubydebug"
}
}
FYI.
filter {
ruby {
code =>'hash=Hash[event]
puts hash
'
}
}
Related
I'm receving events in Logstash containing measurement, values and tags. I do not know ahead of time what field there are and what tags. So i wanted to do something like this:
input {
http {}
}
filter {
ruby {
code => '
tags = event.get("stats_tags").split(",")
samples = event.get("stats_samples").split(" ")
datapoints = {}
samples.each {|s|
splat = s.split(" ")
datapoints[splat[0]] = splat[1]
}
event.set("[#metadata][stats-send-as-tags]", tags)
event.set("[#metadata][stats-datapoints]", datapoints)
'
}
}
output {
influxdb {
host => "influxdb"
db => "events_db"
measurement => measurement
send_as_tags => [#metadata][stats-send-as-tags]
data_points => [#metadata][stats-datapoints]
}
}
But this produce error. After much googling to no avail i'm starting to think this is imposible.
Is there a way to pass hash and array from event field to output/filter configuration?
EDIT: If i doublequote it, the error i'm getting is
output {
influxdb {
# This setting must be a hash
# This field must contain an even number of items, got 1
data_points => "[#metadata][stats-datapoints]"
...
}
}
I have the following ruby hash.
config = {
'output' => {
'elasticsearch' => {
'hosts' => ['localhost:9200']
}
}
}
Which I'm trying to represent as a logstash configuration file (https://www.elastic.co/guide/en/logstash/current/configuration.html). In this case, something that looks similar to this.
output {
elasticsearch { hosts => ["localhost:9200"] }
}
I've tried using map which is close, but "elasticsearch" should not have "=>" and "elasticsearch" and "hosts" should not be quoted.
puts config.map{|k, v| "#{k} #{v}"}.join('&')
output {"elasticsearch"=>{"hosts"=>["localhost:9200"]}}
I've also tried converting to json and using gsub, but in this case I need to unindent the string and "output" and "elasticsearch" should not be quoted.
puts JSON.pretty_generate(config).gsub(/^[{}]$/, "")
.gsub(": {", " {")
.gsub(": ", " => ")[1..-2]
"output" {
"elasticsearch" {
"hosts" => [
"localhost:9200"
]
}
}
While each implementation is close, it's still off by a bit. Is there a simple way to achieve this?
The Logstash config format isn't standard JSON or anything. It may be best to just write a serializer for it. I took a quick stab at it:
def serialize_config(config, tabs = 0)
clauses = []
config.each do |key, val|
case val
when Hash
clauses << format("%s {\n%s%s}", key, serialize_config(val, tabs + 1), "\t" * tabs)
else
clauses << format("%s => %s", key, val.inspect)
end
end
clauses.map {|c| format("%s%s\n", "\t" * tabs, c) }.join
end
config = {
'output' => {
'elasticsearch' => {
'hosts' => ['localhost:9200']
},
'ruby' => {
"code" => "event.cancel if rand <= 0.90"
}
}
}
puts serialize_config(config)
When gives output:
output {
elasticsearch {
hosts => ["localhost:9200"]
}
ruby {
code => "event.cancel if rand <= 0.90"
}
}
You'd want to check it against more complex Logstash configs, though.
The only thing that is certain about [url][queryString] is that it begins with 404; or that the key is long.I need to remove such keys.
If I use the ruby code below it gives cannot convert linked hashmap to string exception.
input {
file {
# Wildcards work, here :)
path => ["C:\Users\ppurush\Desktop\test\*.log"]
start_position => "beginning"
}
}
filter {
ruby {
code=>
"
require json
my_hash = JSON.parse([url][queryString])
my_hash.delete_if { |key,value| key.to_s.match(/^404;/) }
"
}
}
output {
stdout{}
elasticsearch {
host => localhost
}
}
You get a ruby exception because your ruby code is invalid. Try this instead:
filter {
ruby {
init => "require 'json'"
code => "
my_hash = JSON.parse( event['url']['queryString'] )
my_hash.delete_if { |key,value| key.to_s.match(/^404;/) }
"
}
}
This works if your event has a 'url' => 'queryString' field which contains valid json. You might already have some kind of filter to achieve this (e.g. grok). You might also consider using logstash's built-in json filter and maybe drop to delete certain events.
EDIT:
Suppose your input is plain json (I had to tidy this up):
{"id":"val1","host":"val2","app":"val3","#timestamp":"2015-08-04T19:00:03.6429322Z","#timestampEnd":"2015-08-04T19:00:03.6429322Z","vid":"val4","vidNew":"val5","sessionId":"val6","url":{"rawUrl":"val7","path":"val8","queryString":{"404;dfdgfdgfghfhjghhhhhhhhhhhhh":""}},"net":{"method":"GET","status":"200","size":"0","timeTakenMillis":"0"},"context":{"SearchType":""}}
You can use codec => "json" in your file input.
input {
file {
path => ["C:\Users\ppurush\Desktop\test\*.log"]
start_position => "beginning"
codec => "json"
}
}
You will get a field:
"url" => {
"rawUrl" => "val7",
"path" => "val8",
"queryString" => {
"404;dfdgfdgfghfhjghhhhhhhhhhhhh" => ""
}
}
So 404;dfdgfdgfghfhjghhhhhhhhhhhhh is a variable, too. To check for it and delete the event you could do something like this:
if [url][queryString][404;dfdgfdgfghfhjghhhhhhhhhhhhh] {
drop {}
}
I have a log with a format similar to:
name=johnny amount=30 uuid=2039248934
The problem is I am using this parser on multiple log files with each basically containing numerous kv pairs.
Is there a way to recognize when values are integers and cast them as such without having to use mutate on every single key value pair?(Rather than a string)
I found this link but it was very vague in where the template json file was suppose to go and how I was to go about using it.
Can kv be told to auto-detect numeric values and emit them as numeric JSON values?
You can use ruby plugin to do it.
input {
stdin {}
}
filter {
ruby {
code => "
fieldArray = event['message'].split(' ');
for field in fieldArray
name = field.split('=')[0];
value = field.split('=')[1];
if value =~ /\A\d+\Z/
event[name] = value.to_i
else
event[name] = value
end
end
"
}
}
output {
stdout { codec => rubydebug }
}
First, split the message to an array by SPACE.
Then, for each k,v mapping, check whether the value is numberic, if YES, convert it to Integer.
Here is the sample output for your input:
{
"message" => "name=johnny amount=30 uuid=2039248934",
"#version" => "1",
"#timestamp" => "2015-06-25T08:24:39.755Z",
"host" => "BEN_LIM",
"name" => "johnny",
"amount" => 30,
"uuid" => 2039248934
}
Update Solution for Logstash 5:
input {
stdin {}
}
filter {
ruby {
code => "
fieldArray = event['message'].split(' ');
for field in fieldArray
name = field.split('=')[0];
value = field.split('=')[1];
if value =~ /\A\d+\Z/
event.set(name, value.to_i)
else
event.set(name, value)
end
end
"
}
}
output {
stdout { codec => rubydebug }
}
Note, if you decide to upgrade to Logstash 5, there are some breaking changes:
https://www.elastic.co/guide/en/logstash/5.0/breaking-changes.html
In particular, it is the event that needs to be modified to use either event.get or event.set. Here is what I used to get it working (based on Ben Lim's example):
input {
stdin {}
}
filter {
ruby {
code => "
fieldArray = event.get('message').split(' ');
for field in fieldArray
name = field.split('=')[0];
value = field.split('=')[1];
if value =~ /\A\d+\Z/
event.set(name, value.to_i)
else
event.set(name, value)
end
end
"
}
}
output {
stdout { codec => rubydebug }
}
I'm trying display some Mongo data that I've been collecting using logstash using the Mongostat tool. It displays things with a suffix like "b", "k", "g" to signify byte, kilobyte, gigabyte, which is fine if I'm just reading the output, but I want to throw this into kibana and display it in a graphical format to see trends.
I've done this with several other log files and everything is fine. When I use a grok filter everything is fine but I've added a Ruby filter and now data seems to be duplicated in all fields other than the logstash generated fields and my new field created in my Ruby filter.
Here is the relevant parts of my conf file:
input {
file {
path => "/var/log/mongodb/mongostat.log"
type => "mongostat"
start_position => "end"
}
}
filter {
if [type] == "mongostat" {
grok {
patterns_dir => "/opt/logstash/patterns"
match => ["message","###a bunch of filtering that i know works###"]
add_tag => "mongostat"
}
if [mongoMappedQualifier] == 'b' {
ruby {
code => "event['mongoMappedKB'] = event['mongoMapped'].to_f / 1024"
}
}
if [mongoMappedQualifier] == 'k' {
ruby {
code => "event['mongoMappedKB'] = event['mongoMapped'].to_f * 1"
}
}
if [mongoMappedQualifier] == 'm' {
ruby {
code => "event['mongoMappedKB'] = event['mongoMapped'].to_f * 1024"
}
}
if [mongoMappedQualifier] == 'g' {
ruby {
code => "event['mongoMappedKB'] = event['mongoMapped'].to_f * 1048576"
}
}
}
}
output {
if [type] == "mongostat" {
redis {
host => "redis"
data_type => "list"
key => "logstash-mongostat"
}
}
}
Any idea why or how this can be fixed?