How to send different logstash event to different output - elasticsearch

There are many events as fields that in logstash filter section are extracted from message field like below:
match => ["message", "%{type1:f1} %{type2:f2} %{type3:f3}"]
The purpose is to send f1, f2, f3 to one output and only f1 and f3 to other output plugin such that:
output {
elasticsearch {
action => "index"
hosts => "localhost"
index =>"indx1-%{+YYYY-MM}"
.
}
}
output {
elasticsearch {
action => "index"
hosts => "localhost"
index =>"indx2-%{+YYYY-MM}"
}
}
The problem is that all events are involved in every output pluggin but I want to handle which events goes to which output plugin.Is it possible to do this?

I found a solution by using filebeat to forward data to logstash.
If running two instancea of filebeat and one instance of logstash, each filebeat forwarda input data to the same logstash but with different type like:
document_type: type1
In logstash, appropriate filter and output is exceuted using if clause:
filter {
if [type] == "type1" {
}
else {
}
}
output {
if [type] == "type1" {
elasticsearch {
action => "index"
hosts => "localhost"
index => "%{type}-%{+YYYY.MM}"
}
}
else {
elasticsearch {
action => "index"
hosts => "localhost"
index => "%{type}-%{+YYYY.MM}"
}
}
}

If you have two distinct matching patterns in the "filter" section, then you can add specific "tags" for each match. Then in the output section use something like this:
if "matchtype1" in [tags] {
elasticsearch {
hosts => "localhost"
index => "indxtype1-%{+YYYY.MM}"
}
}
if "matchtype2" in [tags]{
elasticsearch {
hosts => "localhost"
index => "indxtype2-%{+YYYY.MM}"
}
}

Related

Read a CSV in Logstash level and filter on basis of the extracted data

I am using Metricbeat to get process-level data and push it to Elastic Search using Logstash.
Now, the aim is to categorize the processes into 2 tags i.e the process running is either a browser or it is something else.
I am able to do that statically using this block of code :
input {
beats {
port => 5044
}
}
filter{
if [process][name]=="firefox.exe" or [process][name]=="chrome.exe" {
mutate {
add_field => { "process.type" => "browsers" }
convert => {
"process.type" => "string"
}
}
}
else {
mutate {
add_field => { "process.type" => "other" }
}
}
}
output {
elasticsearch {
hosts => "localhost:9200"
# manage_template => false
index => "metricbeatlogstash"
}
}
But when I try to make that if condition dynamic by reading the process list from a CSV, I am not getting any valid results in Kibana, nor a error on my LogStash level.
The CSV config file code is as follows :
input {
beats {
port => 5044
}
file{
path=>"filePath"
start_position=>"beginning"
sincedb_path=>"NULL"
}
}
filter{
csv{
separator=>","
columns=>["processList","IT"]
}
if [process][name] in [processList] {
mutate {
add_field => { "process.type" => "browsers" }
convert => {
"process.type" => "string"
}
}
}
else {
mutate {
add_field => { "process.type" => "other" }
}
}
}
output {
elasticsearch {
hosts => "localhost:9200"
# manage_template => false
index => "metricbeatlogstash2"
}
}
What you are trying to do does not work that way in logstash, the events in a logstash pipeline are independent from each other.
The events received by your beats input have no knowledge about the events received by your csv input, so you can't use fields from different events in a conditional.
To do what you want you can use the translate filter with the following config.
translate {
field => "[process][name]"
destination => "[process][type]"
dictionary_path => "process.csv"
fallback => "others"
refresh_interval => 300
}
This filter will check the value of the field [process][name] against a dictionary, loaded into memory from the file process.csv, the dictionary is a .csv file with two columns, the first is the name of the browser process and the second is always browser.
chrome.exe,browser
firefox.exe,browser
If the filter got a match, it will populate the field [process][type] (not process.type) with the value from the second column, in this case, always browser, if there is no match, it will populate the field [process][type] with the value of the fallback config, in this case, others, it will also reload the content of the process.csv file every 300 seconds (5 minutes)

elaticsearch monitoring search queries

For more than a week I am struggling with trying to log into an index in elasticsearch information regarding queries which I run so I could compare performance between different types of queries. I have configured this config file on logstash home directory
input {
beats {
port> 5044
}
}
filter {
if "search" in [request]{
grok {
match => { "request" => ".*\n\{(?<query_body>.*)"}
}
grok {
match => { "path" => "\/(?<index>.*)\/_search"}
}
if [index] {
} else {
mutate {
add_field => { "index" => "All" }
}
}
mutate {
update => { "query_body" => "{%{query_body}" }
}
}
}
output {
if "search" in [request] and "ignore_unmapped" not in [query_body]{
elasticsearch {
hosts => "http://localhost:9200"
}
}
}
and also installed and configured packetbeat.yml
logstash hosts to :http://localhost:9200
Also in the tutorial that I have followed is mentioned that after starting Packetbeat it will listen for packets on 9200 sending them to Logstash and from there to the monitoring Elasticsearch cluster, it will be indexed in indexes like: logstash-2016.05.24. But these indexes does not exists.

Parse url parameters in logstash

I'm sending data in a url to logstash 5.2 and I would like to parse it in logstash, so every url parameter becomes a variable in logstash and I can visualize it properly in kibana.
http://127.0.0.1:31311/?id=ID-XXXXXXXX&uid=1-37zbcuvs-izotbvbe&ev=pageload&ed=&v=1&dl=http://127.0.0.1/openpixel/&rl=&ts=1488314512294&de=windows-1252&sr=1600x900&vp=1600x303&cd=24&dt=&bn=Chrome%2056&md=false&ua=Mozilla/5.0%20(Macintosh;%20Intel%20Mac%20OS%20X%2010_11_3)%20AppleWebKit/537.36%20(KHTML,%20like%20Gecko)%20Chrome/56.0.2924.87%20Safari/537.36&utm_source=&utm_medium=&utm_term=&utm_content=&utm_campaign=
This is my logstash conf file:
input
{
http
{
host => "127.0.0.1"
port => 31311
}
}
output
{
elasticsearch
{
hosts => ["localhost:9200"]
}
stdout
{
codec => rubydebug
}
}
You could use the grok filter to match your params in your url as such:
filter {
grok {
match => [ "message", "%{URIPARAM:url}" ]
}
And then you might have to use kv filter in order to split your data:
kv {
source => "url"
field_split => "&"
}
This SO might become handy. Hope this helps!

logstash output to elasticsearch with document_id; what to do when I don't have a document_id?

I have some logstash input where I use the document_id to remove duplicates. However, most input doesn't have a document_id. The following plumbs the actual document_id through, but if it doesn't exist, it gets accepted as literally %{document_id}, which means most documents are seen as a duplicate of each other. Here's what my output block looks like:
output {
elasticsearch_http {
host => "127.0.0.1"
document_id => "%{document_id}"
}
}
I thought I might be able to use a conditional in the output. It fails, and the error is given below the code.
output {
elasticsearch_http {
host => "127.0.0.1"
if document_id {
document_id => "%{document_id}"
}
}
}
Error: Expected one of #, => at line 101, column 8 (byte 3103) after output {
elasticsearch_http {
host => "127.0.0.1"
if
I tried a few "if" statements and they all fail, which is why I assume the problem is having a conditional of any sort in that block. Here are the alternatives I tried:
if document_id <> "" {
if [document_id] <> "" {
if [document_id] {
if "hello" <> "" {
You're close with the conditional idea but you can't place it inside a plugin block. Do this instead:
output {
if [document_id] {
elasticsearch_http {
host => "127.0.0.1"
document_id => "%{document_id}"
}
} else {
elasticsearch_http {
host => "127.0.0.1"
}
}
}
(But the suggestion in one of the other answers to use the uuid filter is good too.)
One way to solve this is to make sure a document_idis always available. You can achieve this by adding a UUID filter in the filter section that would create the document_id field if it is not present.
filter {
if "" in [document_id] {
uuid {
target => "document_id"
}
}
}
Edited per Magnus Bäck's suggestion. Thanks!
Reference : docinfo_fields
For any document added in elasticsearch, the _id is auto-generated if not specified during insert. We can use this same _id later to update/delete/search queries by using docinfo_fields feature.
Example :
filter {
json {
source => "message"
}
elasticsearch {
hosts => "http://localhost:9200/"
user => elastic
password => elastic
query => "..."
docinfo_fields => {
"_id" => "docid"
"_index" => "document_index"
}
}
if ("_elasticsearch_lookup_failure" not in [tags]) {
#... doc update logic ...
}
}
output {
elasticsearch {
hosts => "http://localhost:9200/"
user => elastic
password => elastic
index => "%{document_index}"
action => "update"
doc_as_upsert => true
document_id => "%{docid}"
}
}

Logstash configuration condition

i am new to Logstash, elasticsearch.
I have NodeJS app, where i am sending logs trough Winston:Redis. I have different types of logs, like Requests, system, etc. And i want these logs to be in separate index_type inside ElasticSearch.
I am sending these keys fe. : "web:production:request", "web:production:system" and im sending JSON obejcts.
My configuration is:
NodeJS (Winston Redis client) -> Redis -> Logstash -> Elastic search
Its working good, except index_types.
I have 1 redis client (stream/subcribe) and i want to filter these logs depending on key value to different index_types inside elastic search output.
I tried this config:
input {
redis {
host => "127.0.0.1"
data_type => "pattern_channel"
key => "web:production:*"
codec => json
}
filter {
if [key] == "web:production:request" {
alter {
add_field => { "index_type" => "request" }
}
}
if [key] == "web:production:system" {
alter {
add_field => { "index_type" => "system" }
}
}
}
output {
elasticsearch {
index => "web-production-%{+YYYY.MM.dd}"
index_type => "%{index_type}"
# THIS IS NOT WORKING
protocol => "http"
}
}
So questions are:
How do conditionals right ?
How would you proceed if you want to send different indexes depending on conditions
I cannot have condition inside command ? fe. grok { if [key] == "1" {} } ?
suggestion for a workaround:
output {
if [index_type] == "request"{
elasticsearch {
index => "web-production-request%{+YYYY.MM.dd}"
protocol => "http"
}
}
if [index_type] == "system"{
elasticsearch {
index => "web-production-system%{+YYYY.MM.dd}"
protocol => "http"
}
}
}

Resources