I have setup logstash to use an embedded elastisearch.
I can log events.
My logstash conf looks thus:
https://gist.github.com/khebbie/42d72d212cf3727a03a0
Now I would like to add another udp input and have that input be indexed in another index.
Is that somehow possible?
I would do it to make reporting easier, so I could have system log events in one index, and business log events in another index.
Use an if conditional in your output section, based on e.g. the message type or whatever message field is significant to the choice of index.
input {
udp {
...
type => "foo"
}
file {
...
type => "bar"
}
}
output {
if [type] == "foo" {
elasticsearch {
...
index => "foo-index"
}
} else {
elasticsearch {
...
index => "bar-index"
}
}
}
Or, if the message type can go straight into the index name you can have a single output declaration:
elasticsearch {
...
index => "%{type}-index"
}
Related
Suppose I have some documents with join field, so they may have parent(e.g. [join_field][parent]) field in them, so as per docs, I need to pass that to _routing in logstash elasticsearch output,
...
output {
elasticsearch {
...
routing => "%{[join_field][parent]}"
}
}
Now if there is no join_field in the doc, above will set its routing to literally %{[join_field][parent]} in ES.
Is there anyway I can make it optional so the ES output will have routing set only if [join_field][parent] is there?
Or only way here is to have if else condition on the field and have separate output for each(but it feels odd to have multiple ifs for many options)? Also can this have any performance issue?
...
output {
if [join_field][parent] {
elasticsearch {
...
routing => "%{[join_field][parent]}"
}
} else {
elasticsearch {
...
}
}
}
My goal is to output filtered data into different indices. On my logstash.conf file, I currently have one input that takes in data from multiple logs files and then I filter the data as follows:
filter {
if ([source] ~= "examiner.log") {
json {
source => "message"
add_tag => ["EXAMINER"]
}
} else if ...
...
}
}
My output looks like:
output {
if ("EXAMINER" in [tags]) {
elasticsearch {
host => ["localhost:9200"]
index = "examiner"
}
} else {
elasticsearch {
host => ["localhost:9200"]
index => "data"
}
}
}
The data index gets created, but examiner never does. I'm not sure why the conditional does not seem to be working.
An example of my input data is:
{"#timestamp":"2019-09-27T20:42:12.254Z", "source_host":"WINL12345678", "file":"AppStarter.scala", "method":"start", "level":"INFO", "line_number":"34", "thread_name":"pool-1-thread-1", "#version":"1", "logger_name":"org.http4s.server.AppStarter", "message":"Application is running", "class":"org.http4s.server.AppStarter$class", "mdc":{}}
I want to add the elasticsearch index name as a field in the event when processing in Logstash. This is suppose to be pretty straight forward but the index name does not get printed out. Here is the complete Logstash config.
input {
elasticsearch {
hosts => "elasticsearch.example.com"
index => "*-logs"
}
}
filter {
mutate {
add_field => {
"log_source" => "%{[#metadata][_index]}"
}
}
}
output {
elasticsearch {
index => "logstash-%{+YYYY.MM}"
}
}
This will result in log_source being set to %{[#metadata][_index]} and not the actual name of the index. I have tried this with _id and without the underscores but it will always just output the reference and not the value.
Doing just %{[#metadata]} crashes Logstash with the error that it's trying to accessing the list incorrectly so [#metadata] is being set but it seems like index or any values are missing.
Does anyone have a another way of assigning the index name to the event?
I am using 5.0.1 of both Logstash and Elasticsearch.
You're almost there, you're simply missing the docinfo setting, which is false by default:
input {
elasticsearch {
hosts => "elasticsearch.example.com"
index => "*-logs"
docinfo => true
}
}
I'm relatively new to Logstash but have been successful up to this point. I'm parsing through a log and viewing the output in Kibana.
What I'd like to do is output only the data that I'm interested in. This includes data where the source = linux and the number = 78 or 80.
I'm trying to use the drop{} function to do this by trying to remove anything that does not meet these conditions. If the source is not equal to linux and the number is not 78 or 80, then drop it. Logic tells me this would only send what I want in the output but I'm not having any luck. It works great for one or the other (just filtering on the source or just on the numbers) but when I try to do both, it only takes the first condition. I've tried a few different ways: nested if statements, separate if statements, using !=, not in, etc.
Below is my code (notice the conditional in the filter):
input {
file {
path => "/home/user/logs/os_log.csv"
start_position => beginning
sincedb_path => "/dev/null"
}
}
filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}\,\"?%{NUMBER:number}\s*<%{NUMBER:carrotnumber}>%{SYSLOGTIMESTAMP:syslogtimestamp}\s*%{WORD:object}\s*%{USERNAME:source}\s*%{GREEDYDATA:event}" }
}
if [source] != "linux" and [number] not in ["78","80"] {
drop {}
}
}
output {
elasticsearch { host => localhost }
}
Is there a better way to do this? Thanks!
Feels like you meant:
if [source] != "linux" or [number] not in ["78","80"] {
drop {}
}
Using Logstash, I want to index documents into Elasticsearch and specify the type, id etc of the document that needs to be indexed. How can I specify those in my config without keeping useless fields in my documents?
Example: I want to specify the id used for insertion:
input {
stdin {
codec => json {}
}
}
output {
elasticsearch { document_id => "%{[id]}" }
}
This will insert the document in Elasticsearch with the id id but the document will keep a redundant field "id" in the mapping. How can I avoid that?
I thought of adding
filter{ mutate { remove_field => "%{[id]}"} }
in the config, but the field is removed and cannot consequently be used as document_id...
Right now this isn't possible. Logstash 1.5 introduces a #metadata field whose contents aren't included in what's eventually sent to the outputs, so you'd be able to create a [#metadata][id] field and refer to that in your output,
output {
elasticsearch { document_id => "%{[#metadata][id]}" }
}
without that field polluting the message payload indexed to Elasticsearch. See the #metadata documentation.