I just altered my logstash-elasticearch setup to include rabbitmq rather since I wasn't able to get messages into logstash fast enough with tcp connection. Now it is blazing fast as logstash reads from the queue but I do not see the messages coming through into kibana. One error shows the timestamp field missing. I used the plugin/head to view the data and it is odd:
_index _type _id ▼_score #version #timestamp
pt-index logs Bv4Kp7tbSuy8YyNi7NEEdg 1 1 2014-03-27T12:37:29.641Z
this is what my conf file looks like now and below what it did look like:
input {
rabbitmq {
queue => "logstash_queueII"
host => "xxx.xxx.x.xxx"
exchange => "logstash.dataII"
vhost => "/myhost"
}
}
output {
elasticsearch{
host => "xxx.xxx.xx.xxx"
index => "pt-index"
codec => "json_lines"
}
}
this is what it was before rabbitmq:
input {
tcp {
codec => "json_lines"
port => "1516"
}
}
output {
elasticsearch {
embedded => "true"
}
}
Now the only change I made was to create a specific index in elasticsearch and have the data indexed there but now it seems the format of the message has changed. It is still json messages with 2/3 fields but not sure what logstash is reading or changing from rabbitmq. I can see data flowing into the histogram but the fields are gone.
"2014-03-18T14:32:02" "2014-03-18T14:36:24" "166" "google"
these are the fields I would expect. Like I said all this worked before I made the change.
I have seen examples of a similar configurations, but they do not use the output codec of "json_lines" going into Elasticsearch. The output codec would adjust the formatting of the data as it leaves logstash which I do not believe is nessisary. Try deleting the codec and see what logstash is outputting by adding a file output to a log, be sure this is only short sample...
Related
Is it possible to send logs from different remote machines to elasticsearch using just logstash(no filebeats)? Is so, do I define same index in all the conf.d file in all the machines? I want all the logs to be in the same index.
Would i use logs-%{+YYYY.MM.dd} for the index of all config files to have them indexed into the same folder?
input {
file {
part => /home/ubuntu/logs/data.log
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index =>"logs-%{+YYYY.MM.dd}"
}
}
What you do is ok and it will work. Just one thing I would correct is that you should simply write to a data stream and not have to care about the index name and ILM matters (rollover, retention, etc), like this:
input {
file {
part => /home/ubuntu/logs/data.log
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
data_stream => "true"
data_stream_type => "logs"
data_stream_dataset => "ubuntu"
data_stream_namespace => "prod"
}
}
The data stream name will be logs-ubuntu-prod, you can change the latter two to your liking.
Make sure to properly set up your data stream first, with an adequate Index Lifecycle Management policy, though.
On a different note, it's a waste of resource to install Logstash on all your remote machines which is supposed to work as centralized streaming engine. You should definitely either use Filebeat, or even better now the Elastic Agent which is fully manageable through Fleet in Kibana. You should have a look.
I am working with Filebeat and Logstash to upload logs to Elastic (all are 7.3-oss version).
My log file contain billions of rows, yet elastic only show 10K documents.
When adding stdout output it seems like all the data is coming to Logstash, but for some reason Logstash uploads only 10,000 docs.
I added another output
stdout { codec => rubydebug }
for printing to the screen it seems like the data is coming from Filebeat, but for some reason Logstash only upload 10,000 docs.
Also tried removing the Json Filter in Logstash, but the issue still occur.
Filebeat config
filebeat.inputs:
- type: log
paths:
\\some-path\my.json
output.logstash:
hosts: ["localhost:5044"]
Logstash pipeline
input {
beats {
port => 5044
}
}
filter{
json{
source => "message"
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => [ "machine-name:9200" ]
}
}
Logstash.yml
is empty as the default installation
I found that is was my search that caused the confusion.
According to
https://www.elastic.co/guide/en/elasticsearch/reference/7.3/search-request-body.html#request-body-search-track-total-hits,
Elastic simply didn't return the accurate hits (just stated that its greater than 10000).
Changing my search query
GET logstash-*/_search
{
"track_total_hits": true
}
returned the right size.
I need my logstash conf file to send a message to a kafka topic to indicate that the document processed has been sent to elasticsearch. I have my logstash file ready to structure the data to send to the ElasticSearch but I need to post 'yes' or 'no' message to a kafka topic through the same logstash file.
You can use mutiple outputs like
output
{
#output to console
stdout {
codec => rubydebug
}
#output to elasticsearch
elasticsearch {
hosts => [ "192.168.1.245:9201" ]
}
#output to kafka
kafka {
codec => json
topic_id => "mytopic"
}
}
First you need to have the yes/no value in a field, let's call it value.
Then add a kafka output, with the plain codec using the format option to add the yes/no value:
output {
#rest of your output configuration
kafka {
...
codec => plain {format => "%{[value]}"}
}
}
I am trying to send some raw data to elasticsearch through logstash. I am trying to do this through the udp plugin but for now I dont think this is relevant.
Basically, I with to send key/value pairs, and I wish for this to show up as:
{
"key_1": "value_1"
....
}
instead of:
{
"message": "{\"key1\": \"value1\"}"
}
Is there any way for logstash to somehow "decode" the message as json and insert them as top level keys?
Thanks
I just needed to use a "json" codec on the input like so:
input {
udp {
port => 3425
codec => "json"
}
}
Thanks to Val for pointing this out
I am trying to have elasticsearch index based on field so I can get an index for each source (allowing for secure access to each index).
I tried something along the lines of
output {
stdout { codec => rubydebug }
elasticsearch {
index => [SERVER]"-%{+YYYY.MM.dd}"
}
}
as well as
output {
stdout { codec => rubydebug }
elasticsearch{
index => "[SERVER]-%{+YYYY.MM.dd}"
}
}
and neither work : first errors, second tries to create the index with [SERVER] in it then errors due to uppercase, this might not be supported as I can't find it anywhere in the docs, but I was wondering if anyone has gotten something like this functional for their own ELK stacks?
The right syntax for this is "%{SERVER}-%{+YYYY.MM.dd}"
According to the documentation :
[The index to write] can be dynamic using the %{foo} syntax.