I have a Spring boot application, that produces logs into a file.
I also have running Elastic search (in docker) and Kibana and Logstash (not in docker).
This is my Logstash config:
input {
file {
type => "java"
path => "C:\Users\user\Documents\logs\semblogs.log"
start_position => "beginning"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
}
stdout {
codec => rubydebug
}
}
Elastic is up and running. When I check for the data in index that was created like this:
http://localhost:9200/logstash-2019.11.04-000001/_search
it shows:
took 0
timed_out false
_shards
total 1
successful 1
skipped 0
failed 0
hits
total
value 0
relation "eq"
max_score null
hits []
In Kibana I also can't create an index, it says there are no data in elastic.
I suspect that Logstash is not sending incoming anything to Elastic, but I don't know why. There ARE logs in the log file from the app...
Related
I am working with Filebeat and Logstash to upload logs to Elastic (all are 7.3-oss version).
My log file contain billions of rows, yet elastic only show 10K documents.
When adding stdout output it seems like all the data is coming to Logstash, but for some reason Logstash uploads only 10,000 docs.
I added another output
stdout { codec => rubydebug }
for printing to the screen it seems like the data is coming from Filebeat, but for some reason Logstash only upload 10,000 docs.
Also tried removing the Json Filter in Logstash, but the issue still occur.
Filebeat config
filebeat.inputs:
- type: log
paths:
\\some-path\my.json
output.logstash:
hosts: ["localhost:5044"]
Logstash pipeline
input {
beats {
port => 5044
}
}
filter{
json{
source => "message"
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => [ "machine-name:9200" ]
}
}
Logstash.yml
is empty as the default installation
I found that is was my search that caused the confusion.
According to
https://www.elastic.co/guide/en/elasticsearch/reference/7.3/search-request-body.html#request-body-search-track-total-hits,
Elastic simply didn't return the accurate hits (just stated that its greater than 10000).
Changing my search query
GET logstash-*/_search
{
"track_total_hits": true
}
returned the right size.
How do I avoid elasticsearch duplicate documents?
The elasticsearch index docs count (20,010,253) doesn’t match with logs line count (13,411,790).
documentation:
File input plugin.
File rotation is detected and handled by this input,
regardless of whether the file is rotated via a rename or a copy operation.
nifi:
real time nifi pipeline copies logs from nifi server to elk server.
nifi has rolling log files.
logs line count on elk server:
wc -l /mnt/elk/logstash/data/from/nifi/dev/logs/nifi/*.log
13,411,790 total
elasticsearch index docs count:
curl -XGET 'ip:9200/_cat/indices?v&pretty'
docs.count = 20,010,253
logstash input conf file:
cat /mnt/elk/logstash/input_conf_files/test_4.conf
input {
file {
path => "/mnt/elk/logstash/data/from/nifi/dev/logs/nifi/*.log"
type => "test_4"
sincedb_path => "/mnt/elk/logstash/scripts/sincedb/test_4"
}
}
filter {
if [type] == "test_4" {
grok {
match => {
"message" => "%{DATE:date} %{TIME:time} %{WORD:EventType} %{GREEDYDATA:EventText}"
}
}
}
}
output {
if [type] == "test_4" {
elasticsearch {
hosts => "ip:9200"
index => "test_4"
}
}
else {
stdout {
codec => rubydebug
}
}
}
You can use fingerprint filter plugin: https://www.elastic.co/guide/en/logstash/current/plugins-filters-fingerprint.html
This can e.g. be used to create consistent document ids when inserting
events into Elasticsearch, allowing events in Logstash to cause
existing documents to be updated rather than new documents to be
created.
I've setup a simple elastic stack like so:
LS shipper => Kafka => LS Indexer => ES => Kibana
I'm using all the latest versions. (5.2.2-1)
My indices are not being created on Elasticsearch so I've checked at every level.
I can see my logs coming all the way to the LS indexer.
[2017-03-14T16:08:01,360][DEBUG][logstash.pipeline ] output received {"event"=>{"#timestamp"=>2017-03-14T15:08:01.355Z, "#version"=>"1", "message"=>"{\"severity\":6,\"timestamp8601\":\"2017-03-14T16:08:01+01:00\",\"pid\":\"65156\",\"program\":\"CROND\",\"message\":\"(root) CMD (/home/unix/cron/iodisk >/dev/null 2>&1)||syslog source origin:not defined or not authorized|syslog source name:not defined or not authorized|syslog source env:not defined or not authorized|syslog source security level:0|syslog time received:2017-03-14T16:08:01.349084+01:00|syslog time reported:2017-03-14T16:08:01+01:00||\\n\",\"priority\":78,\"logsource\":\"VRHNDCPUPAPPPR1\",\"type\":\"system\",\"#timestamp\":\"2017-03-14T15:08:01.000Z\",\"#version\":\"1\",\"host\":\"10.64.1.202\",\"facility\":9,\"severity_label\":\"Informational\",\"source_indexer\":\"tcp.50050\",\"timestamp\":\"2017-03-14T16:08:01+01:00\",\"facility_label\":\"clock\"}"}}
Here is my indexer config file:
input {
kafka {
bootstrap_servers => "10.64.2.143:9092"
group_id => "logstash indexer"
topics => "system"
}
}
output {
if [type == "system"] {
elasticsearch {
codec => json
hosts => [ "10.64.2.144:9200" ]
index => "system"
}
}
}
Of course, i can't find any index named system in kibana:
Kibana index pattern configuration
No index created
I'm available for more info if someone is ready to help.
Thanks,
I suspect your conditional is wrong
if [type == "system"] {
I suspect that should be:
if [type] == "system" {
That will probably work better.
I am using a Logstash server 1-> Kafka -> Logstash server 2-> Elasticsearch -> Kibana setup. Below is the configuration files from Logstash server 2 .
1) 03-logstash-logs-kafka-consumer.conf
input {
kafka {
zk_connect => 'zk_netaddress:2181'
topic_id => 'logstash_logs'
codec => "json"
}
}
output{
stdout{}
}
2) 30-elasticsearch-output.conf
output {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "%{[#metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[#metadata][type]}"
}
}
Though Logs are travelling from Logstash server 1 to Logstash server 2 through Kafka and Logstash server 2 can also output to the /var/log/logstash/logstash.stdout file, Logstash server 2 is not able to output to the elasticsearch configured with it. I have checked all services, they are running well and there are no Exception in the logs of all the services.
Please post your suggestions.
I'm trying to reindex the data by having elasticsearch as input and sending back to elasticsearch as output. The script is running fine but the indexing is going indefinitely. The script is as below
input {
elasticsearch {
host => "10.0.0.11"
index => "logstash-2015.02.05"
}
}
output {
elasticsearch {
host => "10.0.0.11"
protocol => "http"
cluster => "logstash"
node_name => "logindexer"
index => "logstash-2015.02.05_new"
}
}
This means if I have 200 docs under logstash-2015.02.05 index then it creates duplicate records in logstash-2015.02.05_new and keeps going until I stop the logstash agent. Is there a way to just restrict the documents in new index to have exactly the same as the old index? Pls help.