Logstash pipeline not showing on Kibana, but logs show Pipelines running - elasticsearch

Question

Trying to set up elastic search, kibana and logstash to read logs from local folder.
It works well on version 7.x.x, but when I try to upgrade to 8 it doesn't.Fx
I am using this YAML file:
version: '3.6'
services:
Elasticsearch:
image: elasticsearch:8.4.0
container_name: elasticsearch
volumes:
- elastic_data:/usr/share/elasticsearch/data/
environment:
- discovery.type=single-node
- xpack.license.self_generated.type=basic
- xpack.security.enabled=false
ports:
- '9200:9200'
- '9300:9300'
networks:
- elk
Logstash:
image: logstash:8.4.0
container_name: logstash
environment:
- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
- xpack.monitoring.enabled=true
volumes:
- ./logstash/:/logstash
- D:/test/Logs/:/test/Logs
command: logstash -f /logstash/logstash.conf
depends_on:
- Elasticsearch
ports:
- '9600:9600'
networks:
- elk
Kibana:
image: kibana:8.4.0
container_name: kibana
ports:
- '5601:5601'
environment:
- ELASTICSEARCH_URL=http://elasticsearch:9200
depends_on:
- Elasticsearch
networks:
- elk
volumes:
elastic_data: {}
networks:
elk:
and config for logstash:
input {
file {
path => "/test/Logs/test.slog"
start_position => "beginning"
}
}
output {
elasticsearch {
hosts => ["elasticsearch:9200"]
}
}
test.slog exist and contain logs.
the logstash docker show the following logs:
[2022-08-27T20:40:32,592][INFO ][logstash.outputs.elasticsearch][main] Installing Elasticsearch template {:name=>"ecs-logstash"}
[2022-08-27T20:40:33,450][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.95}
[2022-08-27T20:40:33,451][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline Java execution initialization time {"seconds"=>0.94}
[2022-08-27T20:40:33,516][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2022-08-27T20:40:33,532][INFO ][logstash.inputs.file ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/usr/share/logstash/data/plugins/inputs/file/.sincedb_327fd1919fa26d08ec354604c3e1a1ce", :path=>["/test/Logs/test.slog"]}
[2022-08-27T20:40:33,559][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-08-27T20:40:33,614][INFO ][filewatch.observingtail ][main][8992bf4e2fad9d8838262d3019319d02ab5ffdcb5b282e821574485618753ce9] START, creating Discoverer, Watch with file and sincedb collections
[2022-08-27T20:40:33,625][INFO ][logstash.agent ] Pipelines running {:count=>2, :running_pipelines=>[:".monitoring-logstash", :main], :non_running_pipelines=>[]}
But when I go to the Data -> Index Management there is nothing. and also in the Ingest pipeline.
What am I doing wrong?

Answer 1

In Elasticsearch 8 the index names created by logstash output follow the pattern .ds-logs-generic-default-%{+yyyy.MM.dd} instead of logstash-%{+yyyy.MM.dd}
This .ds index does not appear under Data -> Index Management but the documents can be queried
You can view the .ds-logs-generic index in Kibana, Management> Dev Tools using
GET _cat/indices
To query the documents you can use the _search API
GET /.ds-logs-generic-default-2022.08.28-000001/_search
{
"query": {
"match_all": {}
}
}
If you want to specify the index name you can add it to the output section of your logstash.conf eg index => "logstash-%{+YYYY.MM.dd}"
output {
elasticsearch {
hosts => ["elasticsearch:9200"]
index => "logstash-%{+YYYY.MM.dd}"
}
}
The newly created index will show in Kibana under Management > Data > Index Management. You may need to add a few log lines at the end of your logfile to kick the indexing pipeline.