I had a functional Logstash and Elasticsearch on version 5.1.
I deleted all indices, then upgraded to 6.1.
Now, when Logstash receives some event from Filebeat (Which stills version 5.1), it throws this error:
[2017-12-27T17:29:16,463][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch.
{
:status => 400,
:action => ["index", {:_id=>nil, :_index=>"logstash-2017.12.27", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x34de85bd>],
:response => {
"index" => {
"_index" => "logstash-2017.12.27",
"_type" => "doc",
"_id" => nil,
"status" => 400,
"error" => {
"type" => "mapper_parsing_exception",
"reason" => "Failed to parse mapping [_default_]: [include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields to create your own catch all field.",
"caused_by" => {
"type" => "mapper_parsing_exception",
"reason" => "[include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields to create your own catch all field."
}
}
}
}
}
I have even tried using an extremely simplistic pipeline, as you can see here:
input {
beats {
port => 5044
}
}
filter {
json {
source => "message"
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
}
Yet it throws this error over and over.
Any idea what can be wrong here?
This answer is to just expand on what #alexanderlz said. From the DevTools page in kibana I ran this:
GET /_template/
That lists all templates
here is the template we need to delete / modify (in part):
"logstash": {
"order": 0,
"version": 60001,
"index_patterns": [
"logstash-*"
],
So then run
DELETE /_template/logstash
once that is done restart logstash and it will reinstall a new, correct, template.
take a look at changes in mapping, introduced in elasticsearch 6.0
you need to remove the include_in_all mapping parameter from your index template.
can you paste here your template/mapping?
Related
I am trying to insert data from a .csv-File to an already existing index (that already has data) using Logstash.
Anyway this is my logstash_true.config File:
input {
file {
path => "pathToFile"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
columns => ["title", "text", "subject", "date"]
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => "127.0.0.1:9200"
index => "news"
document_type => "true_news"
document_id => "%{id}"
}
}
When uploading the data, I can see in the command line that there is nothing wrong with the file or the Data and the document_type true_news actually exist.
But when trying the get the data:
{
"count" : 0,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
}
}
The data wasn't loaded.
UPDATE
when enabling debugging i get the following error:
Could not index event to Elasticsearch. {:status=>400, :action=>
["index", {:_id=>"%{id}", :_index=>"news", :routing=>nil,
:_type=>"true_news"}, #<LogStash::Event:0x7e10d60f>], :response=>
{"index"=>{"_index"=>"news", "_type"=>"true_news", "_id"=>"%{id}",
"status"=>400, "error"=>{"type"=>"illegal_argument_exception",
"reason"=>"Rejecting mapping update to [news] as the final mapping
would have more than 1 type: [fake_news, true_news]"}}}}
Since Elasticsearch version 6.0 you can't have multiple types in your index.
It seems that your index news already have documents or mapping with the type fake_news and you are trying to insert documents with the type true_news, this is not possible, that's why you are getting this error:
"type"=>"illegal_argument_exception",
"reason"=>"Rejecting mapping update to [news] as the final mapping
would have more than 1 type: [fake_news, true_news]"
Since you can have only 1 type and you want to be able to distinguish between true_news and fake_news, it is better to recreate your index to use the default type, doc, for every document, and add a tag with true_news or fake_news to your documents using the config add_tag => ["tag"] in your inpus.
I have a bulk upload for a new index that I'm sending to my ES cluster from logstash. As such I want replication and refreshing turned off until the load is done, and I'll re-enable those values after the upload is complete.
I have a config file that looks like the following
input {
stdin { type => stdin }
}
filter {
csv {
separator => " "
columns => [ ...]
}
}
output {
amazon_es {
hosts =>
["my-domain.us-east-1.es.amazonaws.com"]
index => "my-index"
template => "conf/my-index-template.json"
template_name => "my-index-template-name"
region => "us-east-1"
}
}
And the template file looks like
{
"template" : "my-index-template-name",
"mappings" : {
...
},
"settings" : {
"index" : {
"number_of_shards" : "48",
"number_of_replicas" : "0",
"refresh_interval": "-1"
}
}
}
And when I run logstash and go to look at the settings for that index, the mappings are all respected from this template which is good, but everything in the settings section is ignored and it takes on default values (i.e. number_of_shards=5, and number_of_replicas=1)
Some investigation notes:
If I get the template after it's installed from ES itself I see the proper values in the template (for both mappings and settings). They just don't seem to be applying to the index
Also if I take the contents of the template file and create the index manually w/ a PUT it shows up as I would expect
My logstash version is 7.3.0 and my elasticsearch version is 6.7
Not sure what I'm doing wrong here
Your index name is my-index, but the template setting in your mapping uses my-index-template-name, it needs to be a regular expression or the same name as your index.
Since you are using elasticsearch 6.7 you should use index_patterns instead of template in your mapping.
{
"index_patterns" : ["my-index"],
"mappings" : {
...
},
"settings" : {
"index" : {
"number_of_shards" : "48",
"number_of_replicas" : "0",
"refresh_interval": "-1"
}
}
}
I am new to running the ELK stack. I have Logstash configured to feed my webapp log into Elasticsearch. I am trying to set up a visualization in Kibana that will show the count of unique users, given by the user_email field, which is parsed out of certain log lines.
I am fairly sure that I want to use the Unique Count aggregation, but I can't seem to get Kibana to include user_email in the list of fields which I can aggregate.
Here is my Logstash configuration:
filter {
if [type] == "wl-proxy-log" {
grok {
match => {
"message" => [
"(?<syslog_datetime>%{SYSLOGTIMESTAMP}\s+%{YEAR})\s+<%{INT:session_id}>\s+%{DATA:log_message}\s+license=%{WORD:license}\&user=(?<user_email>%{USERNAME}\#%{URIHOST})\&files=%{WORD:files}",
]
}
break_on_match => true
}
date {
match => [ "syslog_datetime", "MMM dd HH:mm:ss yyyy", "MMM d HH:mm:ss yyyy" ]
target => "#timestamp"
locale => "en_US"
timezone => "America/Los_Angeles"
}
kv {
source => "uri_params"
field_split => "&?"
}
}
}
output {
elasticsearch {
ssl => false
index => "wl-proxy"
manage_template => false
}
}
Here is the relevant mapping in Elasticsearch:
{
"wl-proxy" : {
"mappings" : {
"wl-proxy-log" : {
"user_email" : {
"full_name" : "user_email",
"mapping" : {
"user_email" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
}
}
}
Can anyone tell me what I am missing?
BTW, I am running CentOS with the following versions:
Elasticsearch Version: 6.0.0, Build: 8f0685b/2017-11-10T18:41:22.859Z, JVM: 1.8.0_151
Logstash v.6.0.0
Kibana v.6.0.0
Thanks!
I figured it out. The configuration was correct, AFAICT. The issue was that I simply hadn't refreshed the list of fields in the index in the Kibana UI.
Management -> Index Patterns -> Refresh Field List (the refresh icon)
After doing that, the field began appearing in the list of aggregatable terms, and I was able to create the necessary visualizations.
I have a working ELK stack and would like to enable index compression.
The official store compression documentation tells me that I need to do it at index creation.
I couldn't find anything related to store compression or even index settings in the related logstash output documentation
Below is my logstash output configuration:
output {
elasticsearch {
hosts => [ "localhost:9200" ]
sniffing => true
manage_template => false
index => "%{[#metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[#metadata][type]}"
}
}
And the created index settings:
{
"filebeat-2016.04.28": {
"settings": {
"index": {
"creation_date": "1461915752875",
"uuid": "co8bvXI7RFKFwB7oJqs8cA",
"number_of_replicas": "1",
"number_of_shards": "5",
"version": {
"created": "2030199"
}
}
}
}
}
You need to provide your own index template file in order to enable index compression.
So you need to create your filebeat-template.json file like this. This file will be used by logstash when creating a new filebeat index.
{
"template" : "filebeat-*",
"settings" : {
"index.codec" : "best_compression"
}
}
Then your elasticsearch output should be modified like this:
output {
elasticsearch {
hosts => [ "localhost:9200" ]
sniffing => true
index => "%{[#metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[#metadata][type]}"
template_name => "filebeat-template"
template => "/path/to/filebeat-template.json"
}
}
Then you can delete your existing filebeat-2016.04.28 index and relaunch logstash. The latter will create an index template called /_template/filebeat-template which will kick in everytime ES needs to create a new index whose name starts with filebeat- and it will apply the settings (among which the store compression one) present in the template.
I am using logstash to feed logs into ElasticSearch.
I am configuring logstash output as:
input {
file {
path => "/tmp/foo.log"
codec =>
plain {
format => "%{message}"
}
}
}
output {
elasticsearch {
#host => localhost
codec => json {}
manage_template => false
index => "4glogs"
}
}
I notice that as soon as I start logstash it creates a mapping ( logs ) in ES as below.
{
"4glogs": {
"mappings": {
"logs": {
"properties": {
"#timestamp": {
"type": "date",
"format": "dateOptionalTime"
},
"#version": {
"type": "string"
},
"message": {
"type": "string"
}
}
}
}
}
}
How can I prevent logstash from creating this mapping ?
UPDATE:
I have now resolved this error too. "object mapping for [logs] tried to parse as object, but got EOF, has a concrete value been provided to it?"
As John Petrone has stated below, once you define a mapping, you have to ensure that your documents conform to the mapping. In my case, I had defined a mapping of "type: nested" but the output from logstash was a string.
So I removed all codecs ( whether json or plain ) from my logstash config and that allowed the json document to pass through without changes.
Here is my new logstash config ( with some additional filters for multiline logs ).
input {
kafka {
zk_connect => "localhost:2181"
group_id => "logstash_group"
topic_id => "platform-logger"
reset_beginning => false
consumer_threads => 1
queue_size => 2000
consumer_id => "logstash-1"
fetch_message_max_bytes => 1048576
}
file {
path => "/tmp/foo.log"
}
}
filter {
multiline {
pattern => "^\s"
what => "previous"
}
multiline {
pattern => "[0-9]+$"
what => "previous"
}
multiline {
pattern => "^$"
what => "previous"
}
mutate{
remove_field => ["kafka"]
remove_field => ["#version"]
remove_field => ["#timestamp"]
remove_tag => ["multiline"]
}
}
output {
elasticsearch {
manage_template => false
index => "4glogs"
}
}
You will need a mapping to store data in Elasticsearch and to search on it - that's how ES knows how to index and search those content types. You can either let logstash create it dynamically or you can prevent it from doing so and instead create it manually.
Keep in mind you cannot change existing mappings (although you can add to them). So first off you will need to delete the existing index. You would then modify your settings to prevent dynamic mapping creation. At the same time you will want to create your own mapping.
For example, this will create the mappings for the logstash data but also restrict any dynamic mapping creation via "strict":
$ curl -XPUT 'http://localhost:9200/4glogs/logs/_mapping' -d '
{
"logs" : {
"dynamic": "strict",
"properties" : {
"#timestamp": {
"type": "date",
"format": "dateOptionalTime"
},
"#version": {
"type": "string"
},
"message": {
"type": "string"
}
}
}
}
'
Keep in mind that the index name "4glogs" and the type "logs" need to match what is coming from logstash.
For my production systems I generally prefer to turn off dynamic mapping as it avoids accidental mapping creation.
The following links should be useful if you want to make adjustments to your dynamic mappings:
https://www.elastic.co/guide/en/elasticsearch/guide/current/dynamic-mapping.html
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/custom-dynamic-mapping.html
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/dynamic-mapping.html
logs in this case is the index_type. If you don't want to create it as logs, specify some other index_type on your elasticsearch element. Every record in elasticsearch is required to have an index and a type. Logstash defaults to logs if you haven't specified it.
There's always an implicit mapping created when you insert records into Elasticsearch, so you can't prevent it from being created. You can create the mapping yourself before you insert anything (via say a template mapping).
The setting manage_template of false just prevents it from creating the template mapping for the index you've specified. You can delete the existing template if it's already been created by using something like curl -XDELETE http://localhost:9200/_template/logstash?pretty
Index templates can help you. Please see this jira for more details. You can create index templates with wildcard support to match an index name and put your default mappings.