Cannot turn Elasticsearch dynamic mapping on - elasticsearch

I disabled dynamic mapping with
curl -XPUT 'localhost:9200/_template/template_all?pretty' -H 'Content-Type: application/json' -d' { "template": "*", "order":0, "settings": { "index.mapper.dynamic": false }}'
I wanted to turn it back on with
curl -XPUT 'localhost:9200/_template/template_all?pretty' -H 'Content-Type: application/json' -d' { "template": "*", "order":0, "settings": { "index.mapper.dynamic": true }}'
It has confirmed it as true, but when I try to have logstash send information to it, in logstash error logs I get back-
"reason"=>"trying to auto create mapping, but dynamic mapping is disabled"
How do I actually turn dynamic mapping back on?


Looks like index for logstash was created with old template (before you update template). Because when you update you template only new indexes will have updated mapping and settings.
Check is index exists:
curl -XGET 'localhost:9200/LOGSTASH_INDEX_NAME_HERE'
If index exists and you can delete this index - do it. After this when logstash will try to send something - index will be created with the new mapping.

Related

Elasticsearch 7.7 how to read several documents, not fetch all the documents

I am using ElasticSearch 7.7 in CentOS 8 box. I could creat index, type by REST format by command curl. For example, I could use
curl -X PUT "localhost:9200/testindex2"
curl -H "Content-Type: application/json" -XPOST "http://localhost:9200/testindex2/man/1/" -d '{ "name" : "shiny2", "age": 28}'
curl -XGET "localhost:9200/testindex2/man/1/"
curl -XGET "localhost:9200/testindex2/man/_search?pretty"
But if I have inserted many documents, how could I do query by REST command line using command curl to find particular age = 28's documents?

curl -XGET "localhost:9200/testindex2/_search?pretty&q=age:28"
that is the simplest way to query.
more option and documentation:
https://www.elastic.co/guide/en/elasticsearch/reference/7.8/search-search.html
also you can use Match or Term query with JSON body format.
curl -XGET 'localhost:9200/testindex2/_search?pretty' -d '
{
"query": {
"term": {
"age": {
"value": "28"
}
}
}
}'
more documentation:
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-term-query.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query.html

Post JSON data to elastic search using curl command

I am trying to post Json file to elastic search and facing below errors
curl -XPOST http://localhost/test-index/doc -H "Content-Type: application/json" -d #test.json
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Rejecting mapping update to [test-index] as the final mapping would have more than 1 type: [_doc, doc]"}],"type":"illegal_argument_exception","reason":"Rejecting mapping update to [test-index] as the final mapping would have more than 1 type: [_doc, doc]"},"status":400}
test.json content
{
"name":"John Smith",
"age":"38"
}
am I missing anything

To post Json data to elasticsearch using curl command, you can try out this command:
curl -XPOST http://localhost:9200/test-index/_doc -H "Content-Type: application/json" -d #test.json
The command with which, you are trying to post Json file, works fine with Postman.

The error returned is
"reason":"Rejecting mapping update to [test-index] as the final
mapping would have more than 1 type: [_doc, doc]"}]
it mean that your index already have a type _doc and you try to create a new type doc. As version ~7.?? of elastic support only one type you can't create your data.
You need to add "_" before doc in your request.
http://localhost/test-index/doc <-- not correct
http://localhost/test-index/_doc <-- correct

Elasticsearch : How to disable automatic date detection globally for all indices

How to disable automatic detection for all indices globally in elasticsearch ? I have found that disabling for a single index is possible by dynamic mapping ( Source : https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html )
But I want to do it globally by some command in elasticsearch.yml. Is there any way to do this ?

I've resolved it by changing a global Elasticsearch template (check, if you allready have there any important setting in the global template, that you would like to keep, then you would also need to copy paste them here in json):
curl -X PUT "$HOSTNAME:9200/_template/global?pretty" -H 'Content-Type: application/json' -d'
{
"index_patterns" : ["logstash-*"], ####here your index pattern for the setting####
"order" : 0,
"mappings": {
"doc": {
"date_detection": false
}
}
}'

Search query to match all and return all data without using curl

I am working with postman and i want to try using getting all data from my index, now when using curl
curl -X GET "localhost:9200/_search" -H 'Content-Type: application/json' -d'
{
"query": {
"match_all": {}
}
}
but i want to write a search and return all data, my index name is tourdata and type is tours
127.0.0.1:9200/tourdata/tours/_search
how do i continue

So i found that i need to add size to my query parameter, This worked for me.
127.0.0.1:9200/tourdata/tours/_search?size=7000&pretty=true

How do I import a Kibana 6 visualization into elasticsearch 6 without using the Kibana UI?

I am trying to import a Kibana 6 visualization into Elasticsearch 6, to be viewed in Kibana. I am trying to do this with a curl command, or essentially a script without going through the Kibana UI. This is the command I’m using:
curl -XPUT http://localhost:9200/.kibana/doc/visualization:vis1 -H
'Content-Type: application/json' -d #visual1.json
And this is visual1.json:
{
"type": "visualization",
"visualization": {
"title": "Logins",
"visState": "{\"title\":\"Logins\",\"type\":\"histogram\",\"params\":{\"type\":\"histogram\",\"grid\":{\"categoryLines\":false,\"style\":{\"color\":\"#eee\"}},\"categoryAxes\":[{\"id\":\"CategoryAxis-1\",\"type\":\"category\",\"position\":\"bottom\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\"},\"labels\":{\"show\":true,\"truncate\":100},\"title\":{}}],\"valueAxes\":[{\"id\":\"ValueAxis-1\",\"name\":\"LeftAxis-1\",\"type\":\"value\",\"position\":\"left\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\",\"mode\":\"normal\"},\"labels\":{\"show\":true,\"rotate\":0,\"filter\":false,\"truncate\":100},\"title\":{\"text\":\"Count\"}}],\"seriesParams\":[{\"show\":\"true\",\"type\":\"histogram\",\"mode\":\"stacked\",\"data\":{\"label\":\"Count\",\"id\":\"1\"},\"valueAxis\":\"ValueAxis-1\",\"drawLinesBetweenPoints\":true,\"showCircles\":true}],\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"times\":[],\"addTimeMarker\":false},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"principal.keyword\",\"otherBucket\":true,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\"}}]}",
"uiStateJSON": "{}",
"description": "",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"index\":\"def097e0-550f-11e8-9266-93ce640e5839\”,\”filter\":[{\"meta\":{\"index\":\"def097e0-550f-11e8-9266-93ce640e5839\”,\”negate\":false,\"disabled\":false,\"alias\":null,\"type\":\"phrase\",\"key\":\"requestType.keyword\",\"value\":\"ALOG\”,\”params\":{\"query\":\"AUTH_LOGIN\",\"type\":\"phrase\"}},\"query\":{\"match\":{\"requestType.keyword\":{\"query\":\"AUTH_LOGIN\",\"type\":\"phrase\"}}},\"$state\":{\"store\":\"appState\"}}],\"query\":{\"query\":\"\",\"language\":\"lucene\"}}"
}
}
}
Now a couple things to note about the curl command and this json file. The index I push the visualization to is .kibana. I found that when I pushed these to other index’s such as “test”, my data would not show up as a stored object in Kibana, and thus wouldn’t show up on the visualization tab. When I PUT to .kibana with this syntax ‘.kibana/doc/visualization:vis1 ‘, my object shows up on the visualization tab.
Now concerning the json file. Note that when you export a visualization from Kibana 6, it doesn’t look like this. It looks like:
{
"_id": "vis1",
"_type": "visualization",
"_source": {
"title": "Logins",
"visState": "{\"title\":\"Logins\",\"type\":\"histogram\",\"params\":{\"type\":\"histogram\",\"grid\":{\"categoryLines\":false,\"style\":{\"color\":\"#eee\"}},\"categoryAxes\":[{\"id\":\"CategoryAxis-1\",\"type\":\"category\",\"position\":\"bottom\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\"},\"labels\":{\"show\":true,\"truncate\":100},\"title\":{}}],\"valueAxes\":[{\"id\":\"ValueAxis-1\",\"name\":\"LeftAxis-1\",\"type\":\"value\",\"position\":\"left\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\",\"mode\":\"normal\"},\"labels\":{\"show\":true,\"rotate\":0,\"filter\":false,\"truncate\":100},\"title\":{\"text\":\"Count\"}}],\"seriesParams\":[{\"show\":\"true\",\"type\":\"histogram\",\"mode\":\"stacked\",\"data\":{\"label\":\"Count\",\"id\":\"1\"},\"valueAxis\":\"ValueAxis-1\",\"drawLinesBetweenPoints\":true,\"showCircles\":true}],\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"times\":[],\"addTimeMarker\":false},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"principal.keyword\",\"otherBucket\":true,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\"}}]}",
"uiStateJSON": "{}",
"description": "",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"index\":\"def097e0-550f-11e8-9266-93ce640e5839\",\"filter\":[{\"meta\":{\"index\":\"def097e0-550f-11e8-9266-93ce640e5839\",\"negate\":false,\"disabled\":false,\"alias\":null,\"type\":\"phrase\",\"key\":\"requestType.keyword\",\"value\":\"LOG\",\"params\":{\"query\":\"LOG\",\"type\":\"phrase\"}},\"query\":{\"match\":{\"requestType.keyword\":{\"query\":\"LOG\",\"type\":\"phrase\"}}},\"$state\":{\"store\":\"appState\"}}],\"query\":{\"query\":\"\",\"language\":\"lucene\"}}"
}
}
}
Note the first few lines. I found from this link Unable to create visualization using curl command in elaticearch that you have to modify the json export in order to import it. Seems strange right?
Anyway, then I’ve had two errors on the actual visualization object once in Kibana. The first was that “The index pattern associated with this object no longer exists.” I was able to get around this by creating an index pattern with the id referenced in the searchSourceJson of my visualization. I had to do this within the Kibana UI, so technically this solution would not work for me. In any case, I created an index with a document in it by calling
curl -X PUT "localhost:9200/test57/_doc/1" -H 'Content-Type: application/json' -d'
{
"user" : "kimchy",
"post_date" : "2009-11-15T14:12:12",
"message" : "trying out Elasticsearch"
}
'
And then in the Kibana UI, created an index pattern and gave it the custom index pattern ID def097e0-550f-11e8-9266-93ce640e5839.
Now when I go try to view my visualization, I get a new error. “A field associated with this object no longer exists in the index pattern.”
I am guessing this has something to do with me pushing a random object into the index, but even with debug settings on for elastic and kibana, I don’t really get enough information to fix this problem.
If anyone could point me in the right direction that would be great! Thanks in advance.

You need to make sure that the fields you reference in your visualization definition are also present in the Kibana index pattern (Kibana main screen > Management > Index Patterns). The easiest way to do that would be to include said fields in the dummy index you created and then 'refresh field list' in the Kibana Index Patterns screen.
You can do this via CLI by creating a document of _type index-pattern in the .kibana index.

It is possible to import through kibana endpoint using api saved_objects.
This needs to modify the exported json wrapping it inside {"attributes":....}
Base on your example it should be something like:
curl -XPOST "http://localhost:5601/api/saved_objects/visualization/myvisualisation?overwrite=true" -H "kbn-xsrf: reporting" -H 'Content-Type: application/json' -d'
{"attributes":{
"title": "Logins",
"visState": "{\"title\":\"Logins\",\"type\":\"histogram\",\"params\":{\"type\":\"histogram\",\"grid\":{\"categoryLines\":false,\"style\":{\"color\":\"#eee\"}},\"categoryAxes\":[{\"id\":\"CategoryAxis-1\",\"type\":\"category\",\"position\":\"bottom\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\"},\"labels\":{\"show\":true,\"truncate\":100},\"title\":{}}],\"valueAxes\":[{\"id\":\"ValueAxis-1\",\"name\":\"LeftAxis-1\",\"type\":\"value\",\"position\":\"left\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\",\"mode\":\"normal\"},\"labels\":{\"show\":true,\"rotate\":0,\"filter\":false,\"truncate\":100},\"title\":{\"text\":\"Count\"}}],\"seriesParams\":[{\"show\":\"true\",\"type\":\"histogram\",\"mode\":\"stacked\",\"data\":{\"label\":\"Count\",\"id\":\"1\"},\"valueAxis\":\"ValueAxis-1\",\"drawLinesBetweenPoints\":true,\"showCircles\":true}],\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"times\":[],\"addTimeMarker\":false},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"principal.keyword\",\"otherBucket\":true,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\"}}]}",
"uiStateJSON": "{}",
"description": "",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"index\":\"def097e0-550f-11e8-9266-93ce640e5839\",\"filter\":[{\"meta\":{\"index\":\"def097e0-550f-11e8-9266-93ce640e5839\",\"negate\":false,\"disabled\":false,\"alias\":null,\"type\":\"phrase\",\"key\":\"requestType.keyword\",\"value\":\"LOG\",\"params\":{\"query\":\"LOG\",\"type\":\"phrase\"}},\"query\":{\"match\":{\"requestType.keyword\":{\"query\":\"LOG\",\"type\":\"phrase\"}}},\"$state\":{\"store\":\"appState\"}}],\"query\":{\"query\":\"\",\"language\":\"lucene\"}}"
}
}
}
'

Resources