I create an index with a mapping which contains a field not_analyzed with command below and index a document with next command.
curl -XPUT localhost:9200/twitter -d '{
"settings": {
"number_of_shards": 5,
"number_of_replicas": 1
},
"mappings": {
"tweet" : {
"properties" : {
"message" : { "type" : "string",
"index": "not_analyzed"}
}
}
}
}'
curl -XPOST 'http://localhost:9200/twitter/tweet?' -d '{
"user" : "kimchy",
"postDate" : "2009-11-15T14:12:12",
"message" : "trying out Elasticsearch"
}
'
I checked to mappings with http://localhost:9200/twitter/_mapping?pretty=true and it outputs:
{
"twitter" : {
"mappings" : {
"tweet" : {
"properties" : {
"message" : {
"type" : "string",
"index" : "not_analyzed"
},
"post_date" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"user" : {
"type" : "string"
}
}
}
}
}
}
Finally when I search with this http://localhost:9200/twitter/tweet/_search?pretty=1&q=trying it finds the indexed document. Is it normal? I thought it should not find it unless I search the complete text "trying out Elasticsearch".
not_analyzed means that it's not doing tokenizing/other analysis to index the values, but it does still store the full value in Elasticsearch and it can be used as an exact match in a terms query. The field value is still getting included/analyzed into the _all field and indexed there so that it's searchable.
You need to set "include_in_all": false or "index": "no" to disable that.
See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html for more information.
Related
i am trying to process my logs with custom template svlogs but Inex is not getting created on the go based on my template .i am facing below error
"error"=>{"type"=>"index_not_found_exception", "reason"=>"no such index", "resource.type"=>"index_expression", "resource.id"=>"svlogs-2016.12.29", "index_uuid"=>"_na_", "index"=>"svlogs-2016.12.29"}
My output is
output {
elasticsearch {
hosts => [ "192.168.254.129:9200" ]
user => "logstash"
password => "selva123"
template_name => "svlogs"
index => "svlogs-%{+YYYY.MM.dd}"
}}
my template is :
curl -XPUT '192.168.254.129:9200/_template/svlogs?pretty' -d'
{
"template": "svlogs*",
"settings": {
"number_of_shards": 1
},
"mappings" : {
"_default_" : {
"properties" : {
"MSGID" : {"type": "integer" },
"debug" : {"type": "string" , "index" : "not_analyzed" },
"Error" : { "type" : "string", "index" : "not_analyzed" },
"client" : { "type" : "string" },
"eno" : { "type" : "integer" },
"login" : { "type" : "string" },
"message" : { "type" : "string" },
"pid" : { "type" : "integer" },
"process" : { "type" : "string" },
"sv_date" : { "type": "date", "format": "EEE MMM dd HH:mm:SS yyyy"},
"type" : { "type" : "string" }
}
}
}
}'
i was expecting logstash will create teh Index based on the temple given
Actually it was working till i installed x-pack . Theni have resolved all my privilege related issues , Now i need to create the index manually to make my logstash work . i tried added managae_temaple as "false" still no help .
Please guide Thanks in advance .
Issue got resolved after i commented below line in elasticsearch.yml
#action.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher-history*
My logstash config is something like the following
if "user" in [tags] {
elasticsearch {
hosts => ["localhost:9200"]
action => "index"
index => "user-%{+YYYY.MM.dd}"
template => '/path/to/elastic-template.json'
flush_size => 50
}
}
And the json template contains the lines
"fields" : {
"{name}" : {"type": "string", "index" : "analyzed", "omit_norms" : true, "index_options" : "docs"},
"{name}.raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
}
So I assume the .raw can be used when searching or generating the visualization.
However, I removed the existing index and rebuild again, I can see the data, but I still cannot find the .raw field either Kibana's settings, discover or visualize
How to use the .raw field?
The template you posted isn't even valid JSON. If you want to apply a raw field as in not_analyzed you have to do it like this:
"action" : {
"type" : "string",
"fields" : {
"raw" : {
"index" : "not_analyzed",
"type" : "string"
}
}
}
This will create a action.raw field.
I encountered same issue.
I used ES5.5.1 and logstash 5.5.1, below is my template file
{
"template": "access_log",
"settings": {
"index.refresh_interval" : "5s"
},
"mappings": {
"log": {
"properties":{
"geoip":{
"properties":{
"location" : {
"type" : "geo_point",
"index": "false"
}
}
}
}
}
}
}
I have a 'grade' field in an Elasticsearch index that contains text and numbers. I have set the field mapping to be 'not_analyized' but I can't search for grade ==== 'Year 1'.
I have read the finding exact values section of the docs but it doesn't seem to work for me.
Create the index.
curl -XPUT http://localhost:9200/my_test_index
Create the mapping template.
curl -XPUT http://localhost:9200/_template/my_test_index_mapping -d '
{
"template" : "my_test_index",
"mappings" : {
"my_type": {
"properties": {
"grade": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
'
Create some documents.
curl -XPUT 'http://localhost:9200/my_test_index/my_type/1' -d '{
"title" : "some title",
"grade" : "Year 1"
}'
curl -XPUT 'http://localhost:9200/my_test_index/my_type/3' -d '{
"title" : "some title",
"grade" : "preschool"
}'
Query for "Year 1" returns 0 results.
curl -XPOST http://localhost:9200/my_test_index/_search -d '{
"query": {
"filtered" : {
"filter" : {
"term": {
"grade": "Year 1"
}
}
}
}
}'
Query for 'preschool' returns 1 result.
curl -XPOST http://localhost:9200/my_test_index/_search -d '{
"query": {
"filtered" : {
"filter" : {
"term": {
"grade": "preschool"
}
}
}
}
}'
Checking the mapping and the 'grade' field does not show 'not_analyzed'.
curl -XGET http://localhost:9200/my_test_index/_mapping
{
"my_test_index" : {
"mappings" : {
"my_type" : {
"properties" : {
"grade" : {
"type" : "string"
},
"title" : {
"type" : "string"
}
}
}
}
}
}
The template will only impact newly created indices.
Re-Created the index after the template has been created.
Alternatively, specify the mappings while creating the index, instead of relying on templates to a single index.
If you don't want the field to be analysed you can specify "index" : "not_analyzed" in the mapping. You'll then be able to search for exact matches as desired.
See: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#string
In your case,Please try to Re-create your mapping.
In elasticsearch, each document can specify a _parent document, creating a one-to-many relationship. Can the same relationship be created with nested documents?
It does not appear so. Below are commands that create an index, with a mapping that has a _parent defined for a nested property. After executing these commands, the mapping is successfully created, but the _parent relationship is not. Am I doing something wrong or is this not possible?
curl -XPUT localhost:9200/my_index
curl -XPUT localhost:9200/my_index/_mapping/new_obj -d '{
"new_obj" : {
"properties" : {
"my_prop" : { "type" : "string"},
"my_nested_prop" : {
"type" : "nested",
"_parent" : { "type" : "new_obj"},
"properties" : {
"amount" : { "type" : "integer" },
"description" : { "type" : "string" }
}
}
}
}
}'
I would like to disable all the "raw" fields that are created in Elasticsearch by logstash-forwarder. So if I have a field as "host" logstash-forwarder won't create a "host.raw" field. But I need a general solution for all the string fields.
I have my string fields as "not_analyzed" so having raw fields has no point and just a duplicate of the data.
I tried to remove "fields" part of the mapping below but it's added back after the first log message. The closest thing I could achieve was to add the following mapping but that still creates empty raw fields:
curl -XPUT 'localhost:9200/myindex/' -d '{
"mappings": {
"_default_": {
"dynamic_templates" : [ {
"string_fields" : {
"mapping" : {
"index" : "not_analyzed",
"type" : "string",
"fields" : {
"raw" : {
"ignore_above" : 0,
"index" : "not_analyzed",
"type" : "string"
}
}
},
"match" : "*",
"match_mapping_type" : "string"
}
} ],
"_all": { "enabled": false }
}
}
}'
So how can I disable these fields?