I have been unable to show words occurrency in kibana inside a full_text field mapped as "type": "keyword" across documents in the index.
My first attempt involved the usage of an analyzer. However I have been unable to change the document in any way, the index mapping relfect the analyzer but no field reflect the analysis.
This is the simplified mapping:
{
"mappings": {
"doc": {
"properties": {
"text": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
},
"analyzed": {
"type": "text",
"analyzer": "rebuilt"
}
}
}
}
}
},
"settings": {
"analysis": {
"analyzer": {
"rebuilt": {
"tokenizer": "standard"
}
}
},
"index.mapping.ignore_malformed": true,
"index.mapping.total_fields.limit": 2000
}
}
but still I'm unable to see the array of words that I expect to be saved under the text.analyzed field, indeed that fields does not exists and I'm wondering why
It seems like settings fielddata=true link, in spite of being heavily discouraged, solved my problem (at least for now), and allows me to visualize in kibana the occurrence (or absolute frequency) of each word in the text field across documents.
The final version of the proposed simplified mapping therefore became:
{
"mappings": {
"doc": {
"properties": {
"text": {
"type": "text",
"analyzer": "rebuilt",
"fielddata": true
"fields": {
"keyword": {
"type": "keyword"
}
}
}
}
}
},
"settings": {
"analysis": {
"analyzer": {
"rebuilt": {
"tokenizer": "standard"
}
}
},
"index.mapping.ignore_malformed": true,
"index.mapping.total_fields.limit": 2000
}
}
Getting rid of the useless analyzed field.
I still have to check the performance of kibana. If someone has a performance safe solution to this problem please do not hesitate.
Thanks.
Related
Can somebody explain me please what is the difference between settings fielddata and fields while mapping in Elasticsearch?
For example what is the difference between this two codes:
PUT my_index
{
"mappings": {
"properties": {
"my_field": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword" // for ordering
}
}
}
}
}
}
and
PUT my_index/_mapping
{
"properties": {
"my_field": {
"type": "text",
"fielddata": true // what is the difference?
}
}
}
Or can you tell me if this code does make any sence?
PUT my_index
{
"mappings": {
"properties": {
"my_field": {
"type": "text",
"fielddata": true,
"fields": {
"keyword": {
"type": "keyword" // for ordering
}
}
}
}
}
}
Since the main intent is to do sorting and aggregations, then definitely use the first option, i.e. the keyword (sub-)field.
fielddata is the old-fashioned way of doing it and eats up a lot more memory.
You can find more detailed information and a link to a related article here
Ive been around the houses with this for the past few days trying things in various orders but cant figure out why its not working.
I am trying to create an index in Elasticsearch with an analyzer which is the same as the "standard" analyzer but retains upper case characters when records are stored.
I create my analyzer and index as follows:
PUT /upper
{
"settings": {
"index" : {
"analysis" : {
"analyzer": {
"rebuilt_standard": {
"tokenizer": "standard",
"filter": [
"standard"
]
}
}
}
}
},
"mappings": {
"doc": {
"properties": {
"title": {
"type": "text",
"analyzer": "rebuilt_standard"
}
}
}
}
}
Then add two records to test like this...
POST /upper/doc
{
"text" : "TEST"
}
Add a second record...
POST /upper/doc
{
"text" : "test"
}
Using /upper/_settings gives the following:
{
"upper": {
"settings": {
"index": {
"number_of_shards": "5",
"provided_name": "upper",
"creation_date": "1537788581060",
"analysis": {
"analyzer": {
"rebuilt_standard": {
"filter": [
"standard"
],
"tokenizer": "standard"
}
}
},
"number_of_replicas": "1",
"uuid": "s4oDgdsFTxOwsdRuPAWEkg",
"version": {
"created": "6030299"
}
}
}
}
}
But when I search with the following query I still get two matches! Both the upper and lower cases which must mean the analyser is not applied when I store the records.
Search like so...
GET /upper/_search
{
"query": {
"term": {
"text": {
"value": "test"
}
}
}
}
Thanks in advance!
first thing first you set your analyzer on the title field instead of upon the text field (since your search is on the text property, and since you are indexing doc with only text property)
"properties": {
"title": {
"type": "text",
"analyzer": "rebuilt_standard"
}
}
try
"properties": {
"text": {
"type": "text",
"analyzer": "rebuilt_standard"
}
}
and keep us posted ;)
I have created the index in elasticsearch with the following mapping:
{
"test": {
"mappings": {
"documents": {
"properties": {
"fields": {
"type": "nested",
"properties": {
"uid": {
"type": "keyword"
},
"value": {
"type": "text",
"copy_to": [
"fulltext"
]
}
}
},
"fulltext": {
"type": "text"
},
"tags": {
"type": "text"
},
"title": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"url": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
}
}
}
}
}
}
While searching I want to set the preference of fields for example if search text found in title or url then that document comes first then other documents.
Can we set a field preference for search result sequence(in my case preference like title,url,tags,fields)?
Please help me into this?
This is called "boosting" . Prior to elasticsearch 5.0.0 - boosting could be applied in indexing phase or query phase( added as part of field mapping ). This feature is deprecated now and all mappings after 5.0 are applied in query time .
Current recommendation is to to use query time boosting.
Please read this documents to get details on how to use boosting:
1 - https://www.elastic.co/guide/en/elasticsearch/guide/current/_boosting_query_clauses.html
2 - https://www.elastic.co/guide/en/elasticsearch/guide/current/_boosting_query_clauses.html
I want to sort elasticsearch result documents with icu_collation filter. So I have
settings for index:
"settings": {
"analysis": {
"analyzer": {
"ducet_sort": {
"tokenizer": "keyword",
"filter": [ "icu_collation" ]
}
}
}
}
and mappings
"mappings": {
"card": {
"properties": {
"title": {
"type": "text",
"fields": {
"sort": {
"type": "text",
"analyzer": "ducet_sort",
"index": false
}
}
}
}}}
and query:
{
"sort": ["title.sort"]
}
But query failed:
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [title.sort] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory."
}
In documentation the suggested data type for sorting is keyword. But data type keyword doesn't support analyzer. In addition the fielddata is not recommended:
documentation
So is there a way for sorting documents in elasticsearch with some specific collation e.g. icu_collation without fielddata=true?
Thank you.
In Kibana, open Dev Tools option from left menu and execute the query below after update according to your settings.
PUT _mapping/INDEX_NAME?update_all_types
{
"properties": {
"FIELD_NAME": {
"type": "text",
"fielddata": true
}
}
}
or through Curl or a terminal like Cygwin(for Windows) execute the query below after update according to your settings.
curl -XPUT http://DOCKER_MACHINE_IP:9200/INDEX_NAME -d '{
"mappings": {
"type": {
"properties": {
"FIELD_NAME": {
"type": "text",
"fielddata": true
}
}
}
}
}'
The documentation is sparse and not entirely helpful. So say I have the following fields for my attribute:
{
"my_index": {
"mappings": {
"my_type": {
"my_attribute": {
"mapping": {
"my_attribute": {
"type": "string",
"analyzer": "my_analyzer",
"fields": {
"lowercased": {
"type": "string"
},
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
}
}
}
}
my_analyzer lowercases tokens (in addition to other stuff).
So now I would like to know if the following statements are true:
my_analyzer does not get applied to raw, because the not_analyzed index does not have any analyzers, as its name implies.
my_attribute and my_attribute.lowercased are the exact same, so it is redundant to have the field my_attribute.lowercased
Your first statement is correct, however the second is not. my_attribute and my_attribute.lowercased might not be the same since the former has your custom my_analyzer search and index analyzer, while my_attribute.lowercased has the standard analyzer (since no analyzer is specified the standard one kicks in).
Besides, your mapping is not correct the way it is written, it should be like this:
{
"mappings": {
"my_type": {
"properties": {
"my_attribute": {
"type": "string",
"analyzer": "my_analyzer",
"fields": {
"lowercased": {
"type": "string"
},
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
}
}