Are synonyms in Elastic Search (version 6.2.3) stored in the items when these are created/updated or are synonyms applied in every search query to the index?
We need to remove the synonyms of an index with 6 million items and I cannot see in the documentation if removing these synonyms from the index will be enough
DELETE /api/as/v1/engines/{ENGINE_NAME}/synonyms/{SYNONYM_SET_ID}
Or it is needed to reindex all the items afterwards, in which case it might be better to delete the current index and create a new one.
If synonyms are applied during inserting the document
Deletion of current synonyms, won't change anything in the existing data of an index, existing data should be searchable by synonyms.
If synonyms are applied during query time
In this case, removing the synonyms will stop searching the document using synonym.
Now the question is whether you are using index-time-analysis or query-time-analysis. You can check in your mappings. E.g
"mappings": {
"properties": {
"text": {
"type": "text",
"analyzer": "autocomplete", // <======== For index time analysis
"search_analyzer": "synonym_analyzer" //<====== For Query time analysis
}
}
}
}
Related
I have read an explanation to my question here:
https://discuss.elastic.co/t/whats-the-differece-between-index-pattern-and-index-template/54948
However, I still don't understand the difference. When defining an index PATTERN, does it not affect index creation at all? Also, what happens if I create an index but it doesn't have a corresponding index pattern? How can I see the mapping used for an index pattern so I can know how to use the Mapping API to update it?
And on a side note, the docs say you manage the index patterns by clicking the "Settings" and then "Indices" tab. I'm looking at Kibana and I don't see any settings tab. I can view the index patterns through the management tab, but I don't see any settings tab there
An index template is an ES feature for triggering the creation of new indexes whenever a name pattern is matched. For instance, let's say we create the following index template:
PUT _template/template_1
{
"index_patterns": ["foo*"],
"settings": {
"number_of_shards": 1
},
"mappings": {
...
}
}
As you can see, as soon as we want to index a document inside an index named (e.g.) foo-44 and that index doesn't exist, then that template (settings + mappings) will be used by ES in order to create the foo-44 index automatically.
You can update an index template at any time by simply PUTting a new settings/mappings definition like above.
An index pattern (not to be confounded with the index-patterns property you saw above, those are two totally different things), is a Kibana feature for telling Kibana what makes up an index (all the fields, their types, etc). Nothing can happen in Kibana without creating index patterns, which you can do in Management > Index Patterns.
Creating an index in ES will not create any index pattern in Kibana. Similarly, creating an index pattern in Kibana will not create any index in ES.
The reason why Kibana needs an index pattern is because it needs to store different kind of information as it available in an index mapping. For instance, let's say you create an index with the following mapping:
PUT my_index
{
"mappings": {
"doc": {
"properties": {
"timestamp": {
"type": "date"
},
"name": {
"type": "text"
}
}
}
}
}
Then the corresponding index pattern that you will create in Kibana will have the following content:
GET .kibana/doc/index-pattern:16a98050-a53f-11e8-82ab-af0d48c6ddd8
{
"type": "index-pattern",
"updated_at": "2018-08-21T12:38:22.509Z",
"index-pattern": {
"title": "my_index*",
"timeFieldName": "timestamp",
"fields": """[{"name":"_id","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":false},{"name":"_index","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":false},{"name":"_score","type":"number","count":0,"scripted":false,"searchable":false,"aggregatable":false,"readFromDocValues":false},{"name":"_source","type":"_source","count":0,"scripted":false,"searchable":false,"aggregatable":false,"readFromDocValues":false},{"name":"_type","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":false},{"name":"name","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":false,"readFromDocValues":false},{"name":"timestamp","type":"date","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true}]"""
}
}
As you can see, Kibana also stores the timestamp field, the name of the index pattern (which can span several indexes). Also it stores various properties for each field you have defined, for instance, for the name field, the index-pattern contains the following information that Kibana needs to know:
{
"name": "name",
"type": "string",
"count": 0,
"scripted": false,
"searchable": true,
"aggregatable": false,
"readFromDocValues": false
},
what is offline and online indexing in Elastic search? I did my research but I couldn't find enough resources to see what these terms mean? any idea? and also when do we need to reindex? any examples would be great
The terms offline and online indexing are used here.
https://spark-summit.org/2014/wp-content/uploads/2014/07/Streamlining-Search-Indexing-using-Elastic-Search-and-Spark-Holden-Karau.pdf
Reindexing
The most basic form if reindexing just copies one index to another.
I have used this form of reindexing to change a mapping.
Elasticsearch doesn't allow you to change a mapping, so if you want to change a mapping you have to create a new index (index2) with a new mapping and then reindex. The reindex will fill that new mapping with the data of the old index.
The command below will move everything from index to index2.
curl -XPOST 'localhost:9200/_reindex?pretty' -d'
{
"source": {
"index": "index"
},
"dest": {
"index": "index2"
}
}'
You can also use reindexing to fill a new index with a part of the old one. You can do so by using a couple of parameters. The example below will copy the newest 1000 documents.
POST /_reindex
{
"size": 1000,
"source": {
"index": "index",
"sort": { "date": "desc" }
},
"dest": {
"index": "index2"
}
}
For more examples about reindexing please have a look at the official documentation.
offline vs online indexing
In ONLINE mode the new index is built while the old index is accessible to reads and writes. any update on the old index will also get applied to the new index.
In OFFLINE mode the table is locked up front for any read or write, and then the new index gets built from the old index. No read or write operation is permitted on the table while the index is being rebuilt. Only when the operation is done is the lock on the table released and reads and writes are allowed again.
I have mappings created wrongly for an object in elastic search. Is there a way to update the mappings. The mapping has been created wrongly for type of the object(String instead of double).
In general, the mapping for existing fields cannot be updated. There are some exceptions to this rule. For instance:
new properties can be added to Object datatype fields.
new multi-fields can be added to existing fields.
doc_values can be disabled, but not enabled.
the ignore_above parameter can be updated.
Source : https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html
That's entirely possible, by PUTting the new mapping over the existing one, here are some examples.
Please note, that you will probably need to reindex all your data after you have done this, because I don't think that ES can convert string indexes to double indexes. (what will instead happen is, that you won't find any document when you search in that field)
PUT Mapping API allows you to add/modified datatype in an existing index.
PUT /assets/asset/_mapping
{
"properties": {
"common_attributes.asset_id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"doc_values": true,
"normalizer": "lowercase_normalizer"
}
}
},
}
}
After updating the mapping, update the existing documents using bulk Update API.
POST /_bulk
{"update":{"_id":"59519","_type":"asset","_index":"assets"}}
{"doc":{"facility_id":491},"detect_noop":false}
Note - Use 'detect_noop' for detecting noop update.
I am new to elastic search, I want to updated the existing mapping under my index. My existing mapping looks like
"load":{
"mappings": {
"load": {
"properties":{
"customerReferenceNumbers": {
"type": "string",
"index": "no"
}
}
}
}
}
I would like to update this field from my mapping to be analyzed, so that my 'customerReferenceNumber' field will be available for search.
I am trying to run the following query in Sense plugin to do so,
PUT /load/load/_mapping { "load": {
"properties": {
"customerReferenceNumbers": {
"type": "string",
"index": "analyzed"
}
}
}}
but I am getting following error with this command,
MergeMappingException[Merge failed with failures {[mapper customerReferenceNumbers] has different index values]
Though there exist data associated with these mappings, here I am unable to understand why elastic search not allowing me to update mapping from no-index to indexed?
Thanks in advance!!
ElasticSearch doesn't allow this kind of change.
And even if it was possible, as you will have to reindex your data for your new mapping to be used, it is faster for you to create a new index with the new mapping, and reindex your data into it.
If you can't afford any downtime, take a look at the alias feature which is designed for these use cases.
This is by design. You cannot change the mapping of an existing field in this way. Read more about this at https://www.elastic.co/blog/changing-mapping-with-zero-downtime and https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html.
If I change the mapping so certain properties have new/different boost values, does that work even if the documents have already been indexed? Or do the boost values get applied when the document is indexed?
You cannot change field level boost factors after indexing data. It's not even possible for new data to be indexed once the same fields have been indexed already for previous data.
The only way to change the boost factor is to reindex your data. The pattern to do this without changing the code of your application is to use aliases. An alias points to a specific index. In case you want to change the index, you create a new index, then reindex data from the old index to the new index and finally you change the alias to point to the new index. Reindexing data is either supported by the elasticsearch library or can be achieved with a scan/scroll.
First version of mapping
Index: items_v1
Alias: items -> items_v1
Change necessary, sencond version of the index with new field level boost values :
Create new index: items_v2
Reindex data: items_v1 => items_v2
Change alias: items -> items_v2
This might be useful in other situations where you want to change your mapping.
Field level boosts are, however, not recommended. The better approach is to use boosting at query time.
Alias commands are:
Adding an alias
POST /_aliases
{
"actions": [
{ "add": {
"alias": "tems",
"index": "items_v1"
}}
]
}
Removing an alias
POST /_aliases
{
"actions": [
{ "remove": {
"alias": "tems",
"index": "items_v1"
}}
]
}
They do not.
Index time boosting is generally not recommended. Instead, you should do your boosting when you search.