I am using an elastic search for search purpose. But recently I observer that some random error while adding data into elastic search:
version conflict, required seqNo [113789], primary term [19]. current document has seqNo [113797] and primary term [19]
The above type error comes randomly and I am not able to add/update data in elastic search.
Can you please help to understand:
What is the root cause of this issue?
How I can reproduce this issue? as this coming randomly need to know the basic step to reproduce this issue
What is the solution for this? How I can solve this issue?
This error happened during an update of the document at the same time others updated it. Check the parallelism.
When this process read the document it had the version number 113789 and when ES received the update did not match with the version number (current version 113797). It causes the version conflict.
Related
My opensearch sometimes reaches this error when i adding new index:
Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;
So i have to increase cluster.max_shards_per_node larger.
I wonder if is there any way to check current shards we are using to avoid this error happening?
The best way to see indexing and search activity is by using a monitoring system. And the best monitoring system for Elasticsearch is Opster. You can try it for free at the following link.
https://opster.com/
For the manual check and sort, you can try the following APIs.
You can sort your indices according to the creation date string (cds). It will help you to understand which one is the old one. So you can have an idea about your indices (shards).
GET _cat/indices?v&h=index,cds&s=cds
Also, you check the indices stats to see if is there any activity in searching or indexing.
To check all indices you can use GET _all/_stats
To check only one index you can use GET index_name/_stats
I'm currently in the process of upgrading the spring boot version of my project. After upgrading from 2.5 to 2.6 a few tests started failing which deal with the retrival of elasticsearch documents. I'm trying to fetch only the highest scoring documents, but when expecting 2 identical documents, only 1 is retrieved.
After reading up on the issue I figured out that the problem comes down to the Elasticsearchindex using multiple shards, each having their own scoring logic and (probably?) the identical documents being fetched from different shards, thus resulting in different scores despite being virtually the same.
Now, can anyone tell me why this happens in the newer spring-data-elasticsearch version and if there is a setting to return it to the old functionality?
I've set up a little test project to play around with this. If anyone is interested in trying this for themselves, feel free to check it out: https://github.com/Moldavis/elasticsearch-scoring-poc
Actually found my own answer in the spring data breaking changes documentation (duh).
https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/#elasticsearch-migration-guide-4.2-4.3.breaking-changes
search_type default value
The default value for the search_type in Elasticsearch is query_then_fetch. This now is also set as default value in the Query implementations, it was previously set to dfs_query_then_fetch.
The dfs_query_then_fetch option queries all shards for document and term frequency to equal out the score between different shards. This is no longer used by default, therefore the mentioned problem occurs.
It can be fixed by setting the searchtype for the query like so:
queryBuilder.withSearchType(SearchType.DFS_QUERY_THEN_FETCH);
I am using Elasticsearch 7.9.0
I was updating the document very frequently. So I was getting the below exception
Elasticsearch exception [type=version_conflict_engine_exception, reason=[111]: version conflict, required seqNo [4348], primary term [2]. current document has seqNo [4427] and primary term [2]]
Then I have given a delay of 1 second between each update.(I can't give more then that)
But still the problem exists. How can we solve this.
Please help me.
Thanks.
This issue happens because of the versioning of document in elasticsearch. This feature exists in order to prevent concurrent changes to the same documents by tasks that runs simultaneously.
When you try to update a document that is already being updated by another task you might run into this issue.
If you want to track the update process of documents by your updates you may want to use the Task management API by elastic: https://www.elastic.co/guide/en/elasticsearch/reference/current/tasks.html
Also you might want to check this documentation on Index API as it explains further: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html
I received nearly the same error in OpenSearch configuration but it wasn't due to too frequent updates like in OP's case.
In my case, I was unknowingly trying to update an existing Role in the domain. My requests were trying to create a 'new' Role when it already existed. When I tried to do this, I received the error.
My resolution was to create a Role with an entirely new name and then update that.
is there a way to find out the names of all the indices ever created? Even after the index might have been deleted. Does elastic store such historical info?
Thanks
Using a plugin that keeps an audit trail for all changes that happened in your ES cluster might do the trick.
If you use the changes plugin (or a more recent one), then you can query it for all the changes in all indices using
curl -XGET http://localhost:9200/_changes
and your response will contain all the index names that were at least created. Not sure this plugin works with the latest versions of ES, though.
I would like to update an ElasticSearch Document while maintaining the document's version the same. I'm using version_type=external as indicated in the versioning section of the index_ documentation. Updating a document with another of the same version is normally prevented as indicated in that section: "If the value provided is less than or equal to the stored document’s version number, a version conflict will occur and the index operation will fail."
The reason I want to keep the version unaltered is because I do not create a new version of my object (stored in my database) when one adds new tags to that object, but I would like the new tags to show up in my ElasticSearch index. Is this possible with ElasticSearch?
I tried deleting the document and then adding a new document with the same Id and Version but that still gives me the following exception:
VersionConflictEngineException[[myindex][2] [mytype][6]: version
conflict, current 1, provided 1]
Just for reference, I'm using PHP Elastica (with methods $type->deleteDocument($doc); and $type->addDocument($doc);) but this question should apply to ElasticSearch in general.
The time for which elasticsearch keeps information about deleted documents is controlled by index.gc_deletes parameter. By default this time is 1m. So, theoretically, you can decrease this time to 0s, wait for a second, delete the document, index a new document with the same version, and set index.gc_deletes back to 1m. But at the moment that would work only on master due to a bug. If you are using older version of elasticsearch, you will not be able to change index.gc_deletes without closing the index first.
There is a good blog post on elasticsearch.org web site that describes how versions are handled by elasticsearch in details.