Elasticsearch error resource_already_exists_exception when index doesn't exist for sure - elasticsearch

Question

I use random index name for new indices:
async import_index(alias_name, mappings, loadFn) {
const index = `${alias_name}_${+new Date()}`
console.log('creating new index: ', index)
await this.esService.indices.create({
index: index,
body: {
"settings": this.index_settings(),
"mappings": mappings
}
}).then(res => {
console.log('index created: ', index)
}).catch(async (err) => {
console.error(alias_name, ": creating new index", JSON.stringify(err.meta, null, 2))
throw err
});
I believe an index with this name cannot exist, but ES returns me this error
"error": {
"root_cause": [
{
"type": "resource_already_exists_exception",
"reason": "index [brands_1637707367610/bvY5O_NjTm6mU3nQVx7QiA] already exists",
"index_uuid": "bvY5O_NjTm6mU3nQVx7QiA",
"index": "brands_1637707367610"
}
],
"type": "resource_already_exists_exception",
"reason": "index [brands_1637707367610/bvY5O_NjTm6mU3nQVx7QiA] already exists",
"index_uuid": "bvY5O_NjTm6mU3nQVx7QiA",
"index": "brands_1637707367610"
},
"status": 400
}
ES installed in k8s using bitnami helm chart, 3 master nodes running. Client is connected to master service url. My thoughts: client sends a request to all nodes at the same time, but i cannot prove it.
plz help

Answer 1

We have experienced same error with the python client. But from what i see the javascript client is written in similar way.
There is a retry option in the client.
Most likely this is set to true in your case (can be reconfigured). Then you pass a large mapping to this.esService.indices.create. The operation takes too long, times out, then a retry happens, but the index was created on the cluster.
You need to send a larger timeout to elasticsearch index create api (default 30s). And also set the same timeout for the http connection.
Those are 2 separate settings:
Client http connection requestTimeout . Default: 30s
https://github.com/elastic/elasticsearch-js/blob/v7.17.0/lib/Transport.js#L435
https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/7.17/basic-config.html
Serverside timeout via Create Index API. Default: 30s
https://www.elastic.co/guide/en/elasticsearch/reference/7.17/indices-create-index.html
this.esService = new Client({
// ....
maxRetries: 5,
requestTimeout: 60000 // time in ms
})
// ....
tnis.esService.indices.create({
index: index,
body: {
"settings": this.index_settings(),
"mappings": mappings
},
timeout: "60s" // elasticsearch timestring
})