Elasticsearch: Handling updates out of order - elasticsearch

Question

Imagine I have the following document:
{
"name": "Foo"
"age": 0
}
We receive events that trigger updates to these fields:
Event 1
{
"service_timestamp": "2019-09-15T09:00:01",
"updated_name": "Bar"
}
Event 2
{
"service_timestamp": "2019-09-15T09:00:02",
"updated_name": "Foo"
}
Event 2 was published by our service 1 second later than Event 1, so we would expect our document to first update the "name" property to "Bar", then back to "Foo". However, imagine that for whatever reason these events hit out of order (Event 2 THEN Event 1). The final state of the document will be "Bar", which is not the desired behavior.
We need to guarantee that we update our document in the order of the "service_timestamp" field on the event.
One solution we came up with is to have an additional last_updated_property on each field like so:
{
"name": {
"value": "Foo",
"last_updated_time": 1970-01-01T00:00:00
}
"age": {
"value": 0,
"last_updated_time": 1970-01-01T00:00:00
}
}
We would then only update the property if the service_timestamp of the event occurs after the last_updated_time of the property in the document:
{
"script": {
"source": "if (ctx._source.name.last_updated_time < event.service_timestamp) {
ctx._source.name.value = event.updated_name;
ctx._source.name.last_updated_time = event.service_timestamp;
}"
}
}
While this would work, it seems costly to perform a read, then a write on each update. Are there any other ways to guarantee events update in the correct order?
Edit 1: Some other things to consider
We cannot assume out-of-order events will occur in a small time window. Imagine the following: we attempt to update a customer's name, but this update fails, so we store the update event in some dead letter queue with the intention of refiring it later. We fix the bug that caused the update to fail, and refire all events in the dead letter queue. If no updates occurred that update the name field during the time we were fixing this bug, then the event in the dead letter queue should successfully update the property. However, if some events did update the name, the event in the dead letter queue should not update the property.

Answer 1

Everything Mousa said is correct wrt "Internal" versioning, which is where you let Elasticsearch handle incrementing the version.
However, Elasticsearch also supports "External" versioning, where you can provide a version with each update that gets checked against the current doc's version. I believe this would solve your case of events indexing to ES "out of order", and would prevent those issues across any timeframe of events (whether 1 second or 1 week apart, as in you dead letter queue example).
To do this, you'd track the version of documents in your primary datastore (Elasticsearch should never be a primary datastore!), and attach it to indexing requests.
First you'd create your doc with any version number you want, let's start with 1:
POST localhost:9200/my-index/my-type/<doc id>?version=1&version_type=external -d
{
"name": "Foo"
"age": 0
}
Then the updates would also get assigned versions from your service and/or primary datastore
Event 1
POST localhost:9200/my-index/my-type/<doc id>?version=2&version_type=external -d
{
"service_timestamp": "2019-09-15T09:00:01",
"updated_name": "Bar"
}
Event 2
POST localhost:9200/my-index/my-type/<doc id>?version=3&version_type=external -d
{
"service_timestamp": "2019-09-15T09:00:02",
"updated_name": "Foo"
}
This ensures that even if the updates are applied out of order the most recent one wins. If Event 1 is applied after event 2, you'd get a 409 error code that represents a VersionConflictEngineException, and most importantly Event 1 would NOT override event 2.
Instead of incrementing a version int by 1 each time, you could choose to convert your timestamps to epoch millis and provide that as the version - similar to your idea of creating a last_updated_property field, but taking advantage of Elasticsearch's built in versioning. That way, the most recently timestamped update will always "win" and be applied last.
I highly recommend you read this short blog post on Elasticsearch versioning - it goes into way more detail than I did here: https://www.elastic.co/blog/elasticsearch-versioning-support.
Happy searching!