Preserve wrong messages in Elasticsearch - elasticsearch

Question

I have a static mapping in Elasticsearch index. When a message doesn't match this mapping, it is discarded. Is there a way to route it to a default index for wrong messages?
To give you example, I have some fields with integer type:
"status_code": {
"type": "integer"
},
When a message contains a number
"status_code": 123,
it's ok. But when it is
"status_code": "abc"
it fails.

Answer 1

You can have ES do this triage pretty easily using ingest nodes/processors.
The main idea is to create an ingest pipeline with a convert processor for the status_code field and if the conversion doesn't work, you can add an on_failure condition which will direct the document at another index that you can later process.
So create the failures ingest pipeline:
PUT _ingest/pipeline/failures
{
"processors": [
{
"convert": {
"field": "status_code",
"type": "integer"
}
}
],
"on_failure": [
{
"set": {
"field": "_index",
"value": "failed-{{ _index }}"
}
}
]
}
Then when you index a document, you can simply specify the pipeline in parameter. Indexing a document with correct status code will succeed:
PUT test/doc/1?pipeline=failures
{
"status_code": 123
}
However, trying to index a document with a bad status code, will actually also succeed, but your document will be indexed in the failed-test index and not the test one:
PUT test/doc/2?pipeline=failures
{
"status_code": "abc"
}
After running these two commands, you'll see this:
GET failed-test/_search
{
"took" : 3,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [
{
"_index" : "failed-test",
"_type" : "doc",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"status_code" : "abc"
}
}
]
}
}
To sum up, you didn't have to handle that exceptional case in your client code and could fully leverage ES ingest nodes to achieve the same task.

Answer 2

You can set the parameter ignore malformed to ignore just the field with the type mismatch and not the whole document.
And you can try to combine it with multi-fields, that allows you to map the same value in different ways.
You will probably need something like this:
"status_code": {
"type": "integer",
"fields": {
"as_string": {
"type": "keyword"
}
}
}
This way you will have a field named status_code as an intenger and the same value in a field named status_code.as_string as a keyword, but you should test to see if really does what you want.

Answer 3

Use Strict mapping and you will be able to catch the exception raised by Elastic.
Below is the excerpt from Elastic docs:
By default, when a previously unseen field is found in a document, Elasticsearch will add the new field to the type mapping. This behaviour can be disabled, both at the document and at the object level, by setting the dynamic parameter to false (to ignore new fields) or to strict (to throw an exception if an unknown field is encountered).
As a part of Exception handling, you can push the message to some other index where dynamic mapping is enabled.