Elasticsearch data does not match mapping - elasticsearch

Question

I am migrating elasticsearch prod data from 1.4.3v to 5.5v, for which I am using reindex. When I try to reindex old ES index to new ES index the reindexing fails with an exception Failed Reason: mapper [THROUGHPUT_ROWS_PER_SEC] cannot be changed from type [long] to [float]. Failed Type: illegal_argument_exception
ES mapping for task_history index in ES 1.4.3v
{
"task_history": {
"mappings": {
"task_run_hist": {
"_all": {
"enabled": false
},
"_routing": {
"required": true,
"path": "org_id"
},
"properties": {
"RUN_TIME_IN_MINS": {
"type": "double"
},
"THROUGHPUT_ROWS_PER_SEC": {
"type": "long"
},
"account_name": {
"type": "string",
"index": "not_analyzed",
"store": true
}
}
}
}
}
}
ES mapping for task_history index in ES 5.5v (this mapping gets created as part reindexing)
{
"task_history": {
"mappings": {
"task_run_hist": {
"_all": {
"enabled": false
},
"_routing": {
"required": true
},
"properties": {
"RUN_TIME_IN_MINS": {
"type": "float"
},
"THROUGHPUT_ROWS_PER_SEC": {
"type": "long"
},
"account_name": {
"type": "keyword",
"store": true
}
}
}
}
}
}
Sample data
{
"_index": "task_history",
"_type": "task_run_hist",
"_id": "1421955143",
"_score": 1,
"_source": {
"RUN_TIME_IN_MINS": 0.47,
"THROUGHPUT_ROWS_PER_SEC": 46,
"org_id": "xxxxxx",
"account_name": "Soma Acc1"
}
},
{
"_index": "task_history",
"_type": "task_run_hist",
"_id": "1421943738",
"_score": 1,
"_source": {
"RUN_TIME_IN_MINS": 1.02,
"THROUGHPUT_ROWS_PER_SEC": 65.28,
"org_id": "yyyyyy",
"account_name": "Choma Acc1"
}
}
2 Questions
How elasticsearch 1.4.3 is saving float numbers when mapping for THROUGHPUT_ROWS_PER_SEC type is long?
If it's a data issue in old ES how can I remove all float numbers before starting the reindexing process?
For 2nd option I am trying to list all documents having float numbers using below query, so that I can verify once and delete it, but below query still lists documents having THROUGHPUT_ROWS_PER_SEC as non floating numbers.
Note: Groovy scripting is enabled
GET task_history/task_run_hist/_search?size=100
{
"filter": {
"script": {
"script": "doc['THROUGHPUT_ROWS_PER_SEC'].value % 1 == 0"
}
}
}
Updated with solution provided by Val
When I try below script in reindexing, I get a runtime error. Listed below. Any clue on what is getting wrond here? I added additional condition to convert RUN_TIME_IN_MINS to float as your original script pointed out an error in RUN_TIME_IN_MINS field. mapper [RUN_TIME_IN_MINS] cannot be changed from type [long] to [float]"
POST _reindex?wait_for_completion=false
{
"source": {
"remote": {
"host": "http://esip:15000"
},
"index": "task_history"
},
"dest": {
"index": "task_history"
},
"script": {
"inline": "if (ctx._source.THROUGHPUT_ROWS_PER_SEC % 1 != 0) { ctx.op = 'noop' } ctx._source.RUN_TIME_IN_MINS = (float) ctx._source.RUN_TIME_IN_MINS;",
"lang": "painless"
}
}
Runtime error
{
"completed": true,
"task": {
"node": "wZOzypYlSayIRlhp9y3lVA",
"id": 645528,
"type": "transport",
"action": "indices:data/write/reindex",
"status": {
"total": 18249521,
"updated": 4691,
"created": 181721,
"deleted": 0,
"batches": 37,
"version_conflicts": 0,
"noops": 67076,
"retries": {
"bulk": 0,
"search": 0
},
"throttled_millis": 0,
"requests_per_second": -1,
"throttled_until_millis": 0
},
"description": """
reindex from [host=esip port=15000 query={
"match_all" : {
"boost" : 1.0
}
}][task_history] updated with Script{type=inline, lang='painless', idOrCode='if (ctx._source.THROUGHPUT_ROWS_PER_SEC % 1 != 0) { ctx.op = 'noop' } ctx._source.RUN_TIME_IN_MINS = (float) ctx._source.RUN_TIME_IN_MINS;', options={}, params={}} to [task_history]
""",
"start_time_in_millis": 1502336063507,
"running_time_in_nanos": 93094657751,
"cancellable": true
},
"error": {
"type": "script_exception",
"reason": "runtime error",
"script_stack": [],
"script": "if (ctx._source.THROUGHPUT_ROWS_PER_SEC % 1 != 0) { ctx.op = 'noop' } ctx._source.RUN_TIME_IN_MINS = (float) ctx._source.RUN_TIME_IN_MINS;",
"lang": "painless",
"caused_by": {
"type": "null_pointer_exception",
"reason": null
}
}
}

Answer 1

You obviously want to keep your existing ES 5.x mapping with a longso all you need to do is to add a script to your reindex call that modifies the THROUGHPUT_ROWS_PER_SEC field to a long. Something like this should do:
POST _reindex
{
"source": {
"remote": {
"host": "http://es1host:9200",
},
"index": "task_history"
},
"dest": {
"index": "task_history"
},
"script": {
"inline": "if (ctx._source.THROUGHPUT_ROWS_PER_SEC % 1 != 0) { ctx.op = 'noop' }" },
"lang": "painless"
}
}