ElasticSearch indexing with nested collections in document - elasticsearch

Question

I have been wrestling with an issue trying to index a document into a brand new index in ElasticSearch. My document looks something like this:
{
"id": "",
"name": "Process to run batch of steps",
"defaultErrorStep": {
"id": "d44fdeae-80ff-4509-8504-9dfbd7284631",
"name": "General Error Handler",
"type": "ERROR",
"reference": "error",
"onError": "DEFAULT"
},
"startingStep": "one",
"steps": [
{
"id": "d44fdeae-80ff-4509-8504-9dfbd7284631",
"name": "Step One",
"type": "CHAIN",
"reference": "one",
"onComplete": "two",
"onError": "DEFAULT",
"parameterKeys": {
"param-a": "value-a",
"param-b": "value-b",
"param-c": "value-c"
}
},
{
"id": "d44fdeae-80ff-4509-8504-9dfbd7284631",
"name": "Step Two",
"type": "CHAIN",
"reference": "two",
"onComplete": "two",
"onError": "DEFAULT",
"parameterKeys": {
"param-a": "value-a",
"param-b": "value-b",
"param-c": "value-c"
}
},
{
"id": "d44fdeae-80ff-4509-8504-9dfbd7284631",
"name": "Step Three",
"type": "BOOLEAN",
"reference": "three",
"onTrue": "four",
"onFalse": "five",
"onError": "DEFAULT",
"parameterKeys": {
"param-a": "value-a",
"param-b": "value-b",
"param-c": "value-c"
}
},
{
"id": "d44fdeae-80ff-4509-8504-9dfbd7284631",
"name": "Step Four",
"type": "LOOP",
"startingStep": "seven",
"steps": [
{
"id": "d44fdeae-80ff-4509-8504-9dfbd7284631",
"name": "Step Two",
"type": "CHAIN",
"reference": "six",
"onComplete": "seven",
"onError": "DEFAULT",
"parameterKeys": {
"param-a": "value-a",
"param-b": "value-b",
"param-c": "value-c"
}
},
{
"id": "d44fdeae-80ff-4509-8504-9dfbd7284631",
"name": "Step Five",
"type": "FINISH_VOID",
"end": false,
"reference": "seven",
"onError": "DEFAULT",
"parameterKeys": {
"param-a": "value-a",
"param-b": "value-b",
"param-c": "value-c"
}
}
],
"reference": "four",
"onComplete": "five",
"onError": "DEFAULT",
"parameterKeys": {
"param-a": "value-a",
"param-b": "value-b",
"param-c": "value-c"
}
},
{
"id": "d44fdeae-80ff-4509-8504-9dfbd7284631",
"name": "Step Five",
"type": "FINISH",
"end": true,
"reference": "five",
"onError": "DEFAULT",
"parameterKeys": {
"param-a": "value-a",
"param-b": "value-b",
"param-c": "value-c"
}
}
],
"configuration": {
"settings": {
"property-a": "a",
"property-b": "b",
"property-c": "c",
"property-d": "d",
"property-z": "z123"
}
}
}
My issue is that due to the nested structure of the property "steps" and its ability to also have loop objects with "steps" inside of that, I get into an issue of field duplication when trying to index. I understand the reason (I think) as to why my document is failing but I need to index it all the same. When I try to index the document I get the following error:
ElasticsearchException[Elasticsearch exception [type=json_parse_exception, reason=Duplicate field 'type'\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper#84a0697; line: 1, column: 186]]]
Again, I understand why this is an issue but I figured I could address this with mappings in my index. I have tried nested object type, flattened object types and even setting index:false on the steps field just to see if I could get the document to go in. But, no chance. I know this is a going to be a simple fix somewhere I just cannot see but does anyone have any thoughts on what I can try to get this document to index.
I am using ElasticSearch 7.3.1 via the latest Java SDK release. I have bypassed the java code for now and just using POSTMAN to send the indexing command but still I get the same issue.
Below is an example of one of the mappings I have tried.
{
"_source" : {
"enabled": true
},
"properties" : {
"name": {
"type": "text",
"fields": {
"raw":{"type": "keyword"}
}
},
"steps":{
"type":"nested",
"properties":{
"steps":{
"type":"flattened",
"index":false
}
}
},
"configuration.settings":{"type":"flattened"}
}
}
As well as a more explicit mapping to cover the "defaultErrorStep" object.
{
"_source" : {
"enabled": true
},
"properties" : {
"name": {
"type": "text",
"fields": {
"raw":{"type": "keyword"}
}
},
"defaultErrorStep":{
"type":"object",
"properties":{
"id":{"type":"text"},
"name":{"type":"text"},
"type":{"type":"text"},
"reference":{"type":"text"},
"onError":{"type":"text"}
}
},
"steps":{
"type":"nested",
"properties":{
"id":{"type": "text"},
"name":{
"type": "text",
"fields": {
"raw":{"type": "keyword"}
}
},
"type":{"type": "text"},
"reference":{"type": "text"},
"onComplete":{"type": "text"},
"onError":{"type": "text"},
"parameterKeys":{"type": "object"},
"onTrue":{"type": "text"},
"onFalse":{"type": "text"},
"startingStep":{"type": "text"},
"steps":{
"type":"nested",
"properties":{
"id":{"type": "text"},
"name":{
"type": "text",
"fields": {
"raw":{"type": "keyword"}
}
},
"type":{"type": "text"},
"reference":{"type": "text"},
"onComplete":{"type": "text"},
"onError":{"type": "text"},
"parameterKeys":{"type": "object"},
"onTrue":{"type": "text"},
"onFalse":{"type": "text"},
"startingStep":{"type": "text"},
"steps":{
"type": "flattened",
"index":false
},
"end":{"type": "boolean"}
}
},
"end":{"type": "boolean"}
}
},
"configuration.settings":{"type":"flattened"}
}
}
Please also bear in mind that the nature of the document is to outline a process/workflow of logic and the structure is key and I would also say valid JSON. So in theory the steps property could nest 3,4,10 levels if it had to. So Ideally I wouldn't want to be updating the mapping every time a new level was added in the data.
Any help anyone can give me to get this document to index would be much appreciated.
Thanks,
EDIT:
I have since removed my explicit mapping from my index and let dynamic mapping take over as all my objects fit into the base types dynamic mapping supports. This has been successful and I am able to index the document shown above with infinitely nested steps no problem. I then tried the same operation with the same document structure using the JAVA SDK and this failed with the same duplicate field exception. This indicates to me the issue is with the JAVA SDK and not something native to elasticsearch itself.
Dynamic mapping is the better option in my case as I have no control over how many levels steps could eventually get to.
Has anyone experienced any issues with the SDK behaving differently to the base product?

Answer 1

I am running elastic 7.3.1 and with following index mapping i am successfully able to create index with nested types inside nested type.
PUT new_index_1
{
"mappings": {
"_source": {
"enabled": true
},
"properties": {
"name": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"steps": {
"type": "nested",
"properties": {
"steps": {
"type": "flattened",
"index": false
}
}
},
"configuration.settings": {
"type": "flattened"
}
}
}
}
Following index creation also works for me
PUT new_index_2
{
"mappings": {
"_source": {
"enabled": true
},
"properties": {
"name": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"steps": {
"type": "nested",
"properties": {
"steps": {
"type": "nested"
}
}
}
}
}
}
Document Indexed
POST new_index_1/_doc
{
"name": "ajay",
"steps": [
{
"test": "working",
"steps": [
{
"name": "crow"
}
]
}
]
}