AWS step functions - Nested Map type - aws-lambda

Question

I am new to AWS step functions and I'm trying to nest multiple Map type Lambda tasks. I am experiencing unexpected behaviour with my approach though. get_item_list outputs a list of items to be fed into the get_item_pages Map. The get_item_pages Map, iterates over get_info. I then want to pass each output of get_info to another Map type, get_more_info. The reason I am using Map functions is so that I can utilise the fan out functionality. The first Map type, get_item_pages, fans out as expected. In the Lambda CloudWatch logs, I can see 10+ log streams start. The get_more_info state does not appear to be fanning out in the same fashion though. I often only see a single log stream in the Lambda CloudWatch for the nested Map state.
Am I missing something obvious in my implementation or am I going about this in an entirely wrong way?
{
"Comment": "A nested map example",
"StartAt": "get_item_list",
"States": {
"get_item_list": {
"Type": "Task",
"Resource": "arn:aws:lambda:...",
"Next": "get_item_pages",
"ResultPath": "$.data"
},
"get_item_pages": {
"Type": "Map",
"ItemsPath": "$.data.all_items",
"MaxConcurrency": 100,
"Iterator": {
"StartAt": "get_info",
"States": {
"get_info": {
"Type": "Task",
"Resource": "arn:aws:lambda:...",
"ResultPath": "$.data",
"Next": "get_more_info"
},
"get_more_info": {
"Type": "Map",
"ItemsPath": "$.data.all_data",
"MaxConcurrency": 100,
"Iterator": {
"StartAt": "get_detailed_info",
"States": {
"get_detailed_info": {
"Type": "Task",
"Resource": "arn:aws:lambda:...",
"End": true
}
}
},
"End": true
}
}
},
"End": true
}
}
}

Answer 1

This is an old question, but thought I'd answer for anyone else. I can't say for certain, but it seems likely that you're experiencing that because the "End": true (meant to indicate completion of the map) is supposed to be at the Task level, not the Map level. Putting "End": true at the map level indicates the end of the state machine.
The exact outcome is probably indeterminant because AWS doesn't tell you EXACTLY how the iterations will flow through the map, only that it will be done so in accordance with the concurrency you set, so 100. Even so, all iterations (up to 100) are not guaranteed to move through the map concurrently.
So, what's likely happening is the first iteration is flowing through the get_more_info (nested) map and then triggering "End": true, thus ending the entire state machine.