How to poll in Mule until all jobs are completed - caching

Question

I have a requirement to build a workflow triggers a set of batch jobs by calling an API and then polls another API to check when each batch job is completed. Only when all batch jobs are complete then the workflow can move onto the next step. What is the best way to do this?
I had thought about using the Poll component but I am not sure how I could start and stop the poll as my experience has been to run the poll at a scheduled time or continually poll the external source. My current train of thought is to use a flag in the expression box which is set to true once all batch jobs are completed.
The other issue is that the batch job Ids are all in a JSON object and what would be the best way to check off each batch job Id as the API starts to return results showing the batch jobs completing?
I am using Anypoint Studio 6.2 and Mule 3.8.3
Thanks

Answer 1

First - Assume your api call trigger 5 batch jobs. On each job completes you need to update the status of job success/failure/inprogress in db or in object store ( or any other retrievable way).
Let say your minimum time your jobs will complete in 1 hour.
Assume you are updating status in DB.
Create poll flow to check whether your job status is success/failure in db for every one hour and make your flow in stoppedstate.
<flow name="Job_status_check_flow" initialState="stopped">
<poll doc:name="Poll">
<fixed-frequency-scheduler frequency="1" timeUnit="HOURS"/>
<logger level="INFO" doc:name="Logger"/>
</poll>
<logger message="poll" level="INFO" doc:name="Logger"/>
<db component or object store here --->
</flow>
Since the flow is in stopped condition, poll wont trigger until the flow changed to start state. You have a control.
Always the flow will be in stopped state.When you request api to trigger 5 batch jobs at the same time also start the 'Job_status_check_flow' to start( you can use groovy component- to start and stop the flow based on the condition). Please check the links below
Starting a mule flow programmatically using groovy
auto-starting Mule flow
In this case, Poll flow check the status for every 1 hour until DB retrieves all the status of 5 jobs to completed status. If so have a groovy component in end of the 'Job_status_check_flow' to stopped state. So that poll wont trigger again.