Defining 2 splits to run a set of steps in parallel - spring

Question

I have a job configuration where I load a set of files in parallel, after the set of files is loaded I also want to load another set of files in parallel, but only after the first set is completely loaded. The 2nd set has referential fields to the first set. I thought I can use a second split but never got it working, in the xsd it seems you can define more than one split and obviously a flow does not help me with my requirement.
So how do I define 2 sets of parallel flows which run in sequence to each?
<job>
<split>
<flow>
<step next="step2"/>
<step id="step2"/>
</flow>
<flow>
<step ...>
</flow>
</split>
<split ../>
Asoub was right, it is simply possible, I did a simple config and it worked. So seems the original issue I got has some other issue which causes problems when defining 2 splits.
Simple config I used:
<batch:job id="batchJob" restartable="true">
<batch:split id="x" next="y">
<batch:flow>
<batch:step id="a">
<batch:tasklet allow-start-if-complete="true">
<batch:chunk reader="itemReader" writer="itemWriter" commit-interval="2"/>
</batch:tasklet>
</batch:step>
</batch:flow>
<batch:flow>
<batch:step id="b">
<batch:tasklet allow-start-if-complete="true">
<batch:chunk reader="itemReader" writer="itemWriter" commit-interval="2"/>
</batch:tasklet>
</batch:step>
</batch:flow>
</batch:split>
<batch:split id="y" next="e">
<batch:flow>
<batch:step id="c">
<batch:tasklet allow-start-if-complete="true">
<batch:chunk reader="itemReader" writer="itemWriter" commit-interval="2"/>
</batch:tasklet>
</batch:step>
</batch:flow>
<batch:flow>
<batch:step id="d">
<batch:tasklet allow-start-if-complete="true">
<batch:chunk reader="itemReader" writer="itemWriter" commit-interval="2"/>
</batch:tasklet>
</batch:step>
</batch:flow>
</batch:split>
<batch:step id="e">
<batch:tasklet allow-start-if-complete="true">
<batch:chunk reader="itemReader" writer="itemWriter" commit-interval="2"/>
</batch:tasklet>
</batch:step>
</batch:job>
INFO: Job: [FlowJob: [name=batchJob]] launched with the following parameters: [{random=994444}]
Nov 23, 2016 11:33:24 PM org.springframework.batch.core.job.SimpleStepHandler handleStep
INFO: Executing step: [a]
Nov 23, 2016 11:33:24 PM org.springframework.batch.core.job.SimpleStepHandler handleStep
INFO: Executing step: [b]
Nov 23, 2016 11:33:24 PM org.springframework.batch.core.job.SimpleStepHandler handleStep
INFO: Executing step: [c]
Nov 23, 2016 11:33:24 PM org.springframework.batch.core.job.SimpleStepHandler handleStep
INFO: Executing step: [d]
Nov 23, 2016 11:33:24 PM org.springframework.batch.core.job.SimpleStepHandler handleStep
INFO: Executing step: [e]
Nov 23, 2016 11:33:25 PM org.springframework.batch.core.launch.support.SimpleJobLauncher run
INFO: Job: [FlowJob: [name=batchJob]] completed with the following parameters: [{random=994444}] and the following status: [COMPLETED]

Answer 1

As I said in comments, "So how do I define 2 sets of parallel flows which run in sequence to each?" doesn't make sense per se, you can't start two step in parrallel and sequentially.
Still I think you want to "start loading file2 in step2 when file1 in step1 as finished loading". Which means that loading a file occurs in the middle of a step. I see two way of solving this.
Let's say this is your configuration:
<job id="job1">
<split id="split1" task-executor="taskExecutor" next="step3">
<flow>
<step id="step1" parent="s1"/>
</flow>
<flow>
<step id="step2" parent="s2"/>
</flow>
</split>
<step id="step3" parent="s4"/> <!-- not important here -->
</job>
<beans:bean id="taskExecutor" class="org.spr...SimpleAsyncTaskExecutor"/>
But this will start both of your step in parrallel immediatly. You need to prevent the start of step 2. So, you need to use a Delegate in your step2's reader that will immediatly stop from loading file2, and waits for a signal to start the reading. And somewhere in the code of the step1, where you consider loading to be done, you launch a signal to step2's delegate reader to start loading file2.
The second solution is: you create your own SimpleAsyncTaskExecutor which will start step1 and wait for the signal from step1 to start step2. It's basically the first solution, but you wait for the signal in your custom Executor rather than in a Delegate reader. (you can copy source code from SimpleAsyncTaskExecutor to get an idea)
This comes at a cost, if the step1 never reaches the part where it signal step2 to start loading, your batch will hang forever. Maybe an exception in loading could cause this. As for signal mechanisms, Java has a lot of way to do this (wait() and notifiy(), locks, semaphore, non-standard library maybe).
I don't think there is some king of parrallel step trigger in spring batch (but if there is, someone posts it).
I've already answered a little while asking to your question, you need 2 splits: the first one loads the set of files A, and second, set of files B.
<job id="job1">
<split id="splitForSet_A" task-executor="taskExecutor" next="splitForSet_B">
<flow><step id="step1" parent="s1"/></flow>
<flow><step id="step2" parent="s2"/></flow>
<flow><step id="step3" parent="s3"/></flow>
</split>
<split id="splitForSet_B" task-executor="taskExecutor" next="stepWhatever">
<flow><step id="step4" parent="s4"/></flow>
<flow><step id="step5" parent="s5"/></flow>
<flow><step id="step6" parent="s6"/></flow>
</split>
<step id="stepWhatever" parent="sx"/>
</job>
Steps 1, 2 and 3 will run in parrallel (and load fileset A), then, once they're all over, the second split (splitForSet_B) will start and run steps 4, 5 and 6 in parrallel. A split is basicaly a step that contains steps running in parrallel.
You just need to specify in each steps what file you will be using (so it will be different for steps in first split from steps in second split.

Answer 2

I'd use two partitioned steps. Each partitioner would be responsible for identifying the files in its respective set for the concurrent child-steps to process
<job>
<step name="loadFirstSet">
<partition partitioner="firstSetPartitioner">
<handler task-executor="asyncTaskExecutor" />
<step name="loadFileFromSetOne>
<tasklet>
<chunk reader="someReader" writer="someWriter" commit-interval="#{jobParameters['commit.interval']}" />
</tasklet>
</step>
</partition>
</step>
<step name="loadSecondSet">
<partition partitioner="secondSetPartitioner">
<handler task-executor="asyncTaskExecutor" />
<step name="loadFileFromSecondSet>
<tasklet>
<chunk reader="someOtherReader" writer="someOtherWriter" commit-interval="#{jobParameters['another.commit.interval']}" />
</tasklet>
</step>
</partition>
</step>
</job>