Execute a sub-workflow for each line of a file - hadoop

Question

I'm using Oozie Sqoop Action to import data in the Datalake.
I need a HDFS folder for each table of the database source. I have more than 300 tables.
I could have all the 300 Sqoop Actions hardcoded in a Workflow but then the Workflow would be too big for the Oozie configuration.
Error submitting job /user/me/workflow.xml
E0736: Workflow definition length [107,123] exceeded maximum allowed length [100,000]
Having big file like that isn't a good idea because it slows the system (it is saved in the database) and it's hard to maintain.
Question is, how do I call a sub-workflow for each table name ?
Equivalent shell script would be something like:
while read TABLE; do
sqoop import --connect ${CONNECT} --username ${USERNAME} --password ${PASSWORD} --table ${TABLE} --target-dir ${HDFS_LOCATION}/${TABLE} --num-mappers ${NUM-MAPPERS}
done < tables.data
Where tables.data contains a table names list which is a subset of the database source tables names. For example :
TABLE_ONE
TABLE_TWO
TABLE_SIX
TABLE_TEN
And here the sub-workflow I want to call for each table:
<workflow-app name="sub-workflow-import-table" xmlns="uri:oozie:workflow:0.5">
<start to="sqoop-import"/>
<action name="sqoop-import">
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<command>sqoop import --connect ${CONNECT} --username ${USERNAME} --password ${PASSWORD} --table ${TABLE} --target-dir ${HDFS_LOCATION}/${TABLE} --num-mappers ${NUM-MAPPERS}</command>
</sqoop>
<ok to="end"/>
<error to="log-and-kill"/>
</action>
<end name="end"/>
<kill name="log-and-kill">
<message>Workflow failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
</workflow-app>
Let me know if you need more precision.
Thanks!
David

Answer 1

There's sadly no way to do this nicely in Oozie - you'd need to hardcode all 300 Sqoop actions into an Oozie XML. This is because Oozie deals with directed acyclic graphs, which means loops (like your shell script) don't have an Oozie equivalent.
However I don't think Oozie is the right tool here. Oozie requires one container per action to use as a launcher, which means your cluster will need to allocate 300 additional containers over the space of a single run. This can effectively deadlock a cluster as you end up in situations where launchers prevent the actual jobs running! I've worked on a large cluster with > 1000 tables and we used Bash there to avoid this issue.
If you do want to go ahead with this in Oozie, you can't avoid generating a workflow with 300 actions. I would do it as 300 actions rather than 300 calls to sub-workflows which each call one action, else you're going to generate even more overhead. You can either create this file manually, or preferably write some code to generate the Oozie workflow XML file given a list of tables. The latter is more flexible as it allows tables to be included or excluded on a per-run basis.
But as I initially said, I'd stick to Bash for this one unless you have a very very good reason.

Answer 2

My suggestion would be to create workflows each for 50 table import. So you have 6 of them like that. Call all the 6 workflows as sub workflows from a master or parent workflow. By this way we can have the control at one point and it will be easy to schedule a single workflow.