Oozie custom asynchronous action - hadoop

Question

I have a problem implementing a custom asynchronous action in Oozie. My class extends from ActionExecutor, and overwrites the methods initActionType, start, end, check, kill and isCompleted.
In the start method, i want to to start a YARN job, that is implemented through my BiohadoopClient class. To make the call asynchronous, i wrapped the client.run() method in a Callable:
public void start(final Context context, final WorkflowAction action) {
...
Callable<String> biohadoop = new Callable<String>() {
BiohadoopClient client = new BiohadoopClient();
client.run();
}
// submit callable to executor
executor.submit(biohadoop);
// set the start data, according to https://oozie.apache.org/docs/4.0.1/DG_CustomActionExecutor.html
context.setStartData(externalId, callBackUrl, callBackUrl);
...
}
This works fine, and for example when I use my custom action in a fork/join manner, the execution of the actions runs in parallel.
Now, the problem is, that Oozie remains in a RUNNING state for this actions. It seems impossible to change that to a completed state. The check() method is never called by Oozie, the same is true for the end() method. It doesn't help to set the context.setExternalStatus(), context.setExecutionData() and context.setEndData() manually in the Callable (after the client.run() has finished). I tried also to queue manually an ActionEndXCommand, but without luck.
When I wait in the start() method for the Callable to complete, the state gets updated correctly, but the execution in fork/join isn't parallel anymore (which seem logic, as the execution waits for the Callable to complete).
How external clients notify Oozie workflow with HTTP callback didn't help, as using the callback seems to change nothing (well, I can see that it happened in the log files, but beside from that, nothing...). Also, the answer mentioned, that the SSH action runs asynchronously, but I haven't found out how this is done. There is some wrapping inside a Callable, but at the end, the call() method of the Callable is invoked directly (no submission to an Executor).
So far I haven't found any example howto write an asynchronous custom action. Can anybody please help me?
Thanks
Edit
Here are the implementations of initActionType(), start(), check(), end(), the callable implementation can be found inside the start() action.
The callable is submitted to an executor in the start() action, after which its shutdown() method is invoked - so the executor shuts down after the Callable has finished. As next step, context.setStartData(externalId, callBackUrl, callBackUrl) is invoked.
private final AtomicBoolean finished = new AtomicBoolean(false);
public void initActionType() {
super.initActionType();
log.info("initActionType() invoked");
}
public void start(final Context context, final WorkflowAction action)
throws ActionExecutorException {
log.info("start() invoked");
// Get parameters from Node configuration
final String parameter = getParameters(action.getConf());
Callable<String> biohadoop = new Callable<String>() {
#Override
public String call() throws Exception {
log.info("Starting Biohadoop");
// No difference if check() is called manually
// or if the next line is commented out
check(context, action);
BiohadoopClient client = new BiohadoopClient();
client.run(parameter);
log.info("Biohadoop finished");
finished.set(true);
// No difference if check() is called manually
// or if the next line is commented out
check(context, action);
return null;
}
};
ExecutorService executor = Executors.newCachedThreadPool();
biohadoopResult = executor.submit(biohadoop);
executor.shutdown();
String externalId = action.getId();
String callBackUrl = context.getCallbackUrl("finished");
context.setStartData(externalId, callBackUrl, callBackUrl);
}
public void check(final Context context, final WorkflowAction action)
throws ActionExecutorException {
// finished is an AtomicBoolean, that is set to true,
// after Biohadoop has finished (see implementation of Callable)
if (finished.get()) {
log.info("check(Context, WorkflowAction) invoked -
Callable has finished");
context.setExternalStatus(Status.OK.toString());
context.setExecutionData(Status.OK.toString(), null);
} else {
log.info("check(Context, WorkflowAction) invoked");
context.setExternalStatus(Status.RUNNING.toString());
}
}
public void end(Context context, WorkflowAction action)
throws ActionExecutorException {
log.info("end(Context, WorkflowAction) invoked");
context.setEndData(Status.OK, Status.OK.toString());
}

Answer 1

One thing - I can see you are shutting down the executor right after you have submitted the job - executor.shutdown();. That might be causing the issue. Could you please try moving this statement to the end() method instead?

Answer 2

In the end I didn't find a "real" solution to the problem. The solution that worked for me was to implement an action, that invokes the Biohadoop instances in parallel using the Java Executor framework. After the invokation, I wait (still inside the action) for the threads to finish