How do you run an EmrActivity on an existing EMR cluster? - amazon-data-pipeline

Question

Is there a way to run an EmrActivity in AWS Data Pipeline on an existing cluster? We currently are using Data Pipeline to run jobs in AWS EMR using EmrCluster and EmrActivity but we'd like to have all pipelines run on the same cluster. I've tried reading the documentation and building a pipeline in architect but I can't seem to find a way to do anything but create a cluster and run jobs on it. There doesn't seem to be a way to define a new pipeline which uses an existing cluster. If there is how would I do it? We're currently using CloudFormation to create our pipelines so if possible an example using CloudFormation would be preferable but I'll take what I can get.

Answer 1

Yes it is possible.
Launch your EMR cluster
Start TaskRunner on the master instance with the option --workerGroup=name-of-the-worker-group
In the activities of your pipeline don't specify runsOn parameter, pass your worker group instead.
Here is an example of the activity with such parameter defined using CloudFormation:
...
{
"Id": "S3ToRedshiftCopyActivity",
"Name": "S3ToRedshiftCopyActivity",
"Fields": [
{
"Key": "type",
"StringValue": "RedshiftCopyActivity"
},
{
"Key": "workerGroup",
"StringValue": "name-of-the-worker-group"
},
{
"Key": "insertMode",
"StringValue": "#{myInsertMode}"
},
{
"Key": "commandOptions",
"StringValue": "FORMAT CSV"
},
{
"Key": "dependsOn",
"RefValue": "RedshiftTableCreateActivity"
},
{
"Key": "input",
"RefValue": "S3StagingDataNode"
},
{
"Key": "output",
"RefValue": "DestRedshiftTable"
}
]
}
...
You can find detailed documentation how to do that here:
http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-how-task-runner-user-managed.html