How to pass EMR cluster ID in the CloudWatch Alarm - yaml

Question

I am trying to create a SNS alarm for my EMR cluster so when EMR cluster is failed i should get notified .
But my issue is i am not able to pass Cluster ID as JobFlowId in the CloudWatch Alarm .
I am create all resources using CloudFomartion Templet .
When i use REF to refer Cluster Id i get below error .
Error
Template validation error: Template format error: Unresolved resource dependencies [FinancialLineItemEmrCluster] in the Resources block of the template
Here is my templet .
I am having issue specially in at JobFlowId while creating the resources EMRAlarm
AWSTemplateFormatVersion: '2010-09-09'
Description: TRF SDI Optmization Full File Creation
Parameters:
AppName:
Default: trfsdioptmization
Description: trfsdioptmization.
Type: String
Environment:
Type: String
Default: nonprod
FinancialIdentifier:
Type: String
Default: 123456789
ApplicationAssetInsightId:
Type: String
Default: 12345678
EnvironmentType:
Type: String
AllowedValues:
- "prod"
- "PRE-PRODUCTION"
- "QUALITY ASSURANCE"
- "nonprod"
Default: nonprod
ResourceOwner:
Type: String
Default: sudarshan.kumar#abcd.com
EnvironmentPhase:
Type: String
Default: nonprod
RegionAbbreviation:
Default: us-east-1
Description: Region Abbreviation e.g. us-east-1 for us-east
Type: String
Resources:
TRFSDIFullfileGeneration:
Type: "AWS::DataPipeline::Pipeline"
#DeletionPolicy: Retain
Properties:
Name: !Sub "${ApplicationAssetInsightId}-tr-fr-${EnvironmentPhase}-${RegionAbbreviation}-${AppName}-DataPipeline"
Description: "Pipeline to create full file for TRFSDI full file Optmization"
Activate: false
PipelineObjects:
-
Id: "FinancialLineItemActivity"
Name: "FinancialLineItemActivity"
Fields:
-
Key: "type"
StringValue: "EmrActivity"
-
Key: "runsOn"
RefValue: "FinancialLineItemEmrCluster"
-
Key: "step"
StringValue: "command-runner.jar,spark-submit,--master,yarn-cluster,--deploy-mode,cluster,--class,start.EntryFileCreation,s3://205147-trf-fr-nonprdo-us-east-1-trfsdioptmization/AJAR/SparkJob-0.1-jar-with-dependencies.jar,FinancialLineItem"
Id: "Default"
Name: "Default"
Fields:
-
Key: "type"
StringValue: "Default"
-
Key: "scheduleType"
StringValue: "ONDEMAND"
-
Key: "failureAndRerunMode"
StringValue: "CASCADE"
-
Key: "role"
StringValue: "DataPipelineDefaultRole"
-
Key: "resourceRole"
StringValue: "DataPipelineDefaultResourceRole"
-
Key: "pipelineLogUri"
StringValue: "s3://205147-tr-fr-nonprod-us-east-1-trfsdioptmization/EMRLOGS"
-
Id: "FinancialLineItemEmrCluster"
Name: "FinancialLineItemEmrCluster"
Fields:
-
Key: "terminateAfter"
StringValue: "30 Minutes"
-
Key: "releaseLabel"
StringValue: "emr-5.9.0"
-
Key: "masterInstanceType"
StringValue: "m3.xlarge"
-
Key: "coreInstanceType"
StringValue: "m3.2xlarge"
-
Key: "coreInstanceCount"
StringValue: "2"
-
Key: "type"
StringValue: "EmrCluster"
-
Key: "applications"
StringValue: "spark"
-
Key: "subnetId"
StringValue: "subnet-86febcab"
-
Key: "onSuccess"
RefValue: "FinancialLineItem_Success"
-
Key: "onFail"
RefValue: "FinancialLineItem_Fail"
EMRAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: "Raise alarm if apps running on EMR cluster is killed"
Namespace: AWS/ElasticMapReduce
MetricName: AppsKilled
Dimensions:
- Name: 205147-TRFSDIOPTmization
JobFlowId: !Ref FinancialLineItemEmrCluster
Statistic: Average
Period: 15
EvaluationPeriods: '1'
ComparisonOperator: GreaterThanOrEqualToThreshold
Threshold: 1
AlarmActions:
- "AWSTRF_HEALTH"

Answer 1

There is no resource of the type AWS::EMR::Cluster in your template.
You are referencing something called FinancialLineItemEmrCluster in your CloudWatch Alarm. From the context, I am assuming you are trying to reference a EMR job. However, since you have no parameter or resource in your template named FinancialLineItemEmrCluster, you can not access is.
EMR cluster IDs generally look like this: j-1ABCD123AB1A. You have several options:
If this cluster is in another template, you could create a CloudFormation export in that stack, and use !ImportValue in your template to import it.
Alternatively, you could use a Parameter in your template and pass the ClusterId that way. Add this as a parameter:
Example:
FinancialLineItemEmrCluster:
Description: 'Your EMR cluster id. Example: j-1ABCD123AB1A'
Type: String
A third alternative is to just hardcode it into your template.
In any case, you can not refer directly to a resource in another stack.
If you have not createt an EMR::Cluster at all, and have no Cluster Id, you need to create one first. You could add one to your Template using the AWS::EMR::Cluster resource:
FinancialLineItemEmrCluster:
Type: AWS::EMR::Cluster
Properties:
Instances:
MasterInstanceGroup:
InstanceCount: 1
InstanceType: "m3.xlarge"
Market: "ON_DEMAND"
Name: "Master"
CoreInstanceGroup:
InstanceCount: 2
InstanceType: "m3.xlarge"
Market: "ON_DEMAND"
Name: "Core"
TerminationProtected: true
Name: "TestCluster"
JobFlowRole: "EMR_EC2_DefaultRole"
ServiceRole: "EMR_DefaultRole"
ReleaseLabel: "emr-4.2.0"