Ive found 0 examples of this
I have this template:
resources:
- name: resource-name
type: 'gcp-types/cloudfunctions-v1:projects.locations.functions'
properties:
labels:
- key: testlabel1
value: testlabel1value
- key: testlabel2
value: testlabel2value
parent: projects/sdfsfsdf/locations/us-central1
location: us-central1
function: function-name
sourceArchiveUrl: 'gs://sdfsfsdf/b50d36e265ec71d457bb7ba5cc13e44c.zip'
environmentVariables:
TEST_ENV_VAR: 'zzzzzzzzz'
entryPoint: handler
httpsTrigger: {}
timeout: 60s
availableMemoryMb: 256
runtime: nodejs8
which produces this error:
ERROR: (gcloud.deployment-manager.deployments.create) Error in Operation
- code: CONDITION_NOT_MET
location: /deployments/test-templates/resources/resource-name->$.properties
message: |
error: instance type (array) does not match any allowed primitive type (allowed: ["object"])
level: "error"
schema: {"loadingURI":"#","pointer":"/create/properties/labels"}
instance: {"pointer":"/labels"}
domain: "validation"
keyword: "type"
found: "array"
expected: ["object"]
While there should have been an actual example of this in the docs i was being dumb. this is the correct format
resources:
- name: resource-name
type: 'gcp-types/cloudfunctions-v1:projects.locations.functions'
properties:
labels:
testlabel1: testlabel1value
testlabel2: testlabel2value
Related
I'm assigned to a filebeat plugin of our product. I wrote a IngestPipeline and de default mapping is in various ways unfortunately so i wanted to change this with the _meta/fields.yml.
I configured the yml file accordingly to:
https://www.elastic.co/guide/en/beats/devguide/7.8/filebeat-modules-devguide.html#_metafields_yml_2
https://www.elastic.co/guide/en/beats/devguide/current/event-fields-yml.html
https://github.com/elastic/kibana/issues/82273
My yml file looks like this:
- key: pac
title: "pac"
description: Description missing
pac Module
fields:
- name: pac.log
type: group
description: Description missing
fields:
- name: deo
type: group
description: Description missing
fields:
- name: duration
type: long
description: Duration of the deo
- name: category
type: keyword
description: Category of the deo
- name: owner
type: keyword
description: User of the deo
- name: version
type: float
description: Version of the deo
- name: name
type: keyword
description: Name of the deo
- name: reference
type: double
description: Referencenumber of the deo
- name: state
type: keyword
description: State of the deo
- name: status
type: keyword
description: Status of the deo
- name: trigger
type: group
description: Description missing
fields:
- name: category
type: keyword
description: Category of the deo-trigger
- name: name
type: text
description: Name of the deo-trigger
- name: path
type: text
description: Full-Text of the trigger properties
- name: provider
type: keyword
description: Supplier of the deo-trigger
- name: wiring
type: group
description: Description missing
fields:
- name: async
type: boolean
description: If deos wired asynchronously
- name: deoId
type: keyword
description: Identification number of the deo
- name: execute
type: boolean
description: If deo is executed or not
- name: owner
type: keyword
description: User of the deo
- name: shared
type: boolean
description: Deo was shared
- name: stopOnError
type: boolean
description: If deo stopped on Error
- name: do
type: group
description: Description missing
fields:
- name: name
type: keyword
description: Name of the do task
- name: state
type: group
description: Description missing
fields:
- name: from
type: keyword
description: State from which the do task was used
- name: to
type: keyword
description: State to which the do task was switched
- name: esa
type: group
description: Description missing
fields:
- name: connection
type: keyword
description: Connection status of the ESA
- name: name
type: keyword
description: Name of the ESA
- name: state
type: group
description: Description missing
fields:
- name: from
type: keyword
description: State from which the ESA Connection was established
- name: to
type: keyword
description: State to which the ESA Connection was established
- name: monitor
type: group
description: Description missing
fields:
- name: heap
type: group
description: Description missing
fields:
- name: bytes
type: long
description: Used bytes of the heap
- name: pct
type: float
description: Percentage of the maximum available bytes
- name: heapgc
type: group
description: Description missing
fields:
- name: bytes
type: long
description: Used bytes of the heapgc
- name: pct
type: float
description: Percentage of the maximum available bytes
- name: service
type: group
description: Description missing
fields:
- name: class
type: keyword
description: Class of the service
- name: duration
type: long
description: How long the service call took
- name: name
type: keyword
description: Name of the service call
- name: operation
type: keyword
description: Operation type of the service call
- name: success
type: boolean
description: Was the service call successfull or not
- name: system
type: group
description: Description missing
fields:
- name: category
type: keyword
description: Category of the System
- name: priority
type: keyword
description: Priority of the System
- name: monitor
type: group
description: Description missing
fields:
- name: cpu
type: group
description: Description missing
fields:
- name: pct
type: float
description: Percentage of the cpu usage
- name: wiring
type: text
description: Description missing
- name: meta
type: text
description: Description missing
- name: tags
type: keyword
description: Description missing
- name: timestamp
type: date
description: Description missing
- name: level
type: keyword
description: Description missing
- name: logger
type: keyword
description: Description missing
I would like to have the field types as configured in the fields.yml file. But at first no mapping is available and if i start to send logs the mapping for my fields is created automatically with unfortunate types.
The yaml file works as it is in the description above. My Problem was that my collegue from whom i inherited the project didn't use the filebeat build script. So the uploaded fields.yml file wasn't extended by my fields. I reconfigured the fields.yaml path inside the filebeat.yml configuration file and now it works.
I have the following serverless yaml that I'm using to try to deploy my first step function:
org: bizrob
app: flexipod-2-queue
service: flexipod-2-queue
frameworkVersion: "2 || 3"
package:
exclude:
# list of biggest modules that are in devdepenedecies and similar
- node_modules/aws-sdk/**
- node_modules/serverless-domain-manager/**
- node_modules/#serverless
- node_modules/serverless
- node_modules/java-invoke-local
- node_modules/tabtab
- node_modules/snappy
custom:
region: eu-west-1
provider:
name: aws
runtime: nodejs14.x
plugins:
- serverless-step-functions
functions:
GetConfigDbConnection:
handler: flexipod-2-queue/dbConfig.getConfigDbConnection
environment:
REGION: ${self:custom.region}
GetConfigRec:
handler: flexipod-2-queue/dbConfig.getConfigRec
environment:
REGION: ${self:custom.region}
GetSelectQueries:
handler: flexipod-2-queue/dbConfig.getSelectQueries
environment:
REGION: ${self:custom.region}
PullSqlSvr:
handler: flexipod-2-queue/pullSqlSvrData.pullSqlSvr
environment:
REGION: ${self:custom.region}
API_VERSION_S3: "2006-03-01"
API_VERSION_SQS: "2012-11-05"
SQS_QUEUE_URL: !Ref "MyQueue"
SendToDataLake:
handler: queue-2-datalake/sendToDataLake.sendBatchToQueue
environment:
REGION: ${self:custom.region}
API_VERSION_S3: "2006-03-01"
API_VERSION_SQS: "2012-11-05"
stepFunctions:
stateMachines:
flexipodFlow:
name: flexipodFlow
definition:
StartAt: GetConfigDbConnection
States:
GetConfigDbConnection:
Type: Task
Resource:
Fn::GetAtt: [GetConfigDbConnection, Arn]
Next: GetConfigRec
GetConfigRec:
Type: Task
Resource:
Fn::GetAtt: [GetConfigRec, Arn]
Next: GetSelectQueries
GetSelectQueries:
Type: Task
Resource:
Fn::GetAtt: [GetSelectQueries, Arn]
ResultPath: $.queries
Next: Map
Map:
Type: Map
ItemsPath: $.queries
MaxConcurrency: 2
Next: Final State
Iterator:
StartAt: PullSql
States:
PullSql:
Type: Task
Resource:
Fn::GetAtt: [PullSqlSvr, Arn]
Final State:
Type: Pass
End: true
resources:
Resources:
MyQueue:
Type: "AWS::SQS::Queue"
Properties:
QueueName: "flexipod"
After running serverless deploy, I see get the following error in the vscode terminal:
Error:
TypeError: Cannot read property 'match' of null
at C:\GitBizTalkers\OLD_Wck_Flexipod\node_modules\serverless-step-functions\lib\deploy\stepFunctions\compileIamRole.js:472:61
at arrayMap (C:\GitBizTalkers\OLD_Wck_Flexipod\node_modules\lodash\lodash.js:653:23)
at map (C:\GitBizTalkers\OLD_Wck_Flexipod\node_modules\lodash\lodash.js:9622:14)
at Function.flatMap (C:\GitBizTalkers\OLD_Wck_Flexipod\node_modules\lodash\lodash.js:9325:26) at ServerlessStepFunctions.getIamPermissions (C:\GitBizTalkers\OLD_Wck_Flexipod\node_modules\serverless-step-functions\lib\deploy\stepFunctions\compileIamRole.js:413:12)
at C:\GitBizTalkers\OLD_Wck_Flexipod\node_modules\serverless-step-functions\lib\deploy\stepFunctions\compileIamRole.js:522:56
at Array.forEach (<anonymous>)
at ServerlessStepFunctions.compileIamRole (C:\GitBizTalkers\OLD_Wck_Flexipod\node_modules\serverless-step-functions\lib\deploy\stepFunctions\compileIamRole.js:511:32)
at ServerlessStepFunctions.tryCatcher (C:\GitBizTalkers\OLD_Wck_Flexipod\node_modules\bluebird\js\release\util.js:16:23)
at Promise._settlePromiseFromHandler (C:\GitBizTalkers\OLD_Wck_Flexipod\node_modules\bluebird\js\release\promise.js:547:31)
at Promise._settlePromise (C:\GitBizTalkers\OLD_Wck_Flexipod\node_modules\bluebird\js\release\promise.js:604:18)
at Promise._settlePromiseCtx (C:\GitBizTalkers\OLD_Wck_Flexipod\node_modules\bluebird\js\release\promise.js:641:10)
at _drainQueueStep (C:\GitBizTalkers\OLD_Wck_Flexipod\node_modules\bluebird\js\release\async.js:97:12)
at _drainQueue (C:\GitBizTalkers\OLD_Wck_Flexipod\node_modules\bluebird\js\release\async.js:86:9)
at Async._drainQueues (C:\GitBizTalkers\OLD_Wck_Flexipod\node_modules\bluebird\js\release\async.js:102:5)
at Immediate.Async.drainQueues [as _onImmediate] (C:\GitBizTalkers\OLD_Wck_Flexipod\node_modules\bluebird\js\release\async.js:15:14)
at processImmediate (node:internal/timers:464:21)
Anyone see what I've done wrong?
Problem was due to yaml formatting. Line 192
Fn::GetAtt: [PullSqlSvr, Arn]
This needed an extra tab to indent below "Resource:"
I am getting the following error after running CLI command aws cloudformation deploy (after sam package)
"Failed to create the changeset: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state Status: FAILED. Reason: Invalid template property or properties [MyApi]"
This is the template. I cannot get information about which is the invalid property. Is this possible? Else what is wrong with this template?
AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31
Resources:
HelloFunction:
Type: AWS::Serverless::Function
Properties:`enter code here`
Handler: main
Runtime: go1.x
Events:
GetEvent:
Type: Api
Properties:
Path: /
Method: post
#RestApiId: !Ref ApiGateway1
LambdaInvokePermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !GetAtt
- HelloFunction
- Arn
Action: 'lambda:InvokeFunction'
Principal: apigateway.amazonaws.com
SourceAccount: !Ref 'AWS::AccountId'
MyApi:
Type: AWS::Serverless::Api
Properties:
StageName: default
EndpointConfiguration: REGIONAL
DefinitionBody:
swagger: "2.0"
info:
title: "TestAPI"
paths:
/:
get:
# parameters:
# - name: "id"
# in: "query"
# required: true
# type: "string"
# x-amazon-apigateway-request-validator: "Validate query string parameters and\
# \ headers"
x-amazon-apigateway-integration:
uri:
Fn::Sub: arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${HelloFunction.Arn}/invocations
responses: {}
httpMethod: "POST"
type: "aws_proxy"
Outputs:
FunctioArn:
Value: !GetAtt HelloFunction.Arn
Export:
Name: HelloFunctionArn
There is an indentation error at MyApi. Use the following:
AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31
Resources:
HelloFunction:
Type: AWS::Serverless::Function
Properties:`enter code here`
Handler: main
Runtime: go1.x
Events:
GetEvent:
Type: Api
Properties:
Path: /
Method: post
#RestApiId: !Ref ApiGateway1
LambdaInvokePermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !GetAtt
- HelloFunction
- Arn
Action: 'lambda:InvokeFunction'
Principal: apigateway.amazonaws.com
SourceAccount: !Ref 'AWS::AccountId'
MyApi:
Type: AWS::Serverless::Api
Properties:
StageName: default
EndpointConfiguration: REGIONAL
DefinitionBody:
swagger: "2.0"
info:
title: "TestAPI"
paths:
/:
get:
# parameters:
# - name: "id"
# in: "query"
# required: true
# type: "string"
# x-amazon-apigateway-request-validator: "Validate query string parameters and\
# \ headers"
x-amazon-apigateway-integration:
uri:
Fn::Sub: arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${HelloFunction.Arn}/invocations
responses: {}
httpMethod: "POST"
type: "aws_proxy"
Outputs:
FunctioArn:
Value: !GetAtt HelloFunction.Arn
Export:
Name: HelloFunctionArn
I am trying to create a SNS alarm for my EMR cluster so when EMR cluster is failed i should get notified .
But my issue is i am not able to pass Cluster ID as JobFlowId in the CloudWatch Alarm .
I am create all resources using CloudFomartion Templet .
When i use REF to refer Cluster Id i get below error .
Error
Template validation error: Template format error: Unresolved resource dependencies [FinancialLineItemEmrCluster] in the Resources block of the template
Here is my templet .
I am having issue specially in at JobFlowId while creating the resources EMRAlarm
AWSTemplateFormatVersion: '2010-09-09'
Description: TRF SDI Optmization Full File Creation
Parameters:
AppName:
Default: trfsdioptmization
Description: trfsdioptmization.
Type: String
Environment:
Type: String
Default: nonprod
FinancialIdentifier:
Type: String
Default: 123456789
ApplicationAssetInsightId:
Type: String
Default: 12345678
EnvironmentType:
Type: String
AllowedValues:
- "prod"
- "PRE-PRODUCTION"
- "QUALITY ASSURANCE"
- "nonprod"
Default: nonprod
ResourceOwner:
Type: String
Default: sudarshan.kumar#abcd.com
EnvironmentPhase:
Type: String
Default: nonprod
RegionAbbreviation:
Default: us-east-1
Description: Region Abbreviation e.g. us-east-1 for us-east
Type: String
Resources:
TRFSDIFullfileGeneration:
Type: "AWS::DataPipeline::Pipeline"
#DeletionPolicy: Retain
Properties:
Name: !Sub "${ApplicationAssetInsightId}-tr-fr-${EnvironmentPhase}-${RegionAbbreviation}-${AppName}-DataPipeline"
Description: "Pipeline to create full file for TRFSDI full file Optmization"
Activate: false
PipelineObjects:
-
Id: "FinancialLineItemActivity"
Name: "FinancialLineItemActivity"
Fields:
-
Key: "type"
StringValue: "EmrActivity"
-
Key: "runsOn"
RefValue: "FinancialLineItemEmrCluster"
-
Key: "step"
StringValue: "command-runner.jar,spark-submit,--master,yarn-cluster,--deploy-mode,cluster,--class,start.EntryFileCreation,s3://205147-trf-fr-nonprdo-us-east-1-trfsdioptmization/AJAR/SparkJob-0.1-jar-with-dependencies.jar,FinancialLineItem"
Id: "Default"
Name: "Default"
Fields:
-
Key: "type"
StringValue: "Default"
-
Key: "scheduleType"
StringValue: "ONDEMAND"
-
Key: "failureAndRerunMode"
StringValue: "CASCADE"
-
Key: "role"
StringValue: "DataPipelineDefaultRole"
-
Key: "resourceRole"
StringValue: "DataPipelineDefaultResourceRole"
-
Key: "pipelineLogUri"
StringValue: "s3://205147-tr-fr-nonprod-us-east-1-trfsdioptmization/EMRLOGS"
-
Id: "FinancialLineItemEmrCluster"
Name: "FinancialLineItemEmrCluster"
Fields:
-
Key: "terminateAfter"
StringValue: "30 Minutes"
-
Key: "releaseLabel"
StringValue: "emr-5.9.0"
-
Key: "masterInstanceType"
StringValue: "m3.xlarge"
-
Key: "coreInstanceType"
StringValue: "m3.2xlarge"
-
Key: "coreInstanceCount"
StringValue: "2"
-
Key: "type"
StringValue: "EmrCluster"
-
Key: "applications"
StringValue: "spark"
-
Key: "subnetId"
StringValue: "subnet-86febcab"
-
Key: "onSuccess"
RefValue: "FinancialLineItem_Success"
-
Key: "onFail"
RefValue: "FinancialLineItem_Fail"
EMRAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: "Raise alarm if apps running on EMR cluster is killed"
Namespace: AWS/ElasticMapReduce
MetricName: AppsKilled
Dimensions:
- Name: 205147-TRFSDIOPTmization
JobFlowId: !Ref FinancialLineItemEmrCluster
Statistic: Average
Period: 15
EvaluationPeriods: '1'
ComparisonOperator: GreaterThanOrEqualToThreshold
Threshold: 1
AlarmActions:
- "AWSTRF_HEALTH"
There is no resource of the type AWS::EMR::Cluster in your template.
You are referencing something called FinancialLineItemEmrCluster in your CloudWatch Alarm. From the context, I am assuming you are trying to reference a EMR job. However, since you have no parameter or resource in your template named FinancialLineItemEmrCluster, you can not access is.
EMR cluster IDs generally look like this: j-1ABCD123AB1A. You have several options:
If this cluster is in another template, you could create a CloudFormation export in that stack, and use !ImportValue in your template to import it.
Alternatively, you could use a Parameter in your template and pass the ClusterId that way. Add this as a parameter:
Example:
FinancialLineItemEmrCluster:
Description: 'Your EMR cluster id. Example: j-1ABCD123AB1A'
Type: String
A third alternative is to just hardcode it into your template.
In any case, you can not refer directly to a resource in another stack.
If you have not createt an EMR::Cluster at all, and have no Cluster Id, you need to create one first. You could add one to your Template using the AWS::EMR::Cluster resource:
FinancialLineItemEmrCluster:
Type: AWS::EMR::Cluster
Properties:
Instances:
MasterInstanceGroup:
InstanceCount: 1
InstanceType: "m3.xlarge"
Market: "ON_DEMAND"
Name: "Master"
CoreInstanceGroup:
InstanceCount: 2
InstanceType: "m3.xlarge"
Market: "ON_DEMAND"
Name: "Core"
TerminationProtected: true
Name: "TestCluster"
JobFlowRole: "EMR_EC2_DefaultRole"
ServiceRole: "EMR_DefaultRole"
ReleaseLabel: "emr-4.2.0"
Cannot create a google deployment manager runtime config variable
resources:
- name: star-config
type: runtimeconfig.v1beta1.config
properties:
name: star-config
- name: igurl_variable
type: runtimeconfig.v1beta1.variable
properties:
name: igurl_variable
value: 'trek'
parent: $(ref.star-config.name)
I checked the logs and I see that the status is set to bad_request when I create the above deployment.
Audit log
status: {
message: "BAD_REQUEST"
}
What could be the reason for the error ?
You should try the with the properties fields as in the official documentation for both the config and variable resources.
The resource file should be something like:
resources:
- name: star-config
type: runtimeconfig.v1beta1.config
properties:
config: star-config
- name: igurl_variable
type: runtimeconfig.v1beta1.variable
properties:
variable: igurl_variable
text: 'trek'
parent: $(ref.star-config.name)