Getting an error when trying to run a pipeline.
/devops/templates/app-deployment-template.yml (Line: 50, Col: 27): Unable to convert from Array to String. Value: Array
This is the parameter in my yaml file that I'm trying to pass further down into an ARM template. At the top level, this is a string array with elements such as UKSouth, NorthEurope, etc.
parameters:
- name: locations
type: object
default: []
# other parameters
# other jobs and tasks
- task: AzureResourceManagerTemplateDeployment#3
displayName: 'Deploy Azure Core Infrastructure'
inputs:
deploymentScope: 'Resource Group'
azureResourceManagerConnection: '${{parameters.subscriptionName}}'
action: 'Create Or Update Resource Group'
resourceGroupName: '${{parameters.environmentName}}-${{parameters.resourceGroupName}}'
location: 'North Europe'
templateLocation: 'Linked artifact'
csmFile: '$(Pipeline.Workspace)/artifacts/infrastructure/appserviceplan.json'
csmParametersFile: '$(Pipeline.Workspace)/artifacts/infrastructure/appserviceplan.parameters.json'
deploymentMode: 'Incremental'
overrideParameters: '-name ${{parameters.environmentName}}-${{parameters.resourceGroupName}} -locations ${{parameters.locations}}'
Related
I have a strange problem that I can't seem to get my head around. I am trying to define some variables for use as part of the job that will deploy bicep files via Azure CLI and execute PowerShell tasks.
I get this validation error when I try and execute the pipeline: While parsing a block mapping, did not find expected key
The line that it refers to is: - name: managementResourceDNSPrivateResolverName
On the research that I have done on this problem, it sounds like an indentation problem but on the face of it, it seems to look fine.
jobs:
- job: 'Deploy_Management_Resources'
pool:
vmImage: ${{ parameters.buildAgent }}
variables:
- name: managementResourceDNSPrivateResolverName
value: 'acme-$[ lower(parameters['environmentCode']) ]-$[ lower(variables['resourceLocationShort']) ]-private-dns-resolver'
- name: managementResourceGroupManagement
value: 'acme-infrastructure-rg-management'
- name: managementResourceRouteTableName
value: 'acme-$[ lower(variables['subscriptionCode']) ]-$[ lower(variables['resourceLocationShort']) ]-route-table'
- name: managementResourceVirtualNetworkName
value: 'acme-$[ lower(variables['subscriptionCode']) ]-$[ lower(variables['resourceLocationShort']) ]-vnet-internal-mng'
Thanks!
The error message ...parsing a block mapping, did not find expected key is usually a side-effect of malformed yaml. You'll see if often with variables if you have mixed formats of arrays and property elements
variables: # an array of objects
# variable group reference object
- group: myvariablegroup
# variable template reference object
- template: my-variables.yml
# variable object
- name: myVariable
value: 'value1'
# variable shorthand syntax
myVariable: 'value1' # this fails because it's a property instead of an array element
While it doesn't appear that the sample you've provided is malformed, I am curious about the use of $[ ] which is a runtime expression. The expression $[ lower(parameters['environmentcode']) ] refers to parameters which is are only available at compile time.
Change:
$[ lower(parameters['environmentCode']) ] to ${{ lower(parameters.environmentCode) }}
I have am trying to deploy an aws lambda function using the SAM cli. I have some layers defined in the sam template. Testing locally using sam local start-api works quite well. The but deploying using the sam deploy --guided command throws the following error
Error: Failed to create changeset for the stack: sam-app, ex: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state: For expression "Status" we matched expected path: "FAILED" Status: FAILED. Reason: Template format error: Unresolved resource dependencies [arn:aws:lambda:us-west-1:338231645678:layer:ffmpeg:1] in the Resources block of the template
The SAM template is as follows
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
video-processor-functions
Functions to generate gif and thumbnail from uploaded videos
# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
Function:
Timeout: 3
Tracing: Active
Resources:
VideoProcessorFunctions:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: src/
Handler: app.lambdaHandler
Runtime: nodejs14.x
# timeout in seconds - 2 minutes
Timeout: 120
Layers:
- !Ref VideoProcessorDepLayer
- !Ref arn:aws:lambda:us-west-1:338231645678:layer:ffmpeg:1
Architectures:
- x86_64
Events:
HelloWorld:
Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
Properties:
Path: /hello
Method: get
VideoProcessorDepLayer:
Type: AWS::Serverless::LayerVersion
Properties:
LayerName: mh-video-processor-dependencies
Description: Dependencies for sam app [video-processor-functions]
ContentUri: dependencies/
CompatibleRuntimes:
- nodejs14.17
LicenseInfo: 'MIT'
RetentionPolicy: Retain
Outputs:
# ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
# Find out more about other implicit resources you can reference within SAM
# https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
HelloWorldApi:
Description: "API Gateway endpoint URL for Prod stage for Hello World function"
Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
VideoProcessorFunctions:
Description: "Generate GIF and Thumnail from Video"
Value: !GetAtt VideoProcessorFunctions.Arn
VideoProcessorFunctionsIamRole:
Description: "Implicit IAM Role created for MH Video Processor function"
Value: !GetAtt VideoProcessorFunctionsRole.Arn
Any ideas what i'm doing wrong?
You are trying to reference a logical id that doesn’t exist.
Just put the layer's ARN of 'ffmpeg':
Layers:
- !Ref VideoProcessorDepLayer
- arn:aws:lambda:us-west-1:338231645678:layer:ffmpeg:1
I copied Semaphore's config of GoReleaser and saved it as .semaphore/semaphore.yml but when I run it on Semaphore, I can see:
Unprocessable YAML file.
Error: {{:throw, {:yamerl_exception, [{:yamerl_parsing_error, :error, 'Block mapping value not allowed here', 30, 20, :block_mapping_value_not_allowed, :undefined, []}]}}, "# .semaphore/semaphore.yml.\nversion: v1.0\nname: Build\nagent:\n machine:\n type: e1-standard-2\n os_image: ubuntu1804\n\nblocks:\n - name: \"Test\"\n task:\n prologue:\n commands:\n # set go version\n - sem-version go 1.11\n - \"export GOPATH=~/go\"\n - \"export PATH=/home/semaphore/go/bin:$PATH\"\n - checkout\n\n jobs:\n - name: \"Lint\"\n commands:\n - go get ./...\n - go test ./...\n\n# On Semaphore 2.0 deployment and delivery is managed with promotions,\n# which may be automatic or manual and optionally depend on conditions.\npromotions:\n - name: Release\n pipeline_file: goreleaser.yml\n auto_promote_on:\n - result: passed\n branch:\n - \"^refs/tags/v*\"\n"}
How can I fix it?
In this last part:
promotions:
- name: Release
pipeline_file: goreleaser.yml
auto_promote_on:
- result: passed
branch:
- "^refs/tags/v*"
The indentation of pipeline_file and auto_promote_on is wrong – both need to be at the same level as name.
My question: How can Codepipeline read the value of a field in a json file which is in SourceCodeArtifact?
I have Gthub repo that contains a file imageManifest.json which looks like this:
{
"image_id": "docker.pkg.github.com/my-org/my-repo/my-app",
"image_version": "1.0.1"
}
I want my AWS Codepipeline Source stage to be able to read the value of image_version from imageManifest.json and pass it as a parameter to a CloudFormation action in a subsequent stage of my pipeline.
For reference, here is my source stage.
Stages:
- Name: GitHubSource
Actions:
- Name: SourceAction
ActionTypeId:
Category: Source
Owner: ThirdParty
Version: '1'
Provider: GitHub
OutputArtifacts:
- Name: SourceCodeArtifact
Configuration:
Owner: !Ref GitHubOwner
Repo: !Ref GitHubRepo
OAuthToken: !Ref GitHubAuthToken
And here is my deploy stage:
- Name: DevQA
Actions:
- Name: DeployInfrastructure
InputArtifacts:
- Name: SourceCodeArtifact
ActionTypeId:
Category: Deploy
Owner: AWS
Provider: CloudFormation
Version: '1'
Configuration:
StackName: !Ref AppName
Capabilities: CAPABILITY_NAMED_IAM
RoleArn: !GetAtt [CloudFormationRole, Arn]
ParameterOverrides: !Sub '{"ImageId": "${image_version??}"}'
Note that image_version in the last line above is just my aspirational placeholder to illustrate how I hope to use the image_version json value.
How can Codepipeline read the value of a field in a json file which is in SourceCodeArtifact?
StepFunctions? Lambda? CodeBuild?
You can use a CodeBuild step in between Source and Deploy stages.
In CodeBuild step, read the image_version from SourceArtifact (artifact produced by soruce stage) and write to an artifact 'Template configuration' file 1 which is a configuration property of the CloudFormation action. This file can hold parameter values for your CloudFormation stack. Use this file instead of ParameterOverrides you are currently using.
Fn::GetParam is what you want. It can returns a value from a key-value pair in a JSON-formatted file. And the JSON file must be included in an artifact.
Here is the documentation and it gives you some examples: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/continuous-delivery-codepipeline-parameter-override-functions.html#w2ab1c13c20b9
It should be something like:
ParameterOverrides: |
{
"ImageId" : { "Fn::GetParam" : ["SourceCodeArtifact", "imageManifest.json", "image_id"]}
}
I have a commandline tool in cwl which can take the following input:
fastq:
class: File
path: /path/to/fastq_R1.fq.gz
fastq2:
class: File
path: /path/to/fastq_R2.fq.gz
sample_name: foo
Now I want to scatter over this commandline tool and the only way I can think to do it is with scatterMethod: dotproduct and an input of something like:
fastqs:
- class: File
path: /path/to/fastq_1_R1.fq.gz
- class: File
path: /path/to/fastq_2_R1.fq.gz
fastq2s:
- class: File
path: /path/to/fastq_1_R2.fq.gz
- class: File
path: /path/to/fastq_2_R2.fq.gz
sample_names: [foo, bar]
Is there any other way for me to design the workflow and / or input file such that each input group is sectioned together? Something like
paired_end_fastqs:
- fastq:
class: File
path: /path/to/fastq_1_R1.fq.gz
fastq2:
class: File
path: /path/to/fastq_1_R2.fq.gz
sample_name: foo
- fastq:
class: File
path: /path/to/fastq_2_R1.fq.gz
fastq2:
class: File
path: /path/to/fastq_2_R2.fq.gz
sample_name: bar
You could accomplish this with a nested workflow wrapper that maps a record to each individual field, and then using that workflow to scatter over an array of records. The workflow would look something like this:
---
class: Workflow
cwlVersion: v1.0
id: workflow
inputs:
- id: paired_end_fastqs
type:
type: record
name: paired_end_fastqs
fields:
- name: fastq
type: File
- name: fastq2
type: File
- name: sample_name
type: string
outputs: []
steps:
- id: tool
in:
- id: fastq
source: paired_end_fastqs
valueFrom: "$(self.fastq)"
- id: fastq2
source: paired_end_fastqs
valueFrom: "$(self.fastq2)"
- id: sample_name
source: paired_end_fastqs
valueFrom: "$(self.sample_name)"
out: []
run: "./tool.cwl"
requirements:
- class: StepInputExpressionRequirement
Specify a workflow input of type record, with fields for each of the inputs the tool accepts, the ones you want to keep together while scattering. Connect the workflow input to each tool input, its source. Using valueFrom on the step inputs, transform the record (self is the source this context) to pass only the appropriate field to the tool.
More about valueFrom in workflow steps: https://www.commonwl.org/v1.0/Workflow.html#WorkflowStepInput
Then use this wrapper in your actual workflow and scatter over paired_end_fastqs with an array of records.