I've followed the docs for Google CloudBuild here: https://cloud.google.com/cloud-build/docs/configuring-builds/store-images-artifacts
So here's my cloudbuild.yaml configuration:
steps:
- name: gcr.io/cloud-builders/git
id: git-checkout
args: [ 'fetch','--tags','--unshallow']
- name: openjdk
id: gradle-build
args: [
'./gradlew',
'--build-cache',
'-Si',
'-Panalytics.buildId=$BUILD_ID',
'-PgithubToken=$_GITHUB_TOKEN',
'-g', '$_GRADLE_CACHE',
'build'
]
artifacts:
objects:
location: ['gs://my-bucket/artifacts/']
paths: ["build/libs/*.jar"]
If I comment out, the following, then it runs successfully:
artifacts:
objects:
location: ['gs://my-bucket/artifacts/']
paths: ["build/libs/*.jar"]
Without comments, I get the following error from the CloudBuild console:
failed unmarshalling build config cloudbuild.yaml: json: cannot unmarshal array into Go value of type string
And under the Logs section, it simply says Logs unavailable.
You may need to indent objects: line
artifacts:
objects:
location: ['gs://my-bucket/artifacts/']
paths: ["build/libs/*.jar"]
objects.location element should not be an array.
The following should work:
artifacts:
objects:
location: 'gs://my-bucket/artifacts/'
paths: ["build/libs/*.jar"]
I've also run into this error with a section of my cloudbuild.yaml file looking like:
- name: 'gcr.io/cloud-builders/git'
args:
- clone
- -depth
- 1
- --single-branch
- -b
- development
- git#bitbucket.org:aoaoeuoaeuoeaueu/oaeueoaueoauoaeuo.git
volumes:
- name: 'ssh'
path: /root/.ssh
Seems the issue is with the 1. So I just added quotes around which fixed it (- "1").
Related
I want to execute 3 steps in the same cloud build yaml but I get this error:
ERROR while parsing a block mapping
steps:
- name: 'git image'
args: [args]
id: 'Clone Repository'
# Compile the pipeline.
- name: 'custom image'
entrypoint: 'python'
args: [args]
id: 'Compile Pipeline'
artifacts:
objects:
location: '$_GCS_LOCATION'
paths: ['path']
- name: 'custom image'
entrypoint: 'python'
args: ['args']
id: 'Trigger Pipeline'
logsBucket: 'project-bucket-logs'
serviceAccount: 'service-account'
I get the error in the last:
name
expected , but found '-'
You have a different indent after artifacts: so this yaml file is not parsed the way you'd like. This part should look like this:
artifacts:
objects:
location: '$_GCS_LOCATION'
paths: ['path']
Could someone tell me the difference between the PUT step and the OUTPUT step in Concourse? For example, in the following type of YAML files why do we need a put step after a get? Can't we use output instead of put? If not what are the purposes of each two?
jobs:
- name: PR-Test
plan:
- get: some-git-pull-request
trigger: true
- put: some-git-pull-request
params:
context: tests
path: some-git-pull-request
status: pending
....
<- some more code to build ->
....
The purpose of a PUT step is to push to the given resource while an OUTPUT is the result of TASK step.
A task can configure outputs to produce artifacts that can then be propagated to either a put step or to another task step in the same plan.
This means that you send the resource that you are specifying on the GET step to the task as an input, to perform wherever build or scripts executions and the output of that task
is a modified resource that you can later pass to your put step or to another TASK if you don't want to use PUT.
It would also depend on the nature of the defined resource in your pipeline. I'm assuming that you have a git type resource like this:
resources:
- name: some-git-pull-request
type: git
source:
branch: ((credentials.git.branch))
uri: ((credentials.git.uri))
username: ((credentials.git.username))
password: ((credentials.git.pass))
If this is true, the GET step will pull that repo so you can use it as an input for your tasks and if you use PUT against that same resource as you are describing in your sample code, that will push changes to your repo.
Really it depends on the workflow that you want to write but to give an idea it would look something like this:
jobs:
- name: PR-Test
plan:
- get: some-git-pull-request
trigger: true
- task: test-code
config:
platform: linux
image_resource:
type: docker-image
source:
repository: yourRepo/yourImage
tag: latest
inputs:
- name: some-git-pull-request
run:
path: bash
args:
- -exc
- |
cd theNameOfYourRepo
npm install -g mocha
npm test
outputs:
- name: some-git-pull-request-output
Then you can use it on either a PUT
- put: myCloud
params:
manifest: some-git-pull-request-output/manifest.yml
path: some-git-pull-request-output
or a another task whitin the same plan
- task: build-code
config:
platform: linux
image_resource:
type: docker-image
source:
repository: yourRepo/yourImage
tag: latest
inputs:
- name: some-git-pull-request-output
run:
path: bash
args:
- -exc
- |
cd some-git-pull-request-output/
npm install
gulp build
outputs:
- name: your-code-build-output
Hope it helps!
Error parsing config file: yaml: line 22: did not find expected key
Cannot find a job named build to run in the jobs: section of your configuration file.
I got those errors, but I'm really new to yaml so I can't really find reaons why It's not working. any ideas? Some says It might have extra spaces or something, but I can't really find it.
yaml file
defaults: &defaults:
- checkout
- restore_cache:
keys:
- v1-dependencies-{{ checksum "package.json" }}
- v1-dependencies-
- run: npm install
- save_cache:
paths:
- node_modules
key: v1-dependencies-{{ checksum "package.json" }}
version: 2
jobs:
build:
docker:
- image: circleci/node:10.3.0
working_directory: ~/repo
steps:
<<: *defaults // << here
- run: npm run test
- run: npm run build
deploy:
docker:
- image: circleci/node:10.3.0
working_directory: ~/repo
steps:
<<: *defaults
- run:
name: Deploy app scripts to AWS S3
command: npm run update-app
workflows:
version: 2
build-deploy:
jobs:
- build
- deploy:
requires:
- build
filters:
branches:
only: master
What you are trying to do is trying to merge two sequences. ie all elements of default are merged into steps. Which is not supported in YAML spec. Only you can merge maps and nested sequences.
This is invalid:
steps:
<<: *defaults
- run:
as <<: is for merging map elements, not sequences
If you do this:
step_values: &step_values
- run ...
steps:
- *defaults
- *step_values
You will end up with nested sequences, which is not what you intend.
Its not possible for now. Unfortunately, the only solution is to repeat the whole list. Many users are requesting the same feature.
it looks like your YAML is not written properly. You can always check the structure validation of YAML from an open-source website such as http://www.yamllint.com/.
On checking the yaml file, on line 22 you are doing wrong. As explained by Srikanth, that you are trying to do is merging two sequences. i.e. all elements of default are merged into steps. Which is not supported in YAML at the moment.
Only you can merge maps and nested sequences
If you do this:
step_values: &step_values
- run ...
-----------------------------------------------
steps:
- *defaults
- *step_values
You will end up with nested sequences, which is not what you intend.
What is the best way to commit from a pipeline.The job pulls from a different repo and makes some changes + builds - then push the new files to a different repo.Is this possible?
You should use the git-resource.
The basic steps of what you are going to want to do are to
Pull from the repo into a container.
Do some stuff with the code
Move the new code into a different container
Push the contents of that new container to a different git-repository
Your pipeline configuration should look something like this:
jobs:
- name: pull-code
plan:
- get: git-resource-pull
- get: git-resource-push
- task: do-something
inputs:
- name: git-resource-pull
run:
path: /bin/bash
args:
- -c
- |
pushd git-resource-pull
// do something
popd
// move the code from git-resource-pull to git-resource-push
- put: git-resource-push
params: {repository: git-resource-push}
resources:
- name: git-resource-pull
type: git
source:
uri: https://github.com/team/repository-1.git
branch: master
- name: git-resource-push
type: git
source:
uri: https://github.com/team/repository-2.git
branch: master
I've been banging my head on this one for quite some time and I cannot figure it out (I know it must be a simple thing to do though).
Currently, what I'm trying to do is pulling from two repositories (which naturally creates two separate directories) then I'm trying to move files from one directory to the other to successfully execute the Dockerfile.
Here's how my pipeline.yml file looks like:
---
jobs:
- name: build-nexus-docker-image
public: false
plan:
- get: git-nexus-docker-images
trigger: true
- get: git-nexus-license
trigger: true
- task: mv-nexus-license
config:
platform: linux
image_resource:
type: docker-image
source: {repository: ubuntu, tag: "trusty"}
inputs:
- name: git-nexus-license
- name: git-nexus-docker-images
run:
path: /bin/sh
args:
- -c
- mv -v git-nexus-license/nexus.lic git-nexus-docker-images/nexus.lic; ls -la git-nexus-docker-images
- put: nexus-docker-image
params:
build: git-nexus-docker-images/
resources:
- name: git-nexus-docker-images
type: git
source:
uri: git#git.company.com:dev/nexus-pro-dockerfile.git
branch: test
paths: [Dockerfile]
private_key: {{git_ci_key}}
- name: git-nexus-license
type: git
source:
uri: git#git.company.com:secrets/nexus-information.git
branch: master
paths: [nexus.lic]
private_key: {{git_ci_key}}
- name: nexus-docker-image
type: docker-image
source:
username: {{aws-token-username}}
password: {{aws-token-password}}
repository: {{ecr-nexus-repo}}
I've posted the pipeline that actually can be deployed to Concourse; however I tried a lot of things, but I can't figure out how to do this. I'm stuck on the part of moving the license file from git-nexus-license directory to git-nexus-docker-images directory. What I've done doesn't seem to mv the nexus.lic file because when while building the docker image it fails because it cannot find that file in the directory.
EDIT: I've successfully been able to "mv" nexus.lic using the code above, however the build is still failing due to not finding the file! I'm not sure what I'm doing wrong, the build works properly if I do it manually but with Concourse it's failing.
Okay so I figured out what I was doing wrong and as usual it was something small. I forgot to add the outputs to the yml file which tells concourse that this is the new workdir. Here's how it looks like now (which works for me):
---
jobs:
- name: build-nexus-docker-image
public: false
plan:
- get: git-nexus-docker-images
trigger: true
- get: git-nexus-license
trigger: true
- task: mv-nexus-license
config:
platform: linux
image_resource:
type: docker-image
source: {repository: ubuntu, tag: "trusty"}
inputs:
- name: git-nexus-license
- name: git-nexus-docker-images
outputs:
- name: build-nexus-dir
run:
path: /bin/sh
args:
- -c
- mv -v git-nexus-license/nexus.lic build-nexus-dir/nexus.lic; mv -v git-nexus-docker-images/* build-nexus-dir; ls -la build-nexus-dir;
- put: nexus-docker-image
params:
build: build-nexus-dir/
resources:
- name: git-nexus-docker-images
type: git
source:
uri: git#git.company.com:dev/nexus-pro-dockerfile.git
branch: test
paths: [Dockerfile]
private_key: {{git_ci_key}}
- name: git-nexus-license
type: git
source:
uri: git#git.company.com:secrets/nexus-information.git
branch: master
paths: [nexus.lic]
private_key: {{git_ci_key}}
- name: nexus-docker-image
type: docker-image
source:
username: {{aws-token-username}}
password: {{aws-token-password}}
repository: {{ecr-nexus-repo}}
I hope this helps whoever gets stuck on this. :)