Hi I have the following pipeline config
image: maven:3.3.9
definitions:
steps:
- step: &build-step
name: SonarQube analysis
script:
- pipe: sonarsource/sonarcloud-scan:1.4.0
variables:
SONAR_HOST_URL: ${SONAR_HOST_URL} # Get the value from the repository/workspace variable.
SONAR_SCANNER_OPTS: -Xmx1024m
SONAR_TOKEN: ${SONAR_TOKEN} # Get the value from the repository/workspace variable. You shouldn't set secret in clear text here.
caches:
sonar: ~/.sonar
clone:
depth: full
pipelines:
branches:
'{master}': # or the name of your main branch
- step: *build-step
I want to increase the size of the docker to 2x could you please help with the proper YML config for the same?
I tried changing it to
- step:
size: 2x
but it won't work, I get erros that step is null or missing
Related
Google Cloud docs use the first code block but I'm wondering why they don't use the second one. As far as I can tell they achieve the same result. Is there any practical difference?
# config 1
steps:
# Build the container image
- name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/project-id/project-name','.']
- name: 'gcr.io/cloud-builders/docker'
args: ['push', 'gcr.io/project-id/project-name']
# Deploy container image to Cloud Run
- name: 'gcr.io/cloud-builders/gcloud'
args: ['run', 'deploy', 'project-name', '--image', 'gcr.io/project-id/project-name', '--region', 'us-central1']
images: ['gcr.io/project-id/project-name']
# config 2
steps:
# Build the container image
- name: 'gcr.io/cloud-builders/gcloud'
args: ['builds', 'submit', '--region', 'us-central1', '--tag', 'gcr.io/project-id/project-name','.']
# Deploy container image to Cloud Run
- name: 'gcr.io/cloud-builders/gcloud'
args: ['run', 'deploy', 'project-name', '--image', 'gcr.io/project-id/project-name', '--region', 'us-central1']
I run this with gcloud builds submit --config cloudbuild.yaml
In the second config, you call a Cloud Build from inside a Cloud Build, that means you pay twice the docker build/push process in the second config.
That time of timeline in fact
Cloud Build 1
Cloud build 2
Docker Build
Docker Push
Deploy on cloud run
In addition, the number of concurrent build are limited and with the config 2 you use 2 times more quotas.
And the result is the same (it should be slightly faster with the config 1 because you haven't a new Cloud Build to spin up)
I am building a concourse pipeline for a spring-boot service. The pipeline has two jobs (test, package-build-and-push).
The first job (test) has a single task (mvn-test-task) that will be responsible for running a maven test.
The second job has two tasks (mvn-package-task, build-image-task). The second job will only run after the first job (test) has finished. The first task (mvn-package-task) will be doing a maven package. Once the task (mvn-package-task) is finished it should copy the complete project folder with the generated target folder to the directory "concourse-demo-repo-out". This folder (concourse-demo-repo-out) then becomes the input folder for the next task (build-image-task) that will use to generate a docker image. The output will be a docker image that is placed in the directory (image).
I now need to push this image to an Amazon ECR repository. For this I am using a resource of the type docker-image (https://github.com/concourse/docker-image-resource). The problem that I am facing now is:
How do I tag the image that was created, and is placed inside the image directory?
How do I specify that I already have an image created to this docker image resource type, so that it uses that, tags it, and then pushes it to ECR.
My pipeline looks like this
resources:
- name: concourse-demo-repo
type: git
icon: github
source:
branch: main
uri: https://github.xyz.com/gzt/concourse-demo.git
username: <my-username>
password: <my-token>
- name: ecr-docker-reg
type: docker-image
icon: docker
source:
aws_access_key_id: <my-aws-access-key>
aws_secret_access_key: <my-aws-secret-key>
repository: <my-aws-ecr-repo>
jobs:
- name: test
public: true
plan:
- get: concourse-demo-repo
trigger: true
- task: mvn-test-task
file: concourse-demo-repo/ci/tasks/maven-test.yml
- name: package-build-and-push
public: true
serial: true
plan:
- get: concourse-demo-repo
trigger: true
passed: [test]
- task: mvn-package-task
file: concourse-demo-repo/ci/tasks/maven-package.yml
- task: build-image-task
privileged: true # oci-build-task must run in a privileged container
file: concourse-demo-repo/ci/tasks/build-image.yml
- put: ecr-docker-reg
params:
load: image
The maven package task and script
---
platform: linux
image_resource:
type: docker-image
source:
repository: maven
inputs:
- name: concourse-demo-repo
run:
path: /bin/sh
args: ["./concourse-demo-repo/ci/scripts/maven-package.sh"]
outputs:
- name: concourse-demo-repo-out
#!/bin/bash
set -e
cd concourse-demo-repo
mvn clean package
#cp -R ./target ../concourse-demo-repo-out
cp -a * ../concourse-demo-repo-out
The build image task
---
platform: linux
image_resource:
type: registry-image
source:
repository: concourse/oci-build-task
inputs:
- name: concourse-demo-repo-out
outputs:
- name: image
params:
CONTEXT: concourse-demo-repo-out
run:
path: build
So finally when I run the pipeline, everything works fine, I am able to build a docker image that is placed in the image directory but I am not able to use the image as a part of "put: ecr-docker-reg".
The error that I get while running the pipeline is
selected worker: ed5d4164f835
waiting for docker to come up...
open image/image: no such file or directory
Consider using registry-image instead. Use this example for image resource definition where you specify the desired tag.
Your build-image-task is analogous to the build task from the example.
Finally, the put will look something like:
- put: ecr-docker-reg
params:
image: image/image <--- but make sure this file exists, read on...
Given that concourse complained about open image/image: no such file or directory, verify that the filename is what you expect it to be. Hijack the failed task and look inside of the image output directory:
$ fly -t my-concourse i -j mypipeline/package-build-and-push
(select the build-image-task from the list)
# ls -al image/
(actual filename here)
👍
I'm deploying to Kubernettes via Cloud Build. Every now and then the build times out because it exceeds the build-in time out of ten minutes. I can't figure out how to increase this time out. I'm using in-line build config in my trigger. It looks like this
steps:
- name: gcr.io/cloud-builders/docker
args:
- build
- '-t'
- '$_IMAGE_NAME:$COMMIT_SHA'
- .
- '-f'
- $_DOCKERFILE_NAME
dir: $_DOCKERFILE_DIR
id: Build
- name: gcr.io/cloud-builders/docker
args:
- push
- '$_IMAGE_NAME:$COMMIT_SHA'
id: Push
- name: gcr.io/cloud-builders/gke-deploy
args:
- prepare
- '--filename=$_K8S_YAML_PATH'
- '--image=$_IMAGE_NAME:$COMMIT_SHA'
- '--app=$_K8S_APP_NAME'
- '--version=$COMMIT_SHA'
- '--namespace=$_K8S_NAMESPACE'
- '--label=$_K8S_LABELS'
- '--annotation=$_K8S_ANNOTATIONS,gcb-build-id=$BUILD_ID'
- '--create-application-cr'
- >-
--links="Build
details=https://console.cloud.google.com/cloud-build/builds/$BUILD_ID?project=$PROJECT_ID"
- '--output=output'
id: Prepare deploy
- name: gcr.io/cloud-builders/gsutil
args:
- '-c'
- |-
if [ "$_OUTPUT_BUCKET_PATH" != "" ]
then
gsutil cp -r output/suggested gs://$_OUTPUT_BUCKET_PATH/config/$_K8S_APP_NAME/$BUILD_ID/suggested
gsutil cp -r output/expanded gs://$_OUTPUT_BUCKET_PATH/config/$_K8S_APP_NAME/$BUILD_ID/expanded
fi
id: Save configs
entrypoint: sh
- name: gcr.io/cloud-builders/gke-deploy
args:
- apply
- '--filename=output/expanded'
- '--cluster=$_GKE_CLUSTER'
- '--location=$_GKE_LOCATION'
- '--namespace=$_K8S_NAMESPACE'
id: Apply deploy
timeout: 900s
images:
- '$_IMAGE_NAME:$COMMIT_SHA'
options:
substitutionOption: ALLOW_LOOSE
substitutions:
_K8S_NAMESPACE: default
_OUTPUT_BUCKET_PATH: xxxxx-xxxxx-xxxxx_cloudbuild/deploy
_K8S_YAML_PATH: kubernetes/
_DOCKERFILE_DIR: ''
_IMAGE_NAME: xxxxxxxxxxx
_K8S_ANNOTATIONS: gcb-trigger-id=xxxxxxxx-xxxxxxx
_GKE_CLUSTER: xxxxx
_K8S_APP_NAME: xxxxx
_DOCKERFILE_NAME: Dockerfile
_K8S_LABELS: ''
_GKE_LOCATION: xxxxxxxx
tags:
- gcp-cloud-build-deploy
- $_K8S_APP_NAME
I've tried sticking the timeout: 900 arg in in various places with no luck.
The timeout of 10 minutes is the default for the whole build, therefore if you add the timeout: 900s option in any of the steps, it will only apply to the step that it has been added to. You can make a step have a larger timeout than the overall build timeout, but the whole build process will fail if the sum of all the steps exceeds the build timeout. This example shows this behavior:
steps:
- name: 'ubuntu'
args: ['sleep', '600']
timeout: 800s # Step timeout -> Allows the step to run up to 800s, but as the overall timeout is 600s, it will fail after that time has been passed, so the effective timeout value is 600s.
timeout: 600s # Overall build timeout
That said, the solution is to expand the overall build timeout by adding it outside of any step, and then you can have a build with up to 24h to finish before it fails with a timeout error.
Something like the following example should work out for you:
steps:
- name: 'ubuntu'
args: ['sleep', '600']
timeout: 3600s
Another way to solve this problem is to use a high-end machine so that overall it takes less time in the build process.
You can specify it like
options:
machineType: N1_HIGHCPU_8
Note: This performance benefits come at a cost. Please look into the pricing section to use optimal machine as per your requirement and budget.
I want to config.yml that has a "build" and "test" jobs.
We have a Makefile in our docker image but it seems it's not finding the make file since I'm always getting this error
#!/bin/bash -eo pipefail
make test
make: *** No rule to make target 'test'. Stop.
Exited with code exit status 2
CircleCI received exit code 2
Is anyone familiar with this? I'm not sure what I have missed.
Thank you!
Here is the yml file:
version: 2.1
workflows:
my-workflow:
jobs:
- build:
context: circleci-aws-credentials
- test
jobs:
build:
docker:
- image: circleci/openjdk:11-jdk
working_directory: ~/my-project
environment:
# Customize the JVM maximum heap limit
JVM_OPTS: -Xmx3200m
steps:
- checkout
# Download and cache dependencies
- restore_cache:
key: my-project-service-{{ checksum "./build.gradle" }}
test:
docker:
- image: circleci/openjdk:11-jdk
working_directory: ~/my-project
environment:
# Customize the JVM maximum heap limit
JVM_OPTS: -Xmx3200m
steps:
- run:
name: Gradle test
command: make test
- store_test_results:
path: ./build/test-results
Thank you for your help in advance!
I'm trying to write some aliases into my YAML file but I keep getting an error. How can I configure aliases in YAML?
...rest_of_code
# DEFAULTS
checkout-and-attach: &checkout-and-attach
checkout:
path: ~/app
attach_workspace:
at: ~/app
# iOS Build and Test
build-and-test:
macos:
xcode: "10.2.0"
working_directory: ~/app/ios
steps:
<<: *checkout-and-attach # [ERROR]: Incorrect type expect array
...rest_of_code
I didn't find a way to spread an array with circleci.
In addition, I think that the references section must be before the jobs section.
Example for working config.yml with reference:
version: 2
references:
init_aws: &init_aws
run:
name: Init aws credentials
command: echo 1
init_env_vars: &init_env_vars
run:
name: Init environment variables
command: echo 2
jobs:
build:
docker:
- image: circleci/node:8.11.3
steps:
- *init_aws
- *init_env_vars