I'm setting up a gitlab pipeline with multiple stages and at least two sites each stage. How do iIconditionally allow the next stage to continue even if one site in the first stage failed (and the whole stage as it is marked as failed)? For example: I want to prepare, build and test and I want to do this on Windows & Linux runner. So if my Linux runner failed in preparation but my Windows runner succeeded, then the next stage should start without building the Linux package, because this already failed. But the windows build should start.
My Intention is that if one system fails at least the second is able to continue.
I added dependencies and i thought that this would solve my problem. Because if site "build windows" is dependent on "prepare windows" then it shouldn't matter if "prepare Linux" failed. But this isn't the case :/
image: node:10.6.0
stages:
- prepare
- build
- test
prepare windows:
stage: prepare
tags:
- windows
script:
- npm i
- git submodule foreach 'npm i'
prepare linux:
stage: prepare
tags:
- linux
script:
- npm i
- git submodule foreach 'npm i'
build windows:
stage: build
tags:
- windows
script:
- npm run build-PWA
dependencies:
- prepare windows
build linux:
stage: build
tags:
- linux
script:
- npm run build-PWA
dependencies:
- prepare linux
unit windows:
stage: test
tags:
- windows
script:
- npm run test
dependencies:
- build windows
artifacts:
paths:
- dist/
- package.json
expire_in: 5 days
unit linux:
stage: test
tags:
- linux
script:
- npm run test
dependencies:
- build linux
artifacts:
paths:
- dist/
- package.json
expire_in: 5 days
See allow_failure option:
allow_failure allows a job to fail without impacting the rest of the CI suite.
example:
job1:
stage: test
script:
- execute_script_that_will_fail
allow_failure: true
Related
im writing my first pipeline with bitbucket in yml and was wondering how to fail my pipeline when eslint test results some error and pass only if eslint test has no error logs.
this is how my pipeline looks so far:
image: node:14.15.1
definitions:
caches:
yarn: ~/.yarn
pipelines:
branches:
dev:
- step:
name: Package Install
caches:
- yarn
script:
- cd web_app
- yarn install
- step:
name: lint
caches:
- yarn
script:
- cd web_app
- yarn install
- yarn run lint
- step:
name: build
caches:
- yarn
script:
- cd web_app
- yarn install
- unset CI
- yarn build
will appreciate any help!
I have created one simple react for practicing gitlab's CI/CD pipeline. I have three jobs for CD/CI pipeline. First test the app then build then deploy to the AWS' S3 bucket. After successfully pass the test and run the build production, when it goes deploy stage I got this error : The user-provided path build does not exist. I don't know how to make path in Gitlab's cd/ci pipeline.
This is my gitlab's .gitlab-ci.yml file setup
image: 'node:12'
stages:
- test
- build
- deploy
test:
stage: test
script:
- yarn install
- yarn run test
variables:
AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
AWS_REGION: $AWS_REGION
S3_BUCKET_NAME: $S3_BUCKET_NAME
build:
stage: build
only:
- master
script:
- npm install
- npm run build
deploy:
stage: deploy
only:
- master
image: python:latest
script:
- pip install awscli
- aws s3 cp build/ s3://$S3_BUCKET_NAME/ --recursive --include "*"
If build/ folder is created as part of build stage then it should be passed as artefact to the deploy stage and deploy should reference build stage using dependencies:
build:
stage: build
only:
- master
script:
- npm install
- npm run build
artifacts:
paths:
- build/
deploy:
stage: deploy
only:
- master
image: python:latest
dependencies:
- build
script:
- pip install awscli
- aws s3 cp build/ s3://$S3_BUCKET_NAME/ --recursive --include "*"
im not sure if this has already been indicated in the docs but i want to know if this is possible
i have 2 runners (1 for building, and 1 for deploying)
i have my .gitlab-ci.yml configured to run 2 jobs
job1 to build
job2 to deploy
but currently both jobs run in parallel.
what i need is for job2 to wait for job1.
is the above scenario possible?
here is my sample .gitlab-ci.yml
job1:
tags:
- test1
script:
- echo Starting test build
- ./mvnw clean package -DskipTest
before_script:
- cd backend
- chmod +x mvnw
job2:
tags:
- deploy
script:
- echo Deployment Test
before_script:
- echo Pre-Deployment scripts running
thanks
Define 'stages' for the jobs.
stages:
- job1
- job2
job1:
tags:
- test1
script:
- echo Starting test build
- ./mvnw clean package -DskipTest
before_script:
- cd backend
- chmod +x mvnw
job2:
tags:
- deploy
script:
- echo Deployment Test
before_script:
- echo Pre-Deployment scripts running
More details in the documentation.
Try using the "needs" keyword in combination with unique runner tags.
job1:
tags:
- test1
script:
- echo Starting test build
- ./mvnw clean package -DskipTest
before_script:
- cd backend
- chmod +x mvnw
job2:
tags:
- deploy
script:
- echo Deployment Test
before_script:
- echo Pre-Deployment scripts running
needs: ["job1"]
This is the ideal solution as it bypasses the priority of stages and therefore can be used in combination with any stages you actually might have associated with your pipelines. This option allows more flexibility in more complex (including multi project/pipeline) configurations.
I have built a "deploy review" stage and deploy job in my yaml file with the following code:
deploy review:
stage: deploy review
only:
- merge_requests
environment:
name: review/$CI_COMMIT_REF_NAME
url: https://crazymonk84-$CI_ENVIRONMENT_SLUG.surge.sh
script:
- npm install -g surge
- surge --project ./public --domain
https://crazymonk84-$CI_ENVIRONMENT_SLUG.surge.sh
When I check the pipeline on my gitlab account, I see the commit on my review branch but I do not see "deploy review" job running. I see "test artifact" "test website" jobs running.
The link to the gitlab project is https://gitlab.com/syed.r.abdullah/my-static-website/tree/review
I took the following steps:
Added "deploy review" to the yaml file
Created a new branch "review" locally
Added the change to the yaml file in review branch
Committed the change
Pushed the change to gitlab using git push -u origin review
Visited my pipelines and saw review pipeline in failed state
Jobs, inside the review pipeline are "test artifact" and "test website", not "deploy review"
image: node
variables:
STAGING_DOMAIN: crazymonk84-staging.surge.sh
PRODUCTION_DOMAIN: crazymonk84.surge.sh
stages:
- build
- test
- deploy review
- deploy staging
- deploy production
- production tests
- cache
cache:
key: ${CI_COMMIT_REF_SLUG}
policy: pull
paths:
- node_modules/
update cache:
stage: cache
script:
- npm install
only:
- schedules
cache:
key: ${CI_COMMIT_REF_SLUG}
policy: push
paths:
- node_modules/
build website:
stage: build
only:
- master
- merge_requests
except:
- schedules
script:
- echo $CI_COMMIT_SHORT_SHA
- npm install -g gatsby-cli
- npm i xstate#4.6.4
- gatsby build
- sed -i "s/%%VERSION%%/$CI_COMMIT_SHORT_SHA/" ./public/index.html
artifacts:
paths:
- ./public
test website:
stage: test
except:
- schedules
script:
- npm install -g gatsby-cli
- npm i xstate#4.6.4
- gatsby serve &
- sleep 3
- curl "http://localhost:9000" | tac | tac | grep -q "Gatsby"
test artifact:
image: alpine
stage: test
except:
- schedules
script:
- grep -q "Gatsby" ./public/index.html
cache: {}
deploy review:
stage: deploy review
only:
- merge_requests
environment:
name: review/$CI_COMMIT_REF_NAME
url: https://crazymonk84-$CI_ENVIRONMENT_SLUG.surge.sh
script:
- npm install -g surge
- surge --project ./public --domain https://crazymonk84-$CI_ENVIRONMENT_SLUG.surge.sh
deploy staging:
stage: deploy staging
environment:
name: staging
url: http://$STAGING_DOMAIN
only:
- master
except:
- schedules
script:
- npm install --global surge
- surge --project ./public --domain $STAGING_DOMAIN
cache: {}
deploy production:
stage: deploy production
environment:
name: production
url: http://$PRODUCTION_DOMAIN
only:
- master
when: manual
allow_failure: false
except:
- schedules
script:
- npm install --global surge
- surge --project ./public --domain $PRODUCTION_DOMAIN
cache: {}
production tests:
image: alpine
stage: production tests
only:
- master
except:
- schedules
script:
- apk add --no-cache curl
- curl -s "https://$PRODUCTION_DOMAIN" | grep -q "Hi people"
- curl -s "https://$PRODUCTION_DOMAIN" | grep -q "$CI_COMMIT_SHORT_SHA"
cache: {}
I am expecting to see "deploy review" as the only job in the pipeline. However, I see "test artifact" and "test website." What can I do to fix the issue? Thanks.
I found one solution:
Add the following to the build website and deploy review stages:
only:
- master
- merge_requests
I expect you watched a course by Valentin Despa. I've just come across the same problem. I wonder if he published any solution to the issue.
Basically I'm not positive if I'm correct, but...
if we open the link, there might be found an explanation:
https://docs.gitlab.com/ee/ci/pipelines/merge_request_pipelines.html
only:
- merge_requests
runs the deploy review stage in a detached mode. We don't have an access to our environment variables since they are protected. What I did was I went to Settings -> CI/CD -> Variables -> got rid of a tick in a protected variable option for both variables.
Then, if you run the pipeline again you'll notice it'll throw an error on ./public.
Play his video Dynamic environments and notice at 4:49 (timeline) there are three green pipelines. You have one only (your pipeline is detached)! Meaning build website stage hasn't been run. That's why you'll see the error relating to ./public since your pipeline knows nothing about Gatsby. We need to install Gatsby first and then build it.
deploy review:
stage: deploy review
environment:
name: review/$CI_COMMIT_REF_NAME
url: https://yourdomain-$CI_ENVIRONMENT_SLUG.surge.sh
only:
- merge_requests
script:
- npm install --silent
- npm install -g gatsby-cli
- gatsby build
- npm install --global surge
- surge --project ./public --domain yourdomain-$CI_ENVIRONMENT_SLUG.surge.sh
artifacts:
paths:
- ./public
I'm using GitLab CI.
I have 2 jobs in the build stage that build differently my app. The 2 jobs upload a cache for the branch. I use the compiled sources to launch some tests in the test stage.
build:
stage: build
script:
- ./gradlew build --build-cache --quiet
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- "*/build"
build_with_different_conf:
stage: build
script:
- ./gradlew buildDiff --build-cache --quiet
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- "*/build"
Test:
stage: test
script:
- ./gradlew test --build-cache
In my example, the job build_with_different_conf take more time to finish.
My question is : Is the last finishing build job upload the cache and replace the cache from the first build job or is it merging files whith the precedent job ?
Thanks.
From what i understand you are using global cache for gradle dependencies.
Than you want to have some kind of job, to job cache.
I would do it this way, more or less.
stages:
- build
- test
cache:
paths:
- <your_gradle_cache>
build_classes:
stage: build
script:
- ./gradlew build --build-cache --quiet
artifacts:
expire_in: 1d
paths:
- <your_build_dir>
build_war:
stage: build
dependencies:
- build_classes
script:
- ./gradlew buildDiff --build-cache --quiet
artifacts:
expire_in: 1w
paths:
- <path_to_your_war>
test_classes:
stage: test
dependencies:
- build_war
script:
- ./gradlew test --build-cache
test_war:
stage: test
dependencies:
- build_war
script:
- test # some kind of test to assure your war is in good condition
In this configuration:
build_classes --[.classes]--> build_war -> [.war]
| |
[.classes] [.war]
| |
V V
test_classes test_war
PS. Dont forget you can use shell (or whatever your runner's os) to debug, understand more about this. You can put ls -la all over the place.
build:
stage: build
Same stage jobs runs in parallel.
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- "*/build"
Cache files are managed by cache:key. It means if you're using same cache:key for different jobs they'll share same cache.zip file across jobs even you're defining different cache:paths. If you're using same key, but different path then your cache won't be effective, because of every job will overwrite cache.zip file with different path contents.
In your case you're using same cache:key across different jobs.
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- "*/build"
It means last finished job will overwrite the cache.zip file not merge and will be used for next job and subsequent pipeline jobs with same key defined.
Bonus:
Test:
stage: test
script:
- ./gradlew test --build-cache
Also beware that if this job requires */build directory contents to exist, you have to be careful and better to use artifacts instead. Cache doesn't always exist and it's provided as best effort delivery.
For example I use gitlab ci's cache like this.
nodejs_test:
stage: test
image: node:12.13-alpine
before_script:
- npm install
script:
- yarn test
cache:
key:
files:
# New cache key will be computed on each package.json change.
- package.json
paths:
- node_modules/
nodejs_build:
stage: build
image: node:12.13-alpine
before_script:
# In case if we miss cache, we can simply install packages again.
# If cache is there npm install won't download them again.
- npm install
script:
- yarn build
cache:
policy: pull # totally optional
key:
files:
- package.json
prefix: ${CI_COMMIT_REF_SLUG}
paths:
- node_modules/