I a have a Gitlab stage that I only want to run if there are changes to some file. I have tried using only and if: changes... however those not work as I want.
I want that this stage is only triggered if the current commit (no matter if it is a merge request or pushed to main) contains changes to some file
My stage:
cache frontend:
stage: cache
cache: &frontend_cache
key: frontend-packages
paths:
- frontend/node_modules/
- frontend/.yarn/
script:
- cd frontend
- yarn install --immutable
only:
changes:
- frontend/yarn.lock
Related
On my gitlab repo I have to trigger pipeline only when a specific folder have changes AND when it's a Merge request (both condition). Especially on .zip file extension, i-e, add a new zip file in this folder, create a Merge request, then run the pipeline.
This is my initial pipeline yaml code:
trigger-ci-zip-file-only:
stage: prebuild
extends:
- .prebuild
- .preprod-tags
variables:
PROJECT_FOLDER: "my-specific-folder"
before_script:
- echo "Job currently run in $CI_JOB_STAGE"
- touch ${CI_PROJECT_DIR}/$PROJECT_FOLDER/prebuild.env
- cd $PROJECT_FOLDER
only:
refs:
- merge_requests
changes:
- ${PROJECT_FOLDER}/*.zip
artifacts:
reports:
dotenv: ${CI_PROJECT_DIR}/${PROJECT_FOLDER}/prebuild.env
allow_failure: false
As you can see, my pipeline have to be triggered only when there is a change in a specific folder on zip files and only on MR. But in this state, the pipeline is always running when MR is creatd or when I push on an existing MR, even if there is no changes or adding in the specific folder.
I also tried to make some changes on the pipeline yaml code like this:
only:
refs:
- merge_requests
changes:
- ${PROJECT_FOLDER}/**/*.zip
But the pipeline always running.
Also I tried this:
trigger-ci-zip-file-only:
stage: prebuild
extends:
- .prebuild
- .preprod-tags
variables:
PROJECT_FOLDER: "my-specific-folder"
before_script:
- echo "Job currently run in $CI_JOB_STAGE"
- touch ${CI_PROJECT_DIR}/$PROJECT_FOLDER/prebuild.env
- cd $PROJECT_FOLDER
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
- changes:
- ${PROJECT_FOLDER}/**/*.zip
when: always
artifacts:
reports:
dotenv: ${CI_PROJECT_DIR}/${PROJECT_FOLDER}/prebuild.env
allow_failure: false
But the pipeline always running too.
How to make sure the pipeline only run on Merge Request AND only when a .zip file was added to the specific folder ?
Question: Is there a way to tell gitlab to only wait on an earlier pipeline step, if this step has not been skipped?
Context: We did some optimization steps for our continuous integration pipeline.
With the help of the cache keyword provided by gitlab, we only install our nodejs dependencies once per branch in a separate build step. The node_modules folder is then persisted in the cache and made available to all subsequent steps and pipeline runs for this specific branch. The pipeline only re-installs the dependencies and updates the cache, if the package-lock.json has changed in a new commit.
See: https://docs.gitlab.com/ee/ci/yaml/#cache
The caching does work well so far, the only issue is that I cannot configure subsequent jobs to wait for the install job. If I would do so - for example with needs: ["install"] -, the pipeline would get blocked on all runs, if the package-lock.json has not changed and thus, the install step has been skipped.
Example:
install:
stage: "install"
rules:
- changes:
- package-lock.json
needs: [ ]
cache:
key: $CI_COMMIT_REF_SLUG-$CI_PROJECT_DIR
paths:
- node_modules/
script:
- npm ci ...
build:
stage: "build"
needs: [ ]
cache:
key: $CI_COMMIT_REF_SLUG-$CI_PROJECT_DIR
paths:
- node_modules/
policy: pull
script:
- ./build.sh
artifacts:
paths:
- build/
I have a gitlab job that downloads a bunch of dependencies and stuffs them in a cache (if necessary), then I have a bunch of jobs that use that cache. I notice at the end of the consuming jobs, they spend a bunch of time creating a new cache, even though they made no changes to it.
Is it possible to have them act only as consumers? Read-only?
cache:
paths:
- assets/
configure:
stage: .pre
script:
- conda env update --prefix ./assets/env/base -f ./environment.yml;
- source activate ./assets/env/base
- bash ./download.sh
parse1:
stage: build
script:
- source activate ./assets/env/base;
- ./build.sh -b test -s 2
artifacts:
paths:
- build
parse2:
stage: build
script:
- source activate ./assets/env/base;
- ./build.sh -b test -s 2
artifacts:
paths:
- build
In the very detailed .gitlab-ci.yml documentation is a reference to a cache setting called policy. GitLab caches have the concept of push (aka write) and pull (aka read). By default it is set to pull-push (read at the beginning and write at the end).
If you know the job does not alter the cached files, you can skip the upload step by setting policy: pull in the job specification. Typically, this would be twinned with an ordinary cache job at an earlier stage to ensure the cache is updated from time to time:
.gitlab-ci.yml > cache:policy
Which pretty much describes this situation: the job configure updates the cache, and the parse jobs do not alter the cache.
In the consuming jobs, add:
cache:
paths:
- assets/
policy: pull
For clarity, it probably wouldn't hurt to make that explicit in the global setting:
cache:
paths:
- assets/
policy: pull-push
TLDR. Overwrite cache with no path element.
You probably have to add a key element to your global cache configuration too. I actually have never used without a key element.
See the cache documentation here
Background
Consider a dummy gitlab job with a prebuild and build stage.
stages:
- prebuild
- build
The sole job of the prebuild stage is to create and cache dependencies for later stages. The prebuild stage is only triggered if a python requirements.txt file changes. The build stage has a cache:policy of pull, meaning it defers to prebuild for the cache. For example, imagine the cache consisted of a shared a single file sharedfile:
prebuild_job:
cache:
policy: pull-push
rules:
- changes:
- requirements.txt
paths:
- sharedfile
script:
- echo "FOOBAR" > sharedfile
build_job:
cache:
policy: pull
paths:
- sharedfile
script:
- cat sharedfile
This does what I want - prebuild is triggered only when requirements.txt changes and build uses its output (in this case sharedfile but in reality a full python virtual env.).
This works great until the cache is manually cleared on gitlab via the Clear Runner Caches button, then we have to manually re-run the prebuild step or builds will keep failing as the prebuild was never retriggered after the cache clear.
Question
How can we setup a rule or workaround such that the prebuild_job will know to fire if the cache is empty in addition to its existing change rule?. Something like
prebuild_job:
cache:
policy: pull-push
rules:
- changes:
- requirements.txt
- if: "$CACHE == {}" #<-- something like this
paths:
- sharedfile
script:
- echo "FOOBAR" > sharedfile
Workaround
Can add an optional button to retrigger the prebuild phase manually. allow_failure means the button is optional and won't block subsequent stages.
prebuild_job:
cache:
policy: pull-push
rules:
- changes:
- requirements.txt
- if: '"True"'
when: manual
allow_failure: true
...
I want to run Job only for merge_requests refs AND when changes for specified files, according to official doc i create this Job in .gitlab-ci.yml:
merge-request-test:
<<: *some_anchor
only:
refs:
- merge_requests
changes:
- "*.py"
- "**/*.py"
- postman/some-file.json
But this job starts even if i change any other file.
I understand that GitLab CI apply OR rule for refs and changes sections:
In the example below, the test job will not be created when any of the
following are true:
The pipeline runs for the master. There are changes to the README.md
file in the root directory of the repo.
test:
script: npm run test
except:
refs:
- master
changes:
- "README.md"
I want to start this Job only for MR and when changes specified files.
How i can achieve this behaviour?
If remove refs section, it works only when files changed, but official doc does not recommend do this, because i want to use it in pipelines:
If using only:changes with only allow merge requests to be merged if
the pipeline succeeds, undesired behavior could result if you do not
also use only:merge_requests.
For now i have this working solution:
merge-request-deploy:
<<: *deploy-app
rules:
- if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/ && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "develop"'
changes:
- "*.py"
- "**/*.py"
- postman/some-file.json
when: always
if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/ && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "develop"' - gives me MR branc
changes - gives me changed files
Rule's sections works with AND rule.