I think the problem speak for itselft. I have mutiples included hidden jobs. In my .gitlab-ci.yml and I would like to enjoy the 'variables' section of all of them.
I thought I would have found what I need here https://docs.gitlab.com/ee/ci/yaml/yaml_optimization.html but:
Anchors does not allow included files
!reference cannot be used mutiples times in the 'Variables' section
extends does not merge the content but take the last one.
If anyone as an idea. Here the behavior I am trying to achieve:
hidden_1.yml
.hidden_1:
variables:
toto1: toto1
hidden_2.yml
.hidden_2
variables:
toto2: toto2
hidden_3
.hidden_3
variables:
toto2: toto3
result.yml
include:
- 'hidden_3'
- 'hidden_2'
- 'hidden_1'
Job_test:
stage: test
variables:
toto2: toto3
toto1: toto1
toto2: toto3
script: Echo '$toto1, $toto2, $toto3'
Related
I'm having a gitlab yaml file whose before_scripts section needs to be used in another gitlab yaml. I'm doing something like this:
include:
- remote: 'https://gitlab.xxx.net/awsxxx/job-template/-/blob/master/.gitlab-
ci-template.yml'
extends:
- before_script
The relevant contents of the above url file are:
before_script:
- echo "foo"
- echo "bar"
This is not working, returns that the syntax is incorrect. Can you please help me correct this? Note: There are multiple extends and multiple parent 'include' and so I'm using the '-' format for extends and include here
I guess the error you're getting is because you can only use the extends keyword on a job, see the relavant page https://docs.gitlab.com/ee/ci/yaml/#extends. Are you trying to somehow append the remote yaml with your own before_script ? You should be able to reuse the job name from the remote yaml and do the before script there like:
include:
- remote: 'https://gitlab.xxx.net/awsxxx/job-template/-/blob/master/.gitlab-
ci-template.yml'
job to overwrite from ci-template:
before_script:
- echo "foo"
- echo "bar"
I am having trouble with dynamically passing one of two file based variables to a job.
I have defined two file variables in my CI/CD settings that contain my helm values for deployments to developement and production clusters. They are typical yaml syntax, their content does not really matter.
baz:
foo: bar
I have also defined two jobs for the deployment that depend on a general deployment template .deploy.
.deploy:
variables:
DEPLOYMENT_NAME: ""
HELM_CHART_NAME: ""
HELM_VALUES: ""
before_script:
- kubectl ...
script:
- helm upgrade $DEPLOYMENT_NAME charts/$HELM_CHART_NAME
--install
--atomic
--debug
-f $HELM_VALUES
The specialization happens in two jobs, one for dev and one for prod.
deploy:dev:
extends: .deploy
variables:
DEPLOYMENT_NAME: my-deployment
HELM_CHART_NAME: my-dev-chart
HELM_VALUES: $DEV_HELM_VALUES # from CI/CD variables
deploy:prod:
extends: .deploy
variables:
DEPLOYMENT_NAME: my-deployment
HELM_CHART_NAME: my-prod-chart
HELM_VALUES: $PROD_HELM_VALUES # from CI/CD variables
The command that fails is the one in the script tag of .deploy. If I pass in the $DEV_HELM_VALUES or $PROD_HELM_VALUES, the deployment is triggered. However if I put in the $HELM_VALUES as described above, the command fails (Error: "helm upgrade" requires 2 arguments, which is very misleading).
The problem is that the $HELM_VALUES that are accessed in the command are already the resolved content of the file, whereas passing the $DEV_HELM_VALUES or the $PROD_HELM_VALUES directly works with the -f syntax.
This can be seen using echo in the job's output:
echo "$DEV_HELM_VALUES"
/builds/my-company/my-deployment.tmp/DEV_HELM_VALUES
echo "$HELM_VALUES"
baz:
foo: bar
How can I make sure the $HELM_VALUES only point to one of the files, and do not contain the files' content?
I'm trying to download multiple artifacts into different servers(like web, db) using environments. Currently i have added the task DownloadPipelineArtifact#2 in a file and using template to add that task in azure-pipelines.yml. As i'm having multiple artifacts, im trying to use for loop where i'm getting issues.
#azure-pipelines.yml
- template: artifacts-download.yml
parameters:
pipeline:
- pipeline1
- pipeline2
- pipeline3
path:
- path1
- path2
- path3
I need to write loop in yaml so that it should download the pipeline1 artifacts to path1 and so on. Can someone please help??
Object-type parameters are your friend. They are incredibly powerful. As qBasicBoy answered, you'll want to make sure that you group the multiple properties together. If you're finding that you have a high number of properties per object, though, you can do a multi-line equivalent.
The following is an equivalent parameter structure to what qBasicBoy posted:
parameters:
- name: pipelines
type: object
default:
- Name: pipeline1
Path: path1
- Name: pipeline2
Path: path2
- Name: pipeline3
Path: path3
An example where you can stack many properties to a single object is as follows:
parameters:
- name: big_honkin_object
type: object
default:
config:
- appA: this
appB: is
appC: a
appD: really
appE: long
appF: set
appG: of
appH: properties
- appA: and
appB: here
appC: we
appD: go
appE: again
appF: making
appG: more
appH: properties
settings:
startuptype: service
recovery: no
You can, in essence, create an entire dumping ground for everything that you want to do by sticking it in one single object structure and properly segmenting everything. Sure, you could have had "startuptype" and "recovery" as separate string parameters with defaults of "service" and "no" respectively, but this way, we can pass a single large parameter from a high level pipeline to a called template, rather than passing a huge list of parameters AND defining said parameters in the template yaml scripts (remember, that's necessary!).
If you then want to access JUST a single setting, you can do something along the lines of:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
# Write your PowerShell commands here.
Write-Host "Apps start as a "${{ parameters.settings.startuptype }}
Write-Host "Do the applications recover? "${{ parameters.settings.recovery }}
This will give you the following output:
Apps start as a service
Do the applications recover? no
YAML and Azure Pipelines are incredibly powerful tools. I can't recommend enough going through the entire contents of learn.microsoft.com on the subject. You'll spend a couple hours there, but you'll come out the other end with an incredibly knowledge of how these pipelines can be tailored to do everything you could ever NOT want to do yourself!
Notable links that helped me a TON (only learned this a couple months ago):
How to work with the YAML language in Pipelines
https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema%2Cparameter-schema
How to compose expressions (also contains useful functions like convertToJSON for your object parameters!)
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/expressions?view=azure-devops
How to create variables (separate from parameters, still useful)
https://learn.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml
SLEEPER ALERT!!! Templates are HUGELY helpful!!!
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops
You could use an object with multiple properties
parameters:
- name: pipelines
type: object
default:
- { Name: pipeline1, Path: path1 }
- { Name: pipeline2, Path: path2 }
- { Name: pipeline3, Path: path3 }
steps:
- ${{each pipeline in parameters.pipelines}}:
# use pipeline.Name or pipeline.Path
I want to run a single stage in gitlab from a yml file that contains a lot of stages. I don't want to have to add this to every single stage to avoid running all the stages.
except:
refs:
- schedules
Instead of explicitly defining the except tag for each job. You can define it once as an anchor.
.job_template: &job_definition
except:
refs:
- schedules
test1:
<<: *job_definition
script:
- test1 project
If you don't want to add except in each job, use only instead of except
https://docs.gitlab.com/ee/ci/yaml/#onlyexcept-advanced
below there is an example with variables
only_with_variable:
script: ls -la
only:
variables:
- $VAR == "1234"
after that if you schedule a pipeline, you have the option to add variables to them.
in the example, you just need to add the VAR variable with value = 1234
You can use the following to run the stage only on a scheduled job
build-app:
stage: build-app
only:
- schedules
I'm trying to use variables in my gitlab-ci.yml file. This variable is passed as a parameter to a batch file that'll either only build or build and deploy based on parameter passed in. I've tried many different ways to pass my variable into the batch file but each time the variable is treated more like a static string instead.
I've read gitlabs docs on variables but cant seem to make it work.
- build
variables:
BUILD_PUBLISH_CONFIG_FALSE: 0
BUILD_PUBLISH_CONFIG_TRUE: 1
# BUILD ===============================
build: &build
stage: build
tags:
- webdev
script:
- ./build.bat %BUILD_CONFIG%
build:branch:
<<: *build
variables:
BUILD_CONFIG: $BUILD_PUBLISH_CONFIG_FALSE
only:
- /^(feature|hotfix|release)\/.+$/
build:branch:
<<: *build
variables:
BUILD_CONFIG: $BUILD_PUBLISH_CONFIG_TRUE
only:
- /^(stage)\/.+$/
build:branch:
<<: *build
variables:
BUILD_CONFIG: $BUILD_PUBLISH_CONFIG_TRUE
only:
- /^(master)\/.+$/
When watching gitlab's ci script execute, I expect ./build.bat 0, or ./build.bat 1.
Each time it prints out as ./build.bat %BUILD_CONFIG%
When you place variables inside job, that mean that you want to create new variable (and thats not correct way to do it). You want to output content of variable setup on top? Can u maybe add that to echo? or something like that? I didn't get it what you are trying to achieve.
https://docs.gitlab.com/ee/ci/variables/#gitlab-ciyml-defined-variables