Is it possible, or is there a way, to use a yaml anchor reference in a BASH if clause.
If so, how?
This is what I'm attempting so far.
create-cluster:
needs:
- terra-bootstrap
script:
- export TF_VAR_state_bucket_prefix="${TF_VAR_vsad}/${TF_VAR_cluster_name}"
- pushd terra-cluster
- *init_with_gcs_state
- |
if [[ "${CLUSTER_EXISTS}" == 'false' ]]; then
terraform apply -auto-approve
*does_cluster_exist
fi
- popd
stage: create-cluster
tags:
- gke
No, the YAML spec does not allow you to do this. YAML anchors cannot be used within a string (or any other scalar).
A better approach for GitLab CI YAML would be to define your reusable bash steps as functions then utilize those within your job script logic as-needed.
For example, you can define a function does_cluster_exist and reference it using anchors/reference/etc.
.helper_functions:
script: |
function cluster_exists() {
cluster="$1"
# ... complete the function logic
}
function another_function() {
return 0
}
another_job:
before_script:
# ensure functions are defined
- !reference [.helper_functions, script]
script:
# ...
# use the functions in an `if` or wherever...
- |
if cluster_exists "$cluster"; then
another_function
else
echo "fatal, cluster does not exist" > /dev/stderr
exit 1
fi
Related
I'm trying to use variables inside variables in azure pipelines.
Below is an example of the bash script:
#!/bin/bash
customer=google
environment=preprod
android_google_preprod_account_activation_url=preprod.google.com
echo "Customer is $customer"
echo "Environment is $environment"
var1=android_${customer}_${environment}_account_activation_url
echo "variable is $var1"
echo "original value is ${!var1}"
I get the expected output for the above bash script when I run it on my Ubuntu server, with NO errors:
Customer is google
Environment is preprod
variable is android_google_preprod_account_activation_url
original value is preprod.google.com
The yml code for azure pipelines is:
parameters:
- name: customer
displayName: 'select customer'
type: string
values:
- google
- name: environment
displayName: 'select environment'
type: string
values:
- preprod
variables:
- group: android-${{ parameters.customer }}-${{ parameters.environment }}
- name: var1
value: android-${{ parameters.customer }}-${{ parameters.environment }}-account-activation-url
script: |
echo "Customer is $(customer)"
echo "Environment is $(environment)"
echo "variable is $(var1)"
echo "original value is $(!var1)"
displayName: 'echo variables'
The value of android-google-preprod-account-activation-url is being taken from variable groups inside library.
It gives me an error for the 4th line:
invalid indirect expansion
The first 3 lines output is as expected.
Expected output is:
Customer is google
Environment is preprod
variable is android_google_preprod_account_activation_url
original value is preprod.google.com
Is there a different syntax that needs to be followed in azure pipelines?
I`m not a bash expert ))) however... you're trying to use the parameters expansion What is indirect expansion? What does ${!var*} mean?
But it refers to the bash variables.... when you define variables in the devops pipeline, you have to use them as environment variables or through the macro.
or something like that:
android_google_preprod_account_activation_url=preprod.google.com
echo "Customer is $(customer)"
echo "Environment is $(environment)"
var1=android_$(customer)_$(environment)_account_activation_url
echo "variable is $var1"
echo "original value is ${!var1}"
The macro syntax "$(varName)" is a proprietary syntax in Azure Pipelines to interpolate variable values. It is processed during runtime and different with the syntax "${varName}" in Bash scripts.
For your case, you can try to use the compile time syntax "${{ variables.varName }}" to get the value in the pipeline.
echo "original value is $(${{ variables.var1 }})"
With above change, after you triggered the pipeline:
At the compile time, the expression "${{ variables.var1 }}" will be replaced with the actual value "android_google_preprod_account_activation_url". So, the expression "$(${{ variables.var1 }})" will be changed to "$(android_google_preprod_account_activation_url)".
Then at the runtime, the expression will be parsed as the correct value "preprod.google.com".
Below is an example I have tested on my side.
YAML
variables:
android_google_preprod_account_activation_url: 'preprod.google.com'
var1: 'android_google_preprod_account_activation_url'
jobs:
- job: A
displayName: 'Job A'
pool:
vmImage: ubuntu-latest
steps:
- checkout: none
- task: Bash#3
displayName: 'Print variables'
inputs:
targetType: inline
script: |
echo "android_google_preprod_account_activation_url = $(android_google_preprod_account_activation_url)"
echo "var1 = $(var1)"
echo "original value = $(${{ variables.var1 }})"
Result
For more details, you can reference the related document "Understand variable syntax".
I have job in my gitlab-cicd.yml file:
unit_test:
stage: test
image: $MAVEN_IMAGE
script:
- *tests_variables_export
- mvn ${MAVEN_CLI_OPTS} clean test
- cat ${CI_PROJECT_DIR}/rest-service/target/site/jacoco/index.html
- cat ${CI_PROJECT_DIR}/soap-service/target/site/jacoco/index.html
artifacts:
expose_as: 'code coverage'
paths:
- ${CI_PROJECT_DIR}/soap-service/target/surefire-reports/
- ${CI_PROJECT_DIR}/rest-service/target/surefire-reports/
- ${CI_PROJECT_DIR}/soap-service/target/site/jacoco/index.html
- ${CI_PROJECT_DIR}/rest-service/target/site/jacoco/index.html
And I want to change it to this one:
unit_test:
stage: test
image: $MAVEN_IMAGE
script:
- *tests_variables_export
- mvn ${MAVEN_CLI_OPTS} clean test
- cat ${CI_PROJECT_DIR}/rest-service/target/site/jacoco/index.html
- cat ${CI_PROJECT_DIR}/soap-service/target/site/jacoco/index.html
artifacts:
expose_as: 'code coverage'
paths:
- *resolve_paths
I try to use this bash script:
.resolve_paths: &resolve_paths |-
if [ "${MODULE_FIRST}" != "UNKNOWN" ]; then
echo "- ${CI_PROJECT_DIR}/${MODULE_FIRST}/target/surefire-reports/"
echo "- ${CI_PROJECT_DIR}/${MODULE_FIRST}/target/site/jacoco/index.html"
fi
if [ "${MODULE_SECOND}" != "UNKNOWN" ]; then
echo "- ${CI_PROJECT_DIR}/${MODULE_SECOND}/target/surefire-reports/"
echo "- ${CI_PROJECT_DIR}/${MODULE_SECOND}/target/site/jacoco/index.html"
fi
And right now I'm getting this error in pipeline:
WARNING: if [ "rest-service" != "UNKNOWN" ]; then\n echo "- /builds/minv/common/testcommons/taf-api-support/rest-service/target/surefire-reports/"\n echo "- /builds/minv/common/testcommons/taf-api-support/rest-service/target/site/jacoco/index.html"\nfi\nif [ "soap-service" != "UNKNOWN" ]; then\n echo "- /builds/minv/common/testcommons/taf-api-support/soap-service/target/surefire-reports/"\n echo "- /builds/minv/common/testcommons/taf-api-support/soap-service/target/site/jacoco/index.html"\nfi: no matching files ERROR: No files to upload
Can I execute [sic] paths using bash script like this?
No, scripts cannot alter the current YAML, particularly not if you specify the script (which is just a string) in a place where it is interpreted as a path.
You could trigger a dynamically generated YAML:
generate:
stage: build
script:
- |
exec > generated.yml
echo ".resolve_paths: &resolve_paths"
for module in "${MODULE_FIRST}" "${MODULE_SECOND}"; do
[[ "$module" = UNKNOWN ]] && continue
echo "- ${CI_PROJECT_DIR}/${module}/target/surefire-reports/"
echo "- ${CI_PROJECT_DIR}/${module}/target/site/jacoco/index.html"
done
sed '1,/^\.\.\. *$/d' "${CI_CONFIG_PATH}"
artifacts:
paths:
- generated.yml
run:
stage: deploy
trigger:
include:
- artifact: generated.yml
job: generate
...
# Start of actual CI. When this runs, there will be an
# auto-generated job `.resolve_paths: &resolve_paths`.
# Put the rest of your CI (e.g. `unit_test:`) here.
But there are so many extensions in GitLab's YAML that you likely will find a tremendously better solution, which depends on what you plan to do with .resolve_paths. Maybe have a look at
artifacts:exclude
additional jobs with rules:
I want to create some nested conditions: i need this pipeline to work when it is a merge or merge request and with certain name start "feature". So, is there an AND condition in the 'only' option for jobs?
No there is not. You must use rules.
test:
stage: test
script:
- echo "test"
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TITLE =~ /^feature/'
I have several stages in my gitlab-ci.yml file,
For example:
stages:
- one
- two
- three
Is there possibility to run specified stages? In Makefile it's called "targets". I want in different cases execute different stages. For example:
if [ <condition 1> ]; then
run all stages;
if [ <condition 2> ]; then
run stage "one";
if [ <condition 3> ]; then
run stages "two" and "three";
fi
You can do it at the job level using if rules.
Also, a stage can contain only one job. So, create 3 jobs in your .gitlab-ci.yml, one per stage and configure rules like this (check the doc for more example) ex :
stages:
- one
- two
- three
job_one:
stage: one
script: "echo Hello, stage one"
rules:
- if: '$VAR == "string value"'
job_two:
stage: two
script: "echo Hello, stage two"
rules:
- if: '$VAR == "string value"'
job_three:
stage: three
script: "echo Hello, stage three"
rules:
- if: '$VAR == "string value"'
I'm trying to build a Jenkins Pipeline for which a parameter is
optional:
parameters {
string(
name:'foo',
defaultValue:'',
description:'foo is foo'
)
}
My purpose is calling a shell script and providing foo as argument:
stages {
stage('something') {
sh "some-script.sh '${params.foo}'"
}
}
The shell script will do the Right Thing™ if the provided value is the empty
string.
Unfortunately I can't just get an empty string. If the user does not provide
a value for foo, Jenkins will set it to null, and I will get null
(as string) inside my command.
I found this related question but the only answer is not really helpful.
Any suggestion?
OP here realized a wrapper script can be helpful… I ironically called it junkins-cmd and I call it like this:
stages {
stage('something') {
sh "junkins-cmd some-script.sh '${params.foo}'"
}
}
Code:
#!/bin/bash
helpme() {
cat <<EOF
Usage: $0 <command> [parameters to command]
This command is a wrapper for jenkins pipeline. It tries to overcome jenkins
idiotic behaviour when calling programs without polluting the remaining part
of the toolkit.
The given command is executed with the fixed version of the given
parameters. Current fixes:
- 'null' is replaced with ''
EOF
} >&2
trap helpme EXIT
command="${1:?Missing command}"; shift
trap - EXIT
typeset -a params
for p in "$#"; do
# Jenkins pipeline uses 'null' when the parameter is undefined.
[[ "$p" = 'null' ]] && p=''
params+=("$p")
done
exec $command "${params[#]}"
Beware: prams+=("$p") seems not to be portable among shells: hence this ugly script is running #!/bin/bash.