GitHub Actions to use variables set from shell - shell

Goal:
In GitHub Actions, to define my commit message dynamically from shell:
- name: Commit changes
uses: EndBug/add-and-commit#v7
with:
message: "added on $(date -I)"
However, it seems that I have to define a environment variable then use it. I'm following How do I set an env var with a bash expression in GitHub Actions? and other help files like this, but still cannot tell how to make use of such environment variable that I've define previously. This is what I tried but failed:
- name: Checkout repo
uses: actions/checkout#v2
- run: |
touch sample.js
echo "today=$(date -I)" >> $GITHUB_ENV
- name: Commit changes
uses: EndBug/add-and-commit#v7
with:
message: "added on ${today}"
How to make it works?

If you want to reference an environment variable set using the $GITHUB_ENV environment file in the arguments to another task, you'll need to use workflow syntax to access the appropriate key of the top level env key, like this:
- name: Commit changes
uses: EndBug/add-and-commit#v7
with:
message: "added on ${{env.today}}"
You can access it as a standard environment from inside of a running task, for example:
- name: Show an environment variable
run: |
echo "today is $today"
In that example, the expression $today is expanded by the shell,
which looks up the environment variable named today. You could also
write:
- name: Show an environment variable
run: |
echo "today is ${{env.today}}"
In this case, the expansion would be performed by github's workflow
engine before the run commands execute, so the shell would see a
literal command that looks like echo "today is 2021-07-14".
You can accomplish something similar using output parameters, like this:
- name: "Set an output parameter"
id: set_today
run: |
echo "::set-output name=today::$(date -I)"
- name: Commit changes
uses: EndBug/add-and-commit#v7
with:
message: "added on ${{steps.set_today.outputs.today}}"
Using output parameters is a little more granular (because they are
qualified by the step id), and they won't show up in the environment
of processes started by your tasks.

Related

Pass file variable to gitlab job

I am having trouble with dynamically passing one of two file based variables to a job.
I have defined two file variables in my CI/CD settings that contain my helm values for deployments to developement and production clusters. They are typical yaml syntax, their content does not really matter.
baz:
foo: bar
I have also defined two jobs for the deployment that depend on a general deployment template .deploy.
.deploy:
variables:
DEPLOYMENT_NAME: ""
HELM_CHART_NAME: ""
HELM_VALUES: ""
before_script:
- kubectl ...
script:
- helm upgrade $DEPLOYMENT_NAME charts/$HELM_CHART_NAME
--install
--atomic
--debug
-f $HELM_VALUES
The specialization happens in two jobs, one for dev and one for prod.
deploy:dev:
extends: .deploy
variables:
DEPLOYMENT_NAME: my-deployment
HELM_CHART_NAME: my-dev-chart
HELM_VALUES: $DEV_HELM_VALUES # from CI/CD variables
deploy:prod:
extends: .deploy
variables:
DEPLOYMENT_NAME: my-deployment
HELM_CHART_NAME: my-prod-chart
HELM_VALUES: $PROD_HELM_VALUES # from CI/CD variables
The command that fails is the one in the script tag of .deploy. If I pass in the $DEV_HELM_VALUES or $PROD_HELM_VALUES, the deployment is triggered. However if I put in the $HELM_VALUES as described above, the command fails (Error: "helm upgrade" requires 2 arguments, which is very misleading).
The problem is that the $HELM_VALUES that are accessed in the command are already the resolved content of the file, whereas passing the $DEV_HELM_VALUES or the $PROD_HELM_VALUES directly works with the -f syntax.
This can be seen using echo in the job's output:
echo "$DEV_HELM_VALUES"
/builds/my-company/my-deployment.tmp/DEV_HELM_VALUES
echo "$HELM_VALUES"
baz:
foo: bar
How can I make sure the $HELM_VALUES only point to one of the files, and do not contain the files' content?

Bash - Gitlab CI not converting variable to a string

I am using GitLab to deploy a project and have some environmental variables setup in the GitLab console which I use in my GitLab deployment script below:
- export S3_BUCKET="$(eval \$S3_BUCKET_${CI_COMMIT_REF_NAME^^})"
- aws s3 rm s3://$S3_BUCKET --recursive
My environmental variables are declared like so:
Key: s3_bucket_development
Value: https://dev.my-bucket.com
Key: s3_bucket_production
Value: https://prod.my-bucket.com
The plan is that it grabs the bucket URL from the environmental variables depending on which branch is trying to deploy (CI_COMMIT_REF_NAME).
The problem is that the S3_BUCKET variable does not seem to get set properly and I get the following error:
> export S3_BUCKET=$(eval \$S3_BUCKET_${CI_COMMIT_REF_NAME^^})
> /scripts-30283952-2040310190/step_script: line 150: https://dev.my-bucket.com: No such file or directory
It looks like it picks up the environmental variable value fine but does not set it properly - any ideas why?
It seems like you are trying to get the value of the variables S3_BUCKET_DEVELOPMENT and S3_BUCKET_PRODUCTION based on the value of CI_COMMIT_REF_NAME, you can do this by using parameter indirection:
$ a=b
$ b=c
$echo "${!a}" # c
and in your case, you would need a temporary variable as well, something like this might work:
- s3_bucket_variable=S3_BUCKET_${CI_COMMIT_REF_NAME^^}
- s3_bucket=${!s3_bucket_variable}
- aws s3 rm "s3://$s3_bucket" --recursive
You are basically telling bash to execute command, named https://dev.my-bucket.com, which obviously doesn't exist.
Since you want to assign output of command when using VAR=$(command) you should probably use echo
export S3_BUCKET=$(eval echo \$S3_BUCKET_${CI_COMMIT_REF_NAME^^})
Simple test:
VAR=HELL; OUTPUT="$(eval echo "\$S${VAR^^}")"; echo $OUTPUT
/bin/bash
It dynamically creates SHELL variable, and then successfully prints it

How to add commands to before_script inherited from template

Let's say I have a template which contains something like this:
.some-scripts: &some-scripts |
set -e
function somefunction() {
}
.template-job:
before_script:
- *some-scripts
- echo "Example command"
- somefunction
build-job:
extends: .template-job
stage: build
script:
- mvn build
This template is included in another gitlab-ci.yml and I am looking to add some specific commands to the before_script of my build-job without overriding the before_script of the template-job. Is it possible and how?
I found what I was looking for, I needed to use a reference tag.
Here's what I came up with:
build-job:
stage: build
before_script:
- !reference [.template-job, before_script]
- mycommand
- mysecondcommand
script:
- mvn build
Currently you cannot extend a before_script, just overwrite it. But there is an open issue regarding the extending behavior.
As a workaround you could just add the additional commands to your script section as before_script, script and after_script are ultimatley merged together to one block on execution.
.template-job:
before_script:
- echo "Example command"
- echo "Second command"
build-job:
extends: .template-job
stage: build
script:
- echo "third command"
- echo "fourth command"
- mvn build

"gcloud builds submit" is not triggering error for missing required substitutions

I need some help with cloud build --substitutions.
This is the doc: https://cloud.google.com/cloud-build/docs/build-config#substitutions
Here is what is says:
cloudbuild.yaml
substitutions:
_SUB_VALUE: world
options:
substitution_option: 'ALLOW_LOOSE'
The following snippet uses substitutions to print "hello world." The ALLOW_LOOSE substitution option is set, which means the build will not return an error if there's a missing substitution variable or a missing substitution.
My case: I'm NOT using the ALLOW_LOOSE option. I need my substitutions to be required. I don't want any default values being applied. And I need it to fail immediately if I forget to pass any of the substitutions that I need.
Here is my cloudbuild.yaml file:
cloudbuild.yaml
substitutions:
_SERVER_ENV: required
_TAG_NAME: required
_MIN_INSTANCES: required
I'm initializing their default value as required specifically because I'm expecting the build call to fail if I forget to pass any of them to the gcloud builds submit call.
I'm expecting it to fail if I call gcloud builds submit and don't pass any of the defined substitutions. But it's not failing and the build completes normally without that value.
There is this observation in the docs:
Note: If your build is invoked by a trigger, the ALLOW_LOOSE option is set by default. In this case, your build will not return an error if there is a missing substitution variable or a missing substitution. You cannot override the ALLOW_LOOSE option for builds invoked by triggers.
But if I'm calling gcloud builds submit manually, that means that my build is not being invoked by any triggers, right? So the ALLOW_LOOSE options shouldn't be enabled.
Here is my full cloudbuild.yaml:
cloudbuild.yaml
steps:
- name: "gcr.io/cloud-builders/docker"
args:
- "build"
- "--build-arg"
- "SERVER_ENV=$_SERVER_ENV"
- "--tag"
- "gcr.io/$PROJECT_ID/server:$_TAG_NAME"
- "."
timeout: 180s
- name: "gcr.io/cloud-builders/docker"
args:
- "push"
- "gcr.io/$PROJECT_ID/server:$_TAG_NAME"
timeout: 180s
- name: "gcr.io/google.com/cloudsdktool/cloud-sdk"
entrypoint: gcloud
args:
- "beta"
- "run"
- "deploy"
- "server"
- "--image=gcr.io/$PROJECT_ID/server:$_TAG_NAME"
- "--platform=managed"
- "--region=us-central1"
- "--min-instances=$_MIN_INSTANCES"
- "--max-instances=3"
- "--allow-unauthenticated"
timeout: 180s
images:
- "gcr.io/$PROJECT_ID/server:$_TAG_NAME"
substitutions:
_SERVER_ENV: required
_TAG_NAME: required
_MIN_INSTANCES: required
In your cloudbuild.yaml file, when you define a substituions variable you automatically set his default value
substitutions:
# Value = "required"
_SERVER_ENV: required
# Value = ""
_TAG_NAME:
Try to use a variable that is not defined in the substitutions array, such as:
steps:
- name: "gcr.io/google.com/cloudsdktool/cloud-sdk"
entrypoint: bash
args:
- -c
- |
# print "required"
echo $_SERVER_ENV
# print nothing
echo $_TAG_NAME
# Error, except if you allow loose. In this case, print nothing
echo $_MIN_INSTANCES
substitutions:
_SERVER_ENV: required
_TAG_NAME:

How to use variables in gitlab-ci.yml file

I'm trying to use variables in my gitlab-ci.yml file. This variable is passed as a parameter to a batch file that'll either only build or build and deploy based on parameter passed in. I've tried many different ways to pass my variable into the batch file but each time the variable is treated more like a static string instead.
I've read gitlabs docs on variables but cant seem to make it work.
- build
variables:
BUILD_PUBLISH_CONFIG_FALSE: 0
BUILD_PUBLISH_CONFIG_TRUE: 1
# BUILD ===============================
build: &build
stage: build
tags:
- webdev
script:
- ./build.bat %BUILD_CONFIG%
build:branch:
<<: *build
variables:
BUILD_CONFIG: $BUILD_PUBLISH_CONFIG_FALSE
only:
- /^(feature|hotfix|release)\/.+$/
build:branch:
<<: *build
variables:
BUILD_CONFIG: $BUILD_PUBLISH_CONFIG_TRUE
only:
- /^(stage)\/.+$/
build:branch:
<<: *build
variables:
BUILD_CONFIG: $BUILD_PUBLISH_CONFIG_TRUE
only:
- /^(master)\/.+$/
When watching gitlab's ci script execute, I expect ./build.bat 0, or ./build.bat 1.
Each time it prints out as ./build.bat %BUILD_CONFIG%
When you place variables inside job, that mean that you want to create new variable (and thats not correct way to do it). You want to output content of variable setup on top? Can u maybe add that to echo? or something like that? I didn't get it what you are trying to achieve.
https://docs.gitlab.com/ee/ci/variables/#gitlab-ciyml-defined-variables

Resources