(Added bash and terminal tags since I'm unsure if my issue is specific to Github actions specifically or if instead is a misunderstanding on how env vars work more generally)
I'm working on a workflow.yml and in a step "Env substitue in sql script" am trying to set some env vars:
on: [push]
env:
GAME: "FunGame"
TRAIN_HORIZON: 7
jobs:
ssql:
runs-on: ubuntu-latest
name: Get data
steps:
- name: Checkout cum-rev repo
uses: actions/checkout#v2 # Defaults to current repo - check out current repo
- name: Checkout ds-ssql-gh-action
uses: actions/checkout#v2
with:
repository: ourorg/ds-ssql-gh-action
token: ${{ secrets.cumrev_workflow_token }}
ref: main
path: './ds-ssql-gh-action'
- name: Env substitue in sql script
run: |
INSTALL_DATE=$(date -d "`date +%Y%m01` -12 month" +%Y-%m-%d)
echo "Here is install date $INSTALL_DATE"
IOS_GAME="${{ env.GAME }}_IOS_PROD"
ANDROID_GAME="${{ env.GAME }}_ANDROID_PROD"
envsubst < get-data/training-data.sql
cat get-data/training-data.sql
printenv
After pushing this the job attempts to run. I printenv at the bottom and when I see the env variables, I don't see any of INSTALL_DATE, IOS_GAME or ANDROID_GAME.
Why are those env variables not being set with the lines:
INSTALL_DATE=$(date -d "`date +%Y%m01` -12 month" +%Y-%m-%d)
echo "Here is install date $INSTALL_DATE"
IOS_GAME="${{ env.GAME }}_IOS_PROD"
ANDROID_GAME="${{ env.GAME }}_ANDROID_PROD"
Note line echo "Here is install date $INSTALL_DATE" does indeed print out the correct value as expected. But it's not showing when I run printenv?
You have to export the variables you want to see in the environment:
export INSTALL_DATE=$(date -d "`date +%Y%m01` -12 month" +%Y-%m-%d)
...
Related
Under my Git Repo Settings > Environments : I define an env name PPM_DEV, under this env PPM_DEV I define an environment variable named HOSTNAME and give it a value in the git configurations page.
Now what is the YAML syntax under GitActions WF to read the value of this variable ?
basically I may have 2 Env's defined PPM_DEV , PPM_TEST
but where do I set the Env context to pull the variable HOSTNAME from the PPM_DEV env ?
In the example below , I am trying to populate a variable VARHOSTNAME with the value of the Env variable HOSTNAME that is pre-defined against the Env named PPM_DEV
However it fails with
The workflow is not valid. .github/workflows/Ext_Conn_v2.yml (Line: 10, Col: 7): Unexpected value 'VARHOSTNAME'
name: Ext_Conn_v2
on:
workflow_dispatch:
jobs:
RunonVM:
runs-on: ubuntu-latest
environment:
name: ppm_dev
VARHOSTNAME: ${{ env.HOSTNAME }}
steps:
- name: Run a command
run: |
echo "This workflow was manually triggered."
echo "value of the variable HOSTNAME: " ${VARHOSTNAME}
pwd
echo "end of run"
You need to use jobs.<job_id>.steps[*].env to specify the environment variables or secrets using vars or secrets contexts:
Here's an example with the secrets context:
jobs:
job:
runs-on: ubuntu-latest
environment: ppm_dev
steps:
- name: Command
env:
HOSTNAME: ${{ secrets.HOSTNAME }}
run: |
echo "HOSTNAME: $HOSTNAME"
See a linted example here.
I have scoured the forums and couldn't find a solution.
I have a config.yml file which contains a set of key/value pairs. One of those pairs I want to set as a GITHUB_ENV to be used in my workflow. But I am running into issues as the logs say
"reusable workflows should be referenced at the top-level
`jobs.*.uses' key, not within steps"
How do I get around this?
name: "Deploy The Kraken"
on:
push:
branches:
- dev
pull_request:
branches:
- dev
workflow_dispatch:
jobs:
call-build:
steps:
- name: Set env
run: |
echo "FEATURE_BRANCH=$(cat config.yml | awk -F: '/^branch:/ { print $2 }'
| sed 's/ //g')" >> $GITHUB_ENV
- name: Test
run: echo $FEATURE_BRANCH
- uses: ######/#####/.github/workflows/kraken.yml#$FEATURE_BRANCH
with:
environment: #####
workspace: #####
contract: #####
production-ref: #####
I have tried multiple variations of placing the shell command in different parts. But I still can't get it to point to my chosen branch.
So I have two secrets: DEV_SERVER_IP and MASTER_SERVER_IP.
in main.yml I need something like this
run: echo "::set-env name=BRANCH_NAME::$(echo ${GITHUB_REF#refs/heads/} | sed 's/\//_/g')"
run: ssh-keyscan -H ${{ secrets.BRANCH_NAME_SERVER_IP }} >> ~/.ssh/known_hosts
but am getting error
env:
BRANCH_NAME: dev
Error: Input required and not supplied: key
I need here something like this ssh-keyscan -H ${{ secrets.${BRANCH_NAME}_SERVER_IP }}
how can I fix this?
You're trying to use shell style logic inside a Github context
expansion (${{ ... }}) which won't work. Just move all your logic
into your shell script instead:
name: Example
on:
push:
jobs:
example:
runs-on: ubuntu-latest
steps:
- name: get target ip
env:
DEV_SERVER_IP: ${{ secrets.DEV_SERVER_IP }}
MAIN_SERVER_IP: ${{ secrets.MAIN_SERVER_IP }}
run: |
branch_name=$(sed 's|/|_|g' <<< ${GITHUB_REF#refs/heads/})
target="${branch_name^^}_SERVER_IP"
mkdir -p ~/.ssh
ssh-keyscan -H ${!target} >> ~/.ssh/known_hosts
cat ~/.ssh/known_hosts
In the above workflow, the expression ${branch_name^^} is a bash expression that returns the value of $branch_name in uppercase, and ${!target} is a bash expression that returns the value of the variable who name is stored in $target.
Note that I'm not using your "set the BRANCH_NAME environment variable"
task because the ::set-env command is disabled by default for
security reasons.
I'm not able to access environment variables defined at the top-level of a GitHub Action configuration file from within a script run by the action.
For example, given the following config file:
name: x-pull-request
on: pull_request
env:
FOO: bar
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v1
- name: does a thing
run: ./scripts/do-a-thing.sh
... and the following script:
X=${FOO:default}
echo "X: $X" # X: default
The FOO environment variable defined in the config file is not available to the script and the default value is being used.
So, how can I access the environment variable from a Bash script run by the build step? Am I missing a prefix or something? (I know values defined in the input hash require you to use an INPUT_ prefix when referencing them.)
You can use env at any level also in jobs and steps.
I wrote a test action and a test script to validate it:
The action file:
name: Env tests
on: push
env:
FOO_ROOT: bar on root
jobs:
test:
runs-on: ubuntu-latest
env:
FOO_JOB: bar on job
steps:
- uses: actions/checkout#v1
- name: Test envs
run: ./env-test.sh
env:
FOO_STEP: bar on step
The script file:
#!/usr/bin/env bash
echo "FOO_ROOT: $FOO_ROOT"
echo "FOO_JOB: $FOO_JOB"
echo "FOO_STEP: $FOO_STEP"
echo " "
printenv
The results:
FOO_ROOT: bar on root
FOO_JOB: bar on job
FOO_STEP: bar on step
LEIN_HOME=/usr/local/lib/lein
M2_HOME=/usr/share/apache-maven-3.6.3
...
Check my results and, in fact, I don't know why it didn't work on your side because it must work.
If some one else is searching for a solution you have to explicitly pass the environment variables to the bash script. Otherwise they are not available:
name: x-pull-request
on: pull_request
env:
FOO: bar
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v1
- name: does a thing
run: ./scripts/do-a-thing.sh $FOO
and in the script you have to map the parameter to a variable.
X=$1
echo "X: $X" # X: default
I'm trying to build an application using GitLab CI.
The name of the generated file is depending on the time, in this format
DEV_APP_yyyyMMddhhmm
(example: DEV_APP_201810221340, corresponding to the date of today 2018/10/22 13h40).
How can I store this name in a global variable inside the .gitlab-ci.yml file?
Here is my .gitlab-ci.yml file:
image: docker:latest
image: docker:latest
services:
- docker:dind
variables:
DOCKER_DRIVER: overlay
SPRING_PROFILES_ACTIVE: gitlab-ci
# TIME: ""
# BRANCH: ""
# REC_BUILD_NAME: ""
TIME: "timex"
BRANCH: "branchx"
DEV_BUILD_NAME: "DEV_APP_x"
stages:
- preparation
- build
- package
- deploy
- manual_rec_build
- manual_rec_package
job_preparation:
stage: preparation
script:
- echo ${TIME}
- export TIME=$(date +%Y%m%d%H%M)
- "BRANCH=$(echo $CI_BUILD_REF_SLUG | sed 's/[^[[:alnum:]]/_/g')"
- "DEV_BUILD_NAME=DEV_APP_${BRANCH}_${TIME}"
- echo ${TIME}
maven-build:
image: maven:3-jdk-8
stage: build
script:
- echo ${TIME}
- "mvn package -B"
artifacts:
paths:
- target/*.jar
only:
- merge-requests
- /^feature\/sprint.*$/
- /^DEV_.*$/
# when: manual
docker-build:
stage: package
script:
- echo ${TIME}
- docker build -t registry.gitlab.com/mourad.sellam/actuator-simple .
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN registry.gitlab.com
- docker push registry.gitlab.com/mourad.sellam/actuator-simple
only:
- merge-requests
- /^feature\/sprint.*$/
- /^DEV_.*$/
when: manual
k8s-deploy-production:
image: google/cloud-sdk
stage: deploy
script:
- echo ${TIME}
- echo "$GOOGLE_KEY" > key.json
- gcloud auth activate-service-account --key-file key.json
- gcloud config set compute/zone europe-west1-c
- gcloud config set project actuator-sample
- gcloud config set container/use_client_certificate True
- gcloud container clusters get-credentials actuator-example
- kubectl delete secret registry.gitlab.com
- kubectl create secret docker-registry registry.gitlab.com --docker-server=https://registry.gitlab.com --docker-username=myUserName--docker-password=$REGISTRY_PASSWD --docker-email=myEmail#gmail.com
- kubectl apply -f deployment.yml --namespace=production
environment:
name: production
url: https://example.production.com
when: manual
job_manual_rec_build:
image: maven:3-jdk-8
stage: manual_rec_build
script:
- echo ${TIME}
- "mvn package -B"
artifacts:
paths:
- target/*.jar
when: manual
# allow_failure: false
job_manual_rec_package:
stage: manual_rec_package
variables:
script:
- echo ${TIME}
- echo ${DEV_BUILD_NAME}
- docker build -t registry.gitlab.com/mourad.sellam/actuator-simple:${DEV_BUILD_NAME} .
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN registry.gitlab.com
- docker push registry.gitlab.com/mourad.sellam/actuator-simple
artifacts:
paths:
- target/*.jar
when: on_success
#test 1
When I call
echo ${TIME}
It displays "timex".
echo faild
Could you tell me how to store a global variable and set it in each job?
Check if GitLab 13.0 (May 2020) could help in your case:
Inherit environment variables from other jobs
Passing environment variables (or other data) between CI jobs is now possible.
By using the dependencies keyword (or needs keyword for DAG pipelines), a job can inherit variables from other jobs if they are sourced with dotenv report artifacts.
This offers a more graceful approach for updating variables between jobs compared to artifacts or passing files.
See documentation and issue.
You can inherit environment variables from dependent jobs.
This feature makes use of the artifacts:reports:dotenv report feature.
Example with dependencies keyword.
build:
stage: build
script:
- echo "BUILD_VERSION=hello" >> build.env
artifacts:
reports:
dotenv: build.env
deploy:
stage: deploy
script:
- echo $BUILD_VERSION # => hello
dependencies:
- build
Example with the needs keyword:
build:
stage: build
script:
- echo "BUILD_VERSION=hello" >> build.env
artifacts:
reports:
dotenv: build.env
deploy:
stage: deploy
script:
- echo $BUILD_VERSION # => hello
needs:
- job: build
artifacts: true
You can use artifacts for passing data between jobs. Here's example from Flant to check previous pipeline manual decision:
approve:
script:
- mkdir -p .ci_status
- echo $(date +%s) > .ci_status/approved
artifacts:
paths:
- .ci_status/
NOT approve:
script:
- mkdir -p .ci_status
- echo $(date +%s) > .ci_status/not_approved
artifacts:
paths:
- .ci_status/
deploy to production:
script:
- if [[ $(cat .ci_status/not_approved) > $(cat .ci_status/approved) ]]; then echo "Need approve from release engineer!"; exit 1; fi
- echo "deploy to production!"
There's an open issue 47517 'Pass variables between jobs' on Gitlab CE..
CI/CD often needs to pass information from one job to another and
artifacts can be used for this, although it's a heavy solution with
unintended side effects. Workspaces is another proposal for passing
files between jobs. But sometimes you don't want to pass files at all,
just a small bit of data.
I have faced the same issue, and workaround this by storing DATA in file, then access to it in other Jobs..
You kind of can....The way I went about it:
You can send a POST request to the project to save a variable:
export VARIABLE=secret
curl --request POST --header "PRIVATE-TOKEN: $CI_ACCESS_TOKEN" "https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/variables/" --form "key=VARIABLE" --form "value=$VARIABLE"
and cleanup after the work/trigger is finished
curl --request DELETE --header "PRIVATE-TOKEN: $CI_ACCESS_TOKEN" "https://gitlab.seznam.net/api/v4/projects/$CI_PROJECT_ID/variables/VARIABLE"
I'm not sure it suppose to be used this way, but it does the trick. You have the variable accessible for all the following jobs (specially when you use trigger and script in that job is not an option.)
Just please make sure, you run the cleanup job even if previous once fail...