I have a project where I'm using GitHub Actions for the CI / CD. I have a shell script where I would like to inject environment variables from my Actions yml. Here is what I have so far:
- name: docker-push
env:
USER: joesan
SOME_GITHUB_REPO_NAME: github-repo-name
GH_REPO: github.com/$USER/$SOME_GITHUB_REPO_NAME
run: |
echo "Running sbt assembly"
echo $GITHUB_REF
echo "Pushing tag into Docker Registry"
sh ./scripts/tag_deployment.sh
In the tag_deployment.sh, I'm trying to use these variables:
....
....
git clone https://${GH_REPO}
cd "${SOME_GITHUB_REPO_NAME}"
....
....
But I see when the action is run that it does not print what was set!
Cloning into '$SOME_GITHUB_REPO_NAME'...
fatal: unable to access 'https://github.com/$USER/$SOME_GITHUB_REPO_NAME/': The requested URL returned error: 400
How can I inject these environment variables properly into my shell script?
It was not so intuitive to understand from the GitHub documentation. I managed to fix this by moving the environment variables globally like this:
env:
USER: joesan
DEPLOYMENT_REPO_NAME: some-repo-name
And then use it in my some other job definition like this:
- name: docker-push
env:
GH_REPO: github.com/${{ env.USER }}/${{ env.DEPLOYMENT_REPO_NAME }}
Related
I have something like this in my .gitlab-ci.yml file:
.docker_build_cmd: &docker_build_cmd >-
docker build -t $DOCKER_IMAGE_TAG -f $DOCKER_FILE .
variables:
PROJECT_NAME: best-app-ever
BUILD_VERSION: 4.2.0
DOCKER_IMAGE_TAG: $DOCKER_PROJECT_NAME:$BUILD_VERSION
build:
stage: build
variables:
GRADLE_CMD: app:build
DOCKER_PROJECT_NAME: $PROJECT_NAME-service
DOCKER_FILE: docker/service.dockerfile
script:
- echo $DOCKER_PROJECT_NAME
- echo $DOCKER_IMAGE_TAG
- *docker_build_cmd
but runner says:
$ echo $DOCKER_PROJECT_NAME
best-app-ever
$ echo $DOCKER_IMAGE_TAG
$PROJECT_NAME-service:4.2.0
$ docker build -t $DOCKER_IMAGE_TAG -f $DOCKER_FILE .
invalid argument "$PROJECT_NAME-service:4.2.0" for "-t, --tag" flag: invalid reference format: repository name must be lowercase
Does anyone understand why it happens?
How Gitlab processes the .yml?
And if there is a way to make it work as I intended, I am really want to know.
GitLab SaaS version is 14.3.0-ee.
UPD. After some googling it seems that "variable_inside_variable" feature flag has to be enabled. Will do it tomorrow and see what changes.
GitLab CI "variable inside variable" feature was disabled, so I asked admins to toggle it and everything became fine. Don't think that this could be helpful question for many, because GitLab already enabled the feature globally, but if you have a standalone solution and the same issue, then you know what to do.
I would like to save gradle-based project's version into a variable in gitlab-ci script. In my build.gradle I have:
tasks.register('version') {
doLast {
println(version)
}
}
It reads version from gradle.properties (let's say version=0.1) and returns it.
I execute it as gradlew version -q so I get only result, with no unnecessary output. When using unix-style variable creation of command result, that is: version=$(./gradlew version -q), the runner ends script. Is it possible to save the output into a variable for script?
My .gitlab-ci.yml:
image: gradle:jdk11
cache: &wrapper
paths:
- .gradle/wrapper
- .gradle/caches
before_script:
- export GRADLE_USER_HOME=`pwd`/.gradle
- chmod a+x gradlew
stages:
- prepare
- build
- deploy
wrapper:
stage: prepare
script:
- gradle wrapper
compile:
stage: build
script:
- ./gradlew assemble
artifacts:
paths:
- build/classes/**
- build/libs/*.jar
cache:
<<: *wrapper
policy: pull
properties:
stage: deploy
script:
- eval version=$(./gradlew version -q)
- echo $version # not even called
I also tried to omit eval to have version=$(./gradlew version -q) in script, but nothing changes.
CI output:
$ export GRADLE_USER_HOME=`pwd`/.gradle
$ chmod a+x gradlew
$ version=$(./gradlew version -q)
Cleaning up file based variables
Ok, I've found the solution. I mustn't use variable, simply. It's needed to directly pass evaluation to another command, using double quotes, like:
.gitlab-ci.yml (part):
properties:
stage: deploy
script:
- echo "$(./gradlew version -q)"
It started to work
I am in the Debian9. And sudo su to become root user. After that I run the command gitlab-runner register. But for some reasons. I got the isolated environment not the same like when I login to the server over ssh
gitlab-runner --version
Version: 10.8.0
Git revision: 079aad9e
Git branch:
GO version: go1.8.7
Built: 2018-05-22T03:24:56+00:00
OS/Arch: linux/amd64
When I put env in the before_script I found the different environment.
Then I put source ~/.bashrc to it. Also does not make any change.
How to let shell runner use my normal login user environement?
Update:
When I execute env (by putting in before_script) I got this output
$ env
CI_RUNNER_EXECUTABLE_ARCH=linux/amd64
CI_COMMIT_TITLE=add config
CI_JOB_TOKEN=xxxxxxxxxxxxxxxxxxxx
CI_BUILD_REF_NAME=master
CI_REGISTRY_PASSWORD=xxxxxxxxxxxxxxxxxxxx
CI_RUNNER_TAGS=provider, oauth, mp
CI_SHARED_ENVIRONMENT=true
CI_JOB_NAME=test
CI_SERVER_VERSION=10.8.4
DJANGO_SETTINGS_MODULE=config.settings.local
LANG=en_US.UTF-8
GITLAB_CI=true
CI_SERVER_REVISION=2268d0c
CI_PROJECT_VISIBILITY=private
OLDPWD=/home/gitlab-runner
INVOCATION_ID=7dd516612949410c8ca8e4e59696f9fd
CI_COMMIT_SHA=9030892858b5ca92c7b36a81f573f187e6d14090
CI_COMMIT_MESSAGE=add config
CI_BUILD_STAGE=test
CI_PROJECT_URL=https://mbx-git.magicboxasia.com/sarit/mhu_ped_oauth_provider
CI_COMMIT_REF_SLUG=master
CI_SERVER_NAME=GitLab
CI_RUNNER_VERSION=10.8.0
CI_BUILD_NAME=test
CI=true
XDG_SESSION_ID=c5
CI_REGISTRY_USER=gitlab-ci-token
USER=gitlab-runner
CI_PROJECT_ID=96
CI_PIPELINE_ID=4399
CI_COMMIT_DESCRIPTION=
PWD=/home/gitlab-runner/builds/f304ea76/0/sarit/mhu_ped_oauth_provider
GITLAB_FEATURES=
HOME=/home/gitlab-runner
JOURNAL_STREAM=8:19161
CI_REGISTRY=docker-registry.magicboxasia.com
CI_BUILD_TOKEN=xxxxxxxxxxxxxxxxxxxx
CI_BUILD_ID=5514
CONFIG_FILE=/etc/gitlab-runner/config.toml
GITLAB_USER_NAME=Sarit Ritwirune
CI_PROJECT_PATH_SLUG=sarit-mhu-ped-oauth-provider
DATABASE_URL=postgres://postgres:postgres#postgres:5432/mp_oauth_provider
POSTGRES_DB=poinkdb
GITLAB_USER_EMAIL=sarit#magicboxasia.com
CI_COMMIT_REF_NAME=master
CI_REGISTRY_IMAGE=docker-registry.magicboxasia.com/sarit/mhu_ped_oauth_provider
CI_SERVER_TLS_CA_FILE=/home/gitlab-runner/builds/f304ea76/0/sarit/mhu_ped_oauth_provider.tmp/CI_SERVER_TLS_CA_FILE
CI_RUNNER_ID=42
CI_SERVER=yes
CI_JOB_ID=5514
CI_REPOSITORY_URL=https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx#mbx-git.magicboxasia.com/sarit/mhu_ped_oauth_provider.git
MAIL=/var/mail/gitlab-runner
SHELL=/bin/bash
GITLAB_USER_LOGIN=sarit
CI_RUNNER_REVISION=079aad9e
CI_CONFIG_PATH=.gitlab-ci.yml
CI_PROJECT_NAME=mhu_ped_oauth_provider
POSTGRES_PASSWORD=postgres
POSTGRES_USER=postgres
SHLVL=2
CI_RUNNER_DESCRIPTION=MP OAuth Provider
CI_PROJECT_PATH=sarit/mhu_ped_oauth_provider
LOGNAME=gitlab-runner
XDG_RUNTIME_DIR=/run/user/999
GITLAB_USER_ID=39
CI_BUILD_BEFORE_SHA=e36e718d09f89095972ddf60ebf1fc1931282ae4
CI_BUILD_REF=9030892858b5ca92c7b36a81f573f187e6d14090
PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
CI_PIPELINE_SOURCE=push
CI_PROJECT_NAMESPACE=sarit
CI_PROJECT_DIR=/home/gitlab-runner/builds/f304ea76/0/sarit/mhu_ped_oauth_provider
CI_JOB_STAGE=test
CI_BUILD_REF_SLUG=master
_=/usr/bin/env
$ python -V
Python 2.7.13
I am not sure this is the best practice or not, but I directly set the PATH by this in order to let the gitlab-runner use same as I do when ssh remote to server. But at least it works.
variables:
PATH: "/home/gitlab-runner/.pyenv/plugins/pyenv-virtualenv/shims:/home/gitlab-runner/.pyenv/shims:/home/gitlab-runner/.pyenv/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games"
Similar to setting variables directly in gitlab-ci ...
job:
variables:
MY_CUSTOM_VARIABLE:my_variable
Better place an environment file in your project's main directory within e.g a file .ci/env.
In project directory (same level as .gitlab-ci.yml):
$ mkdir .ci
$ vi .ci/env
Then, in .gitlab-ci.yml copy the file into build context:
before_script:
- . .ci/env
In there you can place your custom or modified variables as in:
Any custom variables
export MY_CUSTOM_VARIABLE=my_variable
Pyenv and pipenv context variables:
export PYENV_ROOT="$HOME/.pyenv"
export PATH=/home/gitlab-runner/.pyenv/shims:/home/gitlab-runner/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
export PIPENV_PYTHON="$PYENV_ROOT/shims/python"
We use a Gitlab Project in a team. Each developer has his own Kubernetes cluster in the cloud and an own branch within GitLab. We use GitLab-CI to automatically build new containers and deploy them to our Kubernetes clusters.
At the moment we have a .gitlab-ci.yml looks something like this:
variables:
USERNAME: USERNAME
CI_K8S_PROJECT: ${USERNAME_CI_K8S_PROJECT}
REGISTRY_JSON_KEY_FILE: ${USERNAME_REGISTRY_JSON_KEY_FILE}
[...]
stages:
- build
- deploy
- remove
build-zeppelin:
stage: build
image: docker:latest
variables:
image_name: "zeppelin"
only:
- ${USERNAME}#Gitlab-Repo
tags:
- cloudrunner
script:
- docker login -u _json_key -p "${REGISTRY_JSON_KEY_FILE?}" https://eu.gcr.io
- image_name_fqdn="eu.gcr.io/${CI_K8S_PROJECT?}/${image_name?}:latest"
- docker build -t ${image_name_fqdn?} .
- docker push ${image_name_fqdn?}
- echo "Your new image is '${image_name_fqdn?}'. Have fun!"
[...]
So in the beginning we reference the important information by using a USERNAME-prefix. This works quite well, but is problematic, since we need to correct them after every pull request from another user.
So we search for a way to keep the gitlab-ci file the same to every developer while still referencing some gitlab-variables different for every developer.
Things we thought about, that don't seem to work:
Use multiple yml files and import them into each other => not supported.
Try to combine Gitlab Environment variables as Prefix:
CI_K8S_PROJECT: ${${GITLAB_USER_ID}_CI_K8S_PROJECT}
or
INDIVIDUAL_CI_K8S_PROJECT: ${GITLAB_USER_ID}_CI_K8S_PROJECT
CI_K8S_PROJECT: ${INDIVIDUAL_CI_K8S_PROJECT}
We found a solution using indirect expansion (bash feature):
before_script:
- variableName=${GITLAB_USER_ID}_CI_K8S_PROJECT
- export wantedValue=${!variableName}
But we also recognised, that our setup was somehow stupid: It does not make sense to have multiple branches for each user and use prefixed variables, since this leads to problems such as the above and security concerns, since all variables are accessible to all users.
It is way easier if each user forks the root project and simply creates a merge request for new features. This way there is no renaming/prefixing of variables or branches necessary at all.
Solution from #nik will work only for bash. For sh will work:
before_script:
- variableName=...
- export wantedValue=$( eval echo \$$variableName )
Something like this works (on 15.0.5-ee):
variables:
IMAGE_NAME: "test-$CI_PROJECT_NAME"
Is there a way I can use one my bash environment variable (say $FOO) in my Jekyll's _config.yml file?
I've tried using:
foo = <%= ENV['FOO'] %>
But it didn't work as the Ruby code wasn't interpreted.
Versions used:
Ruby: 2.1.2
Jekyll: 2.5.3
If your goal is to use environment variables as liquid items {{ site.something }}, you might be able to get this thing in your Gemfile a go:
gem 'jekyll-environment-variables', group: :jekyll_plugins
And then you'll be able to use {{ site.env.HOME }} and expect it be converted to something like /home/ubuntu in the output HTML.
Disclosure: I am the owner of the gem and I've been using it personally since long ago.
The answer by #elryco is close but not quite right, at least for my setup. It took some trial and error, but this finally worked. Note this only works for certain env vars supported by the contentful plugin.
Note that you need the gem jekyll-contentful-data-import (v1.7.0 or up) for this solution to actually work.
Bash environment (e.g., ~/.bash_profile):
export CONTENTFUL_ACCESS_TOKEN=foo
export CONTENTFUL_SPACE_ID=bar
In _config.yml, reference them as:
contentful:
spaces:
- example:
space: ENV_CONTENTFUL_SPACE_ID
access_token: ENV_CONTENTFUL_ACCESS_TOKEN
This is the same as what's written in the Github documentation.
I recently had to try and do this myself. It turns out you can't put environment variables directly into a Jekyll config file, but you can write a rake task that will take environment variables and apply them to your config.
Here's an example:
# Rakefile
require 'jekyll'
task default: %w[build]
desc "Build the site"
task :build do
config = Jekyll.configuration({
url: ENV["SITE_URL"],
})
site = Jekyll::Site.new(config)
Jekyll::Commands::Build.build(site, config)
end
Unfortunately there is no direct way of accessing it in liquid tags, At Least not officially.
But I wrote a wrapper script which reads environment variables before jekyll starts and appends it to _config.yml file and deletes the variable post build.
echo "secret-variable: $PASSWORD" >> _config.yml
bundle exec jekyll build -d target
sed '$d' _config.yml //this is to delete the last line
Now I'm free to use site.secret-variable anywhere in the liquid tags.
I know that this not the right way of doing it, But so is writing a custom ruby script.
I personally find the use of a ruby Jekyll plugin more appropriate and portable. There's a very simple yet effective solution available here.
The main idea is ruby will have access to the ENV variables so you can use a small ruby plugin to load into your site.config liquid array all the information you want from the environment. And you can define default values as well.
Please note that the example given in the link isn't the most relevant since the prod/staging environment is already offered by Jekyll natively with the build command options.
It is now possible to use bash environment variable (say $FOO) in Jekyll's _config.yml file with GitHub Actions:
# _config.yml
title: FOO
Create a bash script say sample.sh to replace for a given input string FOO and replace with another string
# github/workflows/sample.sh
export FOO=XYZ
while IFS='' read -r a; do
echo "${a//FOO/$FOO}"
done < /_config.yml > /_config.yml.t
mv /_config.yml{.t,}
Create a workflow file, say github-pages.yml, put the script before Build with Jekyll:
# Sample workflow for building and deploying a Jekyll site to GitHub Pages
name: Deploy Jekyll with GitHub Pages dependencies preinstalled
on:
# Runs on pushes targeting the default branch
push:
branches:
- 'master'
- 'mybranch'
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
permissions:
contents: read
pages: write
id-token: write
# Allow one concurrent deployment
concurrency:
group: "pages"
cancel-in-progress: true
jobs:
# Build job
build:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout#v3
- name: Setup Pages
uses: actions/configure-pages#v2
- name: Utilize FOO
run: |
bash .github/workflows/sample.sh
- name: Build with Jekyll
uses: actions/jekyll-build-pages#v1
with:
source: ./
destination: ./_site
- name: Upload artifact
uses: actions/upload-pages-artifact#v1
# Deployment job
deploy:
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
runs-on: ubuntu-latest
needs: build
steps:
- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages#v1
If your bash environment variables are declared like this
export ENV_ACCESS_TOKEN=xxxxx
export ENV_SPACE_ID=yyyyyy
You can get it like this in your config.yml
space: ENV_SPACE_ID # Required
access_token: ENV_ACCESS_TOKEN # Required