I want to use the GITHUB_SHA with a variable, like this:
name: build
on: ["push"]
env:
PACKAGE: package-$GITHUB_SHA
But, when I use, yaml does not expand the variable and I got the string. How I can do it?
According to https://help.github.com/en/actions/automating-your-workflow-with-github-actions/contexts-and-expression-syntax-for-github-actions#github-context:
As part of an expression, you may access context information using one of two syntaxes.
Index syntax: github['sha']
Property dereference syntax: github.sha
In this case:
name: build
on: ["push"]
env:
PACKAGE: package-${{ github.sha }}
Try the following:
name: build
on: ["push"]
env:
PACKAGE: package-${{ github.sha }}
Related
I have the following Github Action workflow that is intended to read our lines of code coverage from a coverage.txt file and print the coverage as a comment.
name: Report Code Coverage
on:
pull_request:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Use Node.js 12.x
uses: actions/setup-node#v1
with:
node-version: 12.x
- name: Install dependencies
run: npm ci
- name: Run tests
run: npm run test:coverage
## I need help around here, as test:coverage generates a file and I need to get a value from a file
- uses: mshick/add-pr-comment#v1
with:
message: |
**{{Where I want the code coverage value to go}}**
🌏
!
repo-token: ${{ secrets.GITHUB_TOKEN }}
repo-token-user-login: 'github-actions[bot]' # The user.login for temporary GitHub tokens
allow-repeats: false # This is the default
Where I am struggling is on taking the output of the file, which I can obtain with bash using awk '/^Lines/ { print $2 }' < coverage.txt (not sure if there is a better alternative way) and inserting it into the message, which currently just says hello.
I found this article on yaml variables, but when I added some of that code to my own file, it just was not recognized any longer by GitHub actions (which is the only way I know to test yaml). Normally it would fail somewhere and give me an error, but after multiple attempts, the only thing that worked was to remove it.
It is quite possible that I am missing something obvious as I do not know yaml very well nor even what certain key words might be.
Alternatively, is it easier to just dump the contents of the file into the message? That could be acceptable as well.
You can create a step that gets the coverage to an output variable that then can be accessed by the next step.
Notice that for utilizing this method you will need to give the step and id and the set output variable a variable name so that you can access it in follow up steps within the same job.
Sample with your workflow below, but if you want to see a running demo here is the repo https://github.com/meroware/demo-coverage-set-output
- name: Run tests
run: npm run test:coverage
- name: Get coverage output
id: get_coverage
run: echo "::set-output name=coverage::$(awk '/^Lines/ { print $2 }' < test.txt)"
- uses: mshick/add-pr-comment#v1
with:
message: |
Coverage found ${{steps.get_coverage.outputs.coverage}}
🌏
!
repo-token: ${{ secrets.GITHUB_TOKEN }}
repo-token-user-login: 'github-actions[bot]' # The user.login for temporary GitHub tokens
allow-repeats: false # This is the default
I make a lot of github actions tutorials here if you're looking to grasp more on this topic. Cheers!!
What is the correct placement of caching in a GitHub Action? Specifically is in correct to place it before or after running setup of tools using another Action?
For example if I'm using something like haskell/actions/setup should my use of actions/cache precede or follow that? Put another way: if setup subsequently installs updated components on a future run of my Action, will the corresponding parts of the cache be invalidated?
The cache action should be placed before any step that consumes or creates that cache. This step is responsible for:
defining cache parameters.
restoring the cache, if it was cached in the past.
GitHub Actions will then run a "Post *" step after all the steps, which will store the cache for future calls.
See the example workflow from the documentation.
For example, consider this sample workflow:
name: Caching Test
on: push
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Enable Cache
id: cache-action
uses: actions/cache#v2
with:
path: cache-folder
key: ${{ runner.os }}-cache-key
- name: Use or generate the cache
if: steps.cache-action.outputs.cache-hit != 'true'
run: mkdir cache-folder && touch cache-folder/hello
- name: Verify we have our cached file
run: ls cache-folder
This is how it looks on the first run:
And this is how it looks on the second run:
GitHub will not invalidate the cache. Instead, it is the responsibility of the developer to ensure that the cache key is unique to the content it represents.
On common way to do this, is to set the cache key so that it contains a hash of a file that lives in the repository, so that changes to this file, will yield a different cache key. A good example for such a behavior, is when you have lock files that list all your repository's dependencies (requirements.txt for pyrhon, Gemfile.lock for ruby, etc).
This is achieved by a syntax similar to this:
key: ${{ runner.os }}-${{ hashFiles('**/lockfiles') }}
as described in the Creating a cache key section of the documentation.
I would like to use the Secret Manager to store a credential to our artifactory, within a cloud build step. I have it working using a build similar to:
steps:
- name: 'busybox:glibc'
entrypoint: 'sh'
args: ['-c', 'env']
secretEnv: ['SECRET_VALUE']
availableSecrets:
secretManager:
- versionName: "projects/PROJECT_ID/secrets/TEST-SECRET/versions/1"
env: 'SECRET_VALUE'
All great, no problems - I then try and slightly improve it to:
steps:
- name: 'busybox:glibc'
entrypoint: 'sh'
args: ['-c', 'env']
secretEnv: ['SECRET_VALUE']
availableSecrets:
secretManager:
- versionName: "projects/$PROJECT_ID/secrets/TEST-SECRET/versions/1"
env: 'SECRET_VALUE'
But then it throws the error:
ERROR: (gcloud.builds.submit) INVALID_ARGUMENT: failed to get secret name from secret version "projects/$PROJECT_ID/secrets/TEST-SECRET/versions/1"
I have been able to add a TRIGGER level env var (SECRET_MANAGER_PROJECT_ID), and that works fine. The only issue that as that is a trigger env, it is not available on rebuild, which breaks a lot of things.
Does anyone know how to get the PROJECT_ID of a Secret Manager from within CloudBuild without using a Trigger Param?
For now, it's not possible to set dynamic value in the secret field. I already provided this feedback directly to the Google Cloud PM, it has been take into account, but I don't have more info to share, especially for the availability.
EDIT 1
(January 22). Thanks to Seza443 comment, I tested again and now it works with automatically populated variable (PROJECT_ID and PROJECT_NUMBER), but also with customer defined substitution variables!
It appears that Cloud Build now allows for the use of substitution variables within the availableSecrets field of a build configuration.
From Google Cloud's documentation on using secrets:
After all the build steps, add an availableSecrets field to specify the secret version and environment variables to use for your secret. You can include substitution variables in the value of the secretVersion field. You can specify more than one secret in a build.
I was able to use the $PROJECT_ID variable in my own build configuration like so:
...
availableSecrets:
secretManager:
- versionName: projects/$PROJECT_ID/secrets/api-key/versions/latest
env: API_KEY
Granted, there appears to be (at least at present) some discrepancy between the documentation quoted above and the recommended configuration file schema. In the documentation they refer to secretVersion, but that appears to have changed to versionName. In either case, it seems to work properly.
Use the $PROJECT_NUMBER instead.
https://cloud.google.com/build/docs/configuring-builds/substitute-variable-values#using_default_substitutions
I have a bunch of concourse pipeline files that look like the following:
---
resources:
- name: example
type: git
source:
uri: git#github.internal.me.com:me/example.git
branch: {{tracking_branch}}
private_key: {{ssh_key}}
paths:
- code/src/do/teams/sampleapp
params:
depth: 1
- name: deploy-image
type: docker-image
source:
repository: {{docker_image_url}}
And I want to parse them in ruby to perform a bunch of transformations (like validating them and updating some keys if they are missing).
Problem is, whenever I try to load and them dump them back to files the pieces that have {{something}} become:
branch:
? tracking_branch:
:
private_key:
? ssh_key:
:
Why is it doing this and is there any way I can configure the parser not to do this? Just leave these variables as they are?
To avoid conflict with YAML's internal syntax you need to quote your values:
---
resources:
- name: example
type: git
source:
uri: git#github.internal.me.com:me/example.git
branch: '{{tracking_branch}}'
private_key: '{{ssh_key}}'
paths:
- code/src/do/teams/sampleapp
params:
depth: 1
This sort of thing comes up in Ansible configuration files all the time for similar reasons.
The { and } characters are used in Yaml for flow mappings (i.e. hashes). If you don’t provide a value for a mapping entry you get nil.
So in the case of branch: {{tracking_branch}}, since there are two pairs of braces, you get a hash with a key branch and value (in Ruby) of
{{"tracking_branch"=>nil}=>nil}
When this is dumped back out to Yaml you get the somewhat awwkward and verbose:
branch:
? tracking_branch:
:
The solution is simply to quote the value:
branch: "{{tracking_branch}}"
Completely forgot that concourse now offers ((var-name)) for templating, just switched to that instead of {{var-name}} at the pipelines and the YAML parser is now happy!
Is there a way I can use one my bash environment variable (say $FOO) in my Jekyll's _config.yml file?
I've tried using:
foo = <%= ENV['FOO'] %>
But it didn't work as the Ruby code wasn't interpreted.
Versions used:
Ruby: 2.1.2
Jekyll: 2.5.3
If your goal is to use environment variables as liquid items {{ site.something }}, you might be able to get this thing in your Gemfile a go:
gem 'jekyll-environment-variables', group: :jekyll_plugins
And then you'll be able to use {{ site.env.HOME }} and expect it be converted to something like /home/ubuntu in the output HTML.
Disclosure: I am the owner of the gem and I've been using it personally since long ago.
The answer by #elryco is close but not quite right, at least for my setup. It took some trial and error, but this finally worked. Note this only works for certain env vars supported by the contentful plugin.
Note that you need the gem jekyll-contentful-data-import (v1.7.0 or up) for this solution to actually work.
Bash environment (e.g., ~/.bash_profile):
export CONTENTFUL_ACCESS_TOKEN=foo
export CONTENTFUL_SPACE_ID=bar
In _config.yml, reference them as:
contentful:
spaces:
- example:
space: ENV_CONTENTFUL_SPACE_ID
access_token: ENV_CONTENTFUL_ACCESS_TOKEN
This is the same as what's written in the Github documentation.
I recently had to try and do this myself. It turns out you can't put environment variables directly into a Jekyll config file, but you can write a rake task that will take environment variables and apply them to your config.
Here's an example:
# Rakefile
require 'jekyll'
task default: %w[build]
desc "Build the site"
task :build do
config = Jekyll.configuration({
url: ENV["SITE_URL"],
})
site = Jekyll::Site.new(config)
Jekyll::Commands::Build.build(site, config)
end
Unfortunately there is no direct way of accessing it in liquid tags, At Least not officially.
But I wrote a wrapper script which reads environment variables before jekyll starts and appends it to _config.yml file and deletes the variable post build.
echo "secret-variable: $PASSWORD" >> _config.yml
bundle exec jekyll build -d target
sed '$d' _config.yml //this is to delete the last line
Now I'm free to use site.secret-variable anywhere in the liquid tags.
I know that this not the right way of doing it, But so is writing a custom ruby script.
I personally find the use of a ruby Jekyll plugin more appropriate and portable. There's a very simple yet effective solution available here.
The main idea is ruby will have access to the ENV variables so you can use a small ruby plugin to load into your site.config liquid array all the information you want from the environment. And you can define default values as well.
Please note that the example given in the link isn't the most relevant since the prod/staging environment is already offered by Jekyll natively with the build command options.
It is now possible to use bash environment variable (say $FOO) in Jekyll's _config.yml file with GitHub Actions:
# _config.yml
title: FOO
Create a bash script say sample.sh to replace for a given input string FOO and replace with another string
# github/workflows/sample.sh
export FOO=XYZ
while IFS='' read -r a; do
echo "${a//FOO/$FOO}"
done < /_config.yml > /_config.yml.t
mv /_config.yml{.t,}
Create a workflow file, say github-pages.yml, put the script before Build with Jekyll:
# Sample workflow for building and deploying a Jekyll site to GitHub Pages
name: Deploy Jekyll with GitHub Pages dependencies preinstalled
on:
# Runs on pushes targeting the default branch
push:
branches:
- 'master'
- 'mybranch'
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
permissions:
contents: read
pages: write
id-token: write
# Allow one concurrent deployment
concurrency:
group: "pages"
cancel-in-progress: true
jobs:
# Build job
build:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout#v3
- name: Setup Pages
uses: actions/configure-pages#v2
- name: Utilize FOO
run: |
bash .github/workflows/sample.sh
- name: Build with Jekyll
uses: actions/jekyll-build-pages#v1
with:
source: ./
destination: ./_site
- name: Upload artifact
uses: actions/upload-pages-artifact#v1
# Deployment job
deploy:
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
runs-on: ubuntu-latest
needs: build
steps:
- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages#v1
If your bash environment variables are declared like this
export ENV_ACCESS_TOKEN=xxxxx
export ENV_SPACE_ID=yyyyyy
You can get it like this in your config.yml
space: ENV_SPACE_ID # Required
access_token: ENV_ACCESS_TOKEN # Required