How to reuse anchored entry under already unwrapped anchor? - continuous-integration

I am trying to write a CircleCI config that will allow me to reuse both whole list/mapping(?) entries and its properties.
Having the following:
image_definitions:
docker:
- &default_localstack_image
image: localstack/localstack:0.10.3
environment:
KINESIS_LATENCY: 0
defaults_env: &defaults_env
environment:
PG_PORT: 5432
PG_USER: root
I would like to be able to replace:
test: &test
docker:
- image: localstack/localstack:0.10.3
<<: *defaults_env
with something like:
test: &test
docker:
- *default_localstack_image
<<: *defaults_env
but it doesn't work this way.
I've also tried:
test: &test
docker:
- *default_localstack_image
*defaults_env
but that also didn't work.
How can I do that?

According to the documentation:
test: &test
docker:
- <<: [*default_localstack_image, *defaults_env]
However, be aware that the merge feature is not part of the YAML spec and has only been defined for outdated YAML 1.1. I don't know if this is actually implemented. Even if it is, be aware that this merge key is the odd man out – violating the spec that says every tag is to be mapped to a type, it is instead interpreted as transformation instruction even though the loading process as defined by the spec has no place for executing transformation steps.
Similar functionality (for example for concatenating scalars) is more or less frequently requested on SO but is not available (and will probably never be) and if you need to do something like this, my advice is to do what e.g. Ansible and SaltStack do and use a templating engine as preprocessor for your YAML file.

Related

What is the correct way of using secrets in strategy-matrix pattern in GitHub Actions?

I am using GitHub Actions for one of the projects. I have a use case when I need to deploy to 2 different environments. As the number of domains may grow, I want to deploy to all of them at once parametrically.
Part of my job that fails:
jobs:
build:
strategy:
matrix:
domain: [['main', 'books-v1'], ['old-main', 'books-v2']]
The above part works perfectly but if I need to add new variants from the secrets, the workflow doesn't work. See the snippet below:
jobs:
build:
strategy:
matrix:
domain: [['main', 'books-v1', ${{ secrets.URL_V1 }}], ['old-main', 'books-v2', ${{ secrets.URL_V2 }}]]
I checked GitHub Actions docs. I also searched available examples on GitHub to see existing solutions. So far, I didn't find a similar use case.
Is there a way to make it work like that? What are alternatives to my approach that will work?
GitHub Actions failure message:
You have an error in your yaml syntax on line XYZ
At the YAML level, single quotes around ${{ secrets... }} should fix the syntax error.
But, according to the Context availability, the secrets context is not allowed under stratey. The allowed contexts are:
jobs.<job_id>.strategy github, needs, vars, inputs
You can make use of the vars context for your use case.
Apart from that, linting your workflow with https://rhysd.github.io/actionlint/ would be much faster to identify potential issues.
UPDATE (by Dmytro Chasovskyi)
Here is an example with the vars context:
With a variable DOMAINS having this config:
{
"v1": {
"url": "http://localhost:80/api/v1"
},
"v2": {
"url": "http://localhost:80/api/v2"
}
}
the workflow will be:
jobs:
build:
strategy:
matrix:
domain: [['main', 'books-v1', '${{ vars.DOMAINS.v1.url }}'], ['old-main', 'books-v2', '${{ vars.DOMAINS.v2.url }}']]

How to use custom substitutions with secretmanager in cloudbuild?

I'm having an issue with using custom substitutions in my cloudbuild.yaml.
substitutions:
_CUSTOM_SUBSTITUTION: this-is-a-path
availableSecrets:
secretManager:
- versionName: projects/$_CUSTOM_SUBSTITUTION/secrets/client_id/versions/1
env: CLIENT_ID
- versionName: projects/$_CUSTOM_SUBSTITUTION/secrets/client_secret/versions/1
env: CLIENT_SECRET
From what I can tell from trial and error, using something like $PROJECT_ID in the place of $_CUSTOM_SUBSTITUTION will run the build, but if I use a custom substitution like above, the trigger does not run a build at all when a commit is pushed.
I've also tested with various other base substitutions, like $BRANCH_NAME to the same effect. I'm getting the feeling that it's just not possible to do this in cloudbuild at the moment?
It ended up being a combination of need curly braces ${_CUSTOM_SUBSTITUTION} and some syntax fixing in the cloudbuild.yaml. I didn't have enough experience with cloudbuild to find that.
The offending part was something this:
AUTH_TOKEN=$$(cat /workspace/token.txt). Originally I had just 1 $ there, which was also working code pulled from another project.
For anyone running into this in the future, using gloud builds submit can let you run it directly for troubleshooting.

Setting up Idempotency for Parse-Platform on k8s

I'm trying to enable the Idempotency options on a Parse-Platform server.
I've tried adding the following to my k8s config:
spec:
containers:
- args:
- --idempotencyOptions
- '{"paths":["classes/.*"], ttl: 30}'
I've also tried to add it to env vars (according to https://github.com/parse-community/parse-server/issues/7151 you have to write the json object as a string.
env:
- name: PARSE_SERVER_EXPERIMENTAL_IDEMPOTENCY_OPTIONS
value: '{"paths":["classes/.*"], ttl: 30}'
No luck. The first option and I get an error/crash loop stating idempotencyOptions isn't valid. The second one, it boots up fine, but duplicates still occur even when the proper headers are added (X-Parse-Request-Id)
Anyone have any other ideas?
So, always good to double check your versions.
Both of these work, my Parse Server was 4.2.0. This feature was introduced in 4.3.0. Once I upgraded everything worked as intended.

PROJECT_ID env and Secret Manager Access

I would like to use the Secret Manager to store a credential to our artifactory, within a cloud build step. I have it working using a build similar to:
steps:
- name: 'busybox:glibc'
entrypoint: 'sh'
args: ['-c', 'env']
secretEnv: ['SECRET_VALUE']
availableSecrets:
secretManager:
- versionName: "projects/PROJECT_ID/secrets/TEST-SECRET/versions/1"
env: 'SECRET_VALUE'
All great, no problems - I then try and slightly improve it to:
steps:
- name: 'busybox:glibc'
entrypoint: 'sh'
args: ['-c', 'env']
secretEnv: ['SECRET_VALUE']
availableSecrets:
secretManager:
- versionName: "projects/$PROJECT_ID/secrets/TEST-SECRET/versions/1"
env: 'SECRET_VALUE'
But then it throws the error:
ERROR: (gcloud.builds.submit) INVALID_ARGUMENT: failed to get secret name from secret version "projects/$PROJECT_ID/secrets/TEST-SECRET/versions/1"
I have been able to add a TRIGGER level env var (SECRET_MANAGER_PROJECT_ID), and that works fine. The only issue that as that is a trigger env, it is not available on rebuild, which breaks a lot of things.
Does anyone know how to get the PROJECT_ID of a Secret Manager from within CloudBuild without using a Trigger Param?
For now, it's not possible to set dynamic value in the secret field. I already provided this feedback directly to the Google Cloud PM, it has been take into account, but I don't have more info to share, especially for the availability.
EDIT 1
(January 22). Thanks to Seza443 comment, I tested again and now it works with automatically populated variable (PROJECT_ID and PROJECT_NUMBER), but also with customer defined substitution variables!
It appears that Cloud Build now allows for the use of substitution variables within the availableSecrets field of a build configuration.
From Google Cloud's documentation on using secrets:
After all the build steps, add an availableSecrets field to specify the secret version and environment variables to use for your secret. You can include substitution variables in the value of the secretVersion field. You can specify more than one secret in a build.
I was able to use the $PROJECT_ID variable in my own build configuration like so:
...
availableSecrets:
secretManager:
- versionName: projects/$PROJECT_ID/secrets/api-key/versions/latest
env: API_KEY
Granted, there appears to be (at least at present) some discrepancy between the documentation quoted above and the recommended configuration file schema. In the documentation they refer to secretVersion, but that appears to have changed to versionName. In either case, it seems to work properly.
Use the $PROJECT_NUMBER instead.
https://cloud.google.com/build/docs/configuring-builds/substitute-variable-values#using_default_substitutions

Concat variable names in GitLab

We use a Gitlab Project in a team. Each developer has his own Kubernetes cluster in the cloud and an own branch within GitLab. We use GitLab-CI to automatically build new containers and deploy them to our Kubernetes clusters.
At the moment we have a .gitlab-ci.yml looks something like this:
variables:
USERNAME: USERNAME
CI_K8S_PROJECT: ${USERNAME_CI_K8S_PROJECT}
REGISTRY_JSON_KEY_FILE: ${USERNAME_REGISTRY_JSON_KEY_FILE}
[...]
stages:
- build
- deploy
- remove
build-zeppelin:
stage: build
image: docker:latest
variables:
image_name: "zeppelin"
only:
- ${USERNAME}#Gitlab-Repo
tags:
- cloudrunner
script:
- docker login -u _json_key -p "${REGISTRY_JSON_KEY_FILE?}" https://eu.gcr.io
- image_name_fqdn="eu.gcr.io/${CI_K8S_PROJECT?}/${image_name?}:latest"
- docker build -t ${image_name_fqdn?} .
- docker push ${image_name_fqdn?}
- echo "Your new image is '${image_name_fqdn?}'. Have fun!"
[...]
So in the beginning we reference the important information by using a USERNAME-prefix. This works quite well, but is problematic, since we need to correct them after every pull request from another user.
So we search for a way to keep the gitlab-ci file the same to every developer while still referencing some gitlab-variables different for every developer.
Things we thought about, that don't seem to work:
Use multiple yml files and import them into each other => not supported.
Try to combine Gitlab Environment variables as Prefix:
CI_K8S_PROJECT: ${${GITLAB_USER_ID}_CI_K8S_PROJECT}
or
INDIVIDUAL_CI_K8S_PROJECT: ${GITLAB_USER_ID}_CI_K8S_PROJECT
CI_K8S_PROJECT: ${INDIVIDUAL_CI_K8S_PROJECT}
We found a solution using indirect expansion (bash feature):
before_script:
- variableName=${GITLAB_USER_ID}_CI_K8S_PROJECT
- export wantedValue=${!variableName}
But we also recognised, that our setup was somehow stupid: It does not make sense to have multiple branches for each user and use prefixed variables, since this leads to problems such as the above and security concerns, since all variables are accessible to all users.
It is way easier if each user forks the root project and simply creates a merge request for new features. This way there is no renaming/prefixing of variables or branches necessary at all.
Solution from #nik will work only for bash. For sh will work:
before_script:
- variableName=...
- export wantedValue=$( eval echo \$$variableName )
Something like this works (on 15.0.5-ee):
variables:
IMAGE_NAME: "test-$CI_PROJECT_NAME"

Resources