I have gitlab deployment activem and I want to get the deploy script to have some custom information about the deployment process (like $CI_PIPELINE_ID).
However, the script doesn't get the variables, instead it gets the "raw text".
the call performed by the script is: $ python deploy/deploy.py $CI_COMMIT_TAG $CI_ENVIRONMENT_URL $CI_PIPELINE_ID
How can i get it to use the variables?
My .gitlab-ci.yml:
image: python:2.7
before_script:
- whoami
- sudo apt-get --quiet update --yes
- sudo chmod +x deploy/deploy.py
deploy_production:
stage: deploy
environment: Production
only:
- tags
- trigger
except:
# - develop
- /^feature\/.*$/
- /^hotfix\/.*$/
- /^release\/.*$/
script:
- python deploy/deploy.py $CI_COMMIT_TAG $CI_ENVIRONMENT_URL $CI_PIPELINE_ID
It looks like potentially that you could be using a different environmental variable that you should be using.
bash/sh $variable
windows batch %variable%
PowerShell $env:variable
See using CI variables in your job script.
I don't get what you mean with "raw text", but you can declare the variables in your project settings. Also, have you configured you're runner?
Go to Settings->CI/CD->Secret Variables and just put them right there.
You can also find valuable information in the documentation.
Related
When running Github actions on a self hosted runner machine, how do I access existing custom environment variables that have been set on the machine, in my Github action .yaml script?
I have set those variables and restarted the runner virtual machine several times, but they are not accessible using the $VAR syntax in my script.
If you want to set a variable only for one run, you can add an export command when you configure the self-hosted runner on the Github repository, before running the ./run.sh command:
Example (linux) with a TEST variable:
# Create the runner and start the configuration experience
$ ./config.sh --url https://github.com/owner/repo --token ABCDEFG123456
# Add new variable
$ export TEST="MY_VALUE"
# Last step, run it!
$ ./run.sh
That way, you will be able to access the variable by using $TEST, and it will also appear when running env:
job:
runs-on: self-hosted
steps:
- run: env
- run: echo $VAR
If you want to set a variable permanently, you can add a file to the etc/profile.d/<filename>.sh, as suggested by #frennky above, but you will also have to update the shell for it be aware of the new env variables, each time, before running the ./run.sh command:
Example (linux) with a HTTP_PROXY variable:
# Create the runner and start the configuration experience
$ ./config.sh --url https://github.com/owner/repo --token ABCDEFG123456
# Create new profile http_proxy.sh file
$ sudo touch /etc/profile.d/http_proxy.sh
# Update the http_proxy.sh file
$ sudo vi /etc/profile.d/http_proxy.sh
# Add manually new line in the http_proxy.sh file
$ export HTTP_PROXY=http://my.proxy:8080
# Save the changes (:wq)
# Update the shell
$ bash
# Last step, run it!
$ ./run.sh
That way, you will also be able to access the variable by using $HTTP_PROXY, and it will also appear when running env, the same way as above.
job:
runs-on: self-hosted
steps:
- run: env
- run: echo $HTTP_PROXY
- run: |
cd $HOME
pwd
cd ../..
cat etc/profile.d/http_proxy.sh
The etc/profile.d/<filename>.sh will persist, but remember that you will have to update the shell each time you want to start the runner, before executing ./run.sh command. At least that is how it worked with the EC2 instance I used for this test.
Reference
Inside the application directory of the runner, there is a .env file, where you can put all variables for jobs running on this runner instance.
For example
LANG=en_US.UTF-8
TEST_VAR=Test!
Every time .env changes, restart the runner (assuming running as service)
sudo ./svc.sh stop
sudo ./svc.sh start
Test by printing the variable
I know I must be doing something silly here, but I'm trying to pass environment variables to a command being run under /bin/sh -c in Cloud Build.
My Cloud Build file looks like this:
- id: db-migrate
name: node:16.13.0
dir: 'packages/backend'
entrypoint: '/bin/sh'
args:
- '-c'
- '(/workspace/cloud_sql_proxy -dir=/workspace -instances=$_DB_CONNECTION=tcp:127.0.0.1:5432 & sleep 2) && yarn db:migrate'
env:
- 'DB_HOST=127.0.0.1'
- 'DB_USER=$_DB_USER'
- 'DB_PASSWORD=$_DB_PASSWORD'
- 'DB_NAME=$_DATABASE'
My Cloud Build Trigger has the substitutions set, and when I look at the build details it shows the environment variables as set.
However the command yarn db:migrate acts as if there are no env variables set. I believe this is because they aren't being passed from the machine to the command.
Any idea what I'm doing wrong?
The problem here is that when we call bin/sh it's creating a new shell with it's own environment variables. While I read through the manual on SH/Dash I will leave this question here:
How do I retain existing env variables in a new shell?
Alright, I figured this out.
We were using TypeORM, and originally used a ormconfig.json file. Turns out, this was still being picked up somehow on the system and was overriding all env variables.
Posting this response to help others in case they make this same mistake.
I am trying to run maven through a docker image using a shell script
When running docker in the shell, I use sed to remove single quotes:
bash script:
docker run $(echo "-e CUCUMBER_FILTER_TAGS=$CUCUMBER_FILTER_TAGS $RUN_ARGUMENT $INPUT_MAVEN_COMMAND $MAVEN_ARGUMENTS $AUTHENTICATION" | sed "s/'//g")
is translated into
docker run -e 'CUCUMBER_FILTER_TAGS="#bidrag-person' and not '#ignored"' --rm -v /home/deployer/actions-bidrag-cucumber-backend/ws/bidrag-cucumber-backend/bidrag-cucumber-backend:/usr/src/mymaven -v /home/deployer/.m2:/root/.m2 -w /usr/src/mymaven maven:3.6.3-openjdk-15 mvn test -e -DUSERNAME=j104364 -DINTEGRATION_INPUT=json/integrationInput.json -DUSER_AUTH=*** -DTEST_AUTH=*** -DPIP_AUTH=***
how can I remove those extra single quotes around and within CUCUMBER_FILTER_TAGS that seems to pop up from nowhere?
I cannot solve this and are seeking a solution. This script (https://github.com/navikt/bidrag-maven/blob/feature/filter.tags/cucumber-backend/cucumber.sh) is being run from a cron job on GitHub (GitHub Actions, part of a GitHub workflow)
The other variables (which are not inputs to this script) are set as environment variables from GitHub secrets in a GitHub workflow
AUTHENTICATION="-DUSER_AUTH=$USER_AUTHENTICATION -DTEST_AUTH=$TEST_USER_AUTHENTICATION -DPIP_AUTH=$PIP_USER_AUTHENTICATION"
are set in in a GitHub workflow yaml file like this:
- uses: navikt/bidrag-maven/cucumber-backend#v6
with:
maven_image: maven:3.6.3-openjdk-15
username: j104364
env:
USER_AUTHENTICATION: ${{ secrets.USER_AUTHENTICATION }}
TEST_USER_AUTHENTICATION: ${{ secrets.TEST_USER_AUTHENTICATION }}
PIP_USER_AUTHENTICATION: ${{ secrets.PIP_USER_AUTHENTICATION }}
You should be using an array.
docker_options=(
-e "CUCUMBER_FILTER_TAGS=$CUCUMBER_FILTER_TAGS"
"$RUN_ARGUMENT"
"$INPUT_MAVEN_COMMAND"
"$MAVEN_ARGUMENTS"
"$AUTHENTICATION"
)
docker run "${docker_options[#]}"
while these answers work to some degree, they will not function in my use case... I have recently upgraded the server where these scripts are used, and I have moved on to other scripting languages...
Conclusion:
Bash scripting is hard and painstaking... Both of these suggestions are functioning (sort of), but not as intended...
My gitlab ci-cd config file uses many env variables. To set them, I use gitlab ci-cd secret variables.
For example, dev deploy-part:
- echo "====== Deploy to dev server ======"
# Add target server`s secret key
- apk add git openssh bash
- mkdir ~/.ssh
- echo $DEV_SERVER_SECRET_KEY_BASE_64 | base64 -d > ~/.ssh/id_rsa
- chmod 700 ~/.ssh && chmod 600 ~/.ssh/*
- echo "Test ssh connection for"
- echo "$DEV_SERVER_USER#$DEV_SERVER_HOST"
- ssh -o StrictHostKeyChecking=no -T "$DEV_SERVER_USER#$DEV_SERVER_HOST"
# Delploy
- echo "Setup target server directories"
- TARGET_SERVER_HOST=$DEV_SERVER_HOST TARGET_SERVER_USER=$DEV_SERVER_USER TARGET_SERVER_APP_FOLDER=$DEV_SERVER_APP_FOLDER pm2 deploy pm2.config.js dev setup 2>&1 || true
- echo "make deploy"
- TARGET_SERVER_HOST=$DEV_SERVER_HOST TARGET_SERVER_USER=$DEV_SERVER_USER TARGET_SERVER_APP_FOLDER=$DEV_SERVER_APP_FOLDER pm2 deploy pm2.config.js dev
I have 5 repositories in project and 3 servers (dev, preprod, prod). So I must manage many variables. Manage all them using gitlab ci-cd secret variables it's very hurt. I can't see it, change it - only delete and create. I agree to use it for secret ssh keys, but it's not suitable for specifying the names of folders, hosts, etc.
Is there some other way to provide variables to ci-cd script?
You can define custom variables in quite a few ways:
I. using GUI:
per project
per group of projects
per Gitlab instance (for all projects and groups)
II. using .gitlab-ci.yml file:
You can use the variables keyword in a job or at the top level of the .gitlab-ci.yml file. If the variable is at the top level, it’s globally available and all jobs can use it. If it’s defined in a job, only that job can use it.
variables:
TEST_VAR: "All jobs can use this variable's value"
job1:
variables:
TEST_VAR_JOB: "Only job1 can use this variable's value"
script:
- echo "$TEST_VAR" and "$TEST_VAR_JOB"
Make sure to store only non-sensitive variables in your .gitlab-ci.yaml file
Check the official documentation for more information and examples.
From the GitLab CI documentation the bash shell is supported on Windows.
Supported systems by different shells:
Shells Bash Windows Batch PowerShell
Windows ✓ ✓ (default) ✓
In my config.toml, I have tried:
[[runners]]
name = "myTestRunner"
url = xxxxxxxxxxxxxxxxxxx
token = xxxxxxxxxxxxxxxxxx
executor = "shell"
shell = "bash"
But if my .gitlab-ci.yml attempts to execute bash script, for example
stages:
- Stage1
testJob:
stage: Stage1
when: always
script:
- echo $PWD
tags:
- myTestRunner
And then from the folder containing the GitLab multi runner I right-click and select 'git bash here' and then type:
gitlab-runner.exe exec shell testJob
It cannot resolve $PWD, proving it is not actually using a bash executor. (Git bash can usually correctly print out $PWD on Windows.)
Running with gitlab-runner 10.6.0 (a3543a27)
Using Shell executor...
Running on G0329...
Cloning repository...
Cloning into 'C:/GIT/CI_dev_project/builds/0/project-0'...
done.
Checking out 8cc3343d as bashFromBat...
Skipping Git submodules setup
$ echo $PWD
$PWD
Job succeeded
The same thing happens if I push a commit, and the web based GitLab CI terminal automatically runs the .gitlab-ci script.
How do I correctly use the Bash terminal in GitLab CI on Windows?
Firstly my guess is that it is not working as it should (see the comment below your question). I found a workaround, maybe it is not what you need but it works. For some reason the command "echo $PWD" is concatenated after bash command and then it is executed in a Windows cmd. That is why the result is "$PWD". To replicate it execute the following in a CMD console (only bash is open):
bash && echo $PWD
The solution is to execute the command inside bash with option -c (it is not the ideal solution but it works). The .gitlab-ci.yml should be:
stages:
- Stage1
testJob:
stage: Stage1
when: always
script:
- bash -c "echo $PWD"
tags:
- myTestRunner