I am in the Debian9. And sudo su to become root user. After that I run the command gitlab-runner register. But for some reasons. I got the isolated environment not the same like when I login to the server over ssh
gitlab-runner --version
Version: 10.8.0
Git revision: 079aad9e
Git branch:
GO version: go1.8.7
Built: 2018-05-22T03:24:56+00:00
OS/Arch: linux/amd64
When I put env in the before_script I found the different environment.
Then I put source ~/.bashrc to it. Also does not make any change.
How to let shell runner use my normal login user environement?
Update:
When I execute env (by putting in before_script) I got this output
$ env
CI_RUNNER_EXECUTABLE_ARCH=linux/amd64
CI_COMMIT_TITLE=add config
CI_JOB_TOKEN=xxxxxxxxxxxxxxxxxxxx
CI_BUILD_REF_NAME=master
CI_REGISTRY_PASSWORD=xxxxxxxxxxxxxxxxxxxx
CI_RUNNER_TAGS=provider, oauth, mp
CI_SHARED_ENVIRONMENT=true
CI_JOB_NAME=test
CI_SERVER_VERSION=10.8.4
DJANGO_SETTINGS_MODULE=config.settings.local
LANG=en_US.UTF-8
GITLAB_CI=true
CI_SERVER_REVISION=2268d0c
CI_PROJECT_VISIBILITY=private
OLDPWD=/home/gitlab-runner
INVOCATION_ID=7dd516612949410c8ca8e4e59696f9fd
CI_COMMIT_SHA=9030892858b5ca92c7b36a81f573f187e6d14090
CI_COMMIT_MESSAGE=add config
CI_BUILD_STAGE=test
CI_PROJECT_URL=https://mbx-git.magicboxasia.com/sarit/mhu_ped_oauth_provider
CI_COMMIT_REF_SLUG=master
CI_SERVER_NAME=GitLab
CI_RUNNER_VERSION=10.8.0
CI_BUILD_NAME=test
CI=true
XDG_SESSION_ID=c5
CI_REGISTRY_USER=gitlab-ci-token
USER=gitlab-runner
CI_PROJECT_ID=96
CI_PIPELINE_ID=4399
CI_COMMIT_DESCRIPTION=
PWD=/home/gitlab-runner/builds/f304ea76/0/sarit/mhu_ped_oauth_provider
GITLAB_FEATURES=
HOME=/home/gitlab-runner
JOURNAL_STREAM=8:19161
CI_REGISTRY=docker-registry.magicboxasia.com
CI_BUILD_TOKEN=xxxxxxxxxxxxxxxxxxxx
CI_BUILD_ID=5514
CONFIG_FILE=/etc/gitlab-runner/config.toml
GITLAB_USER_NAME=Sarit Ritwirune
CI_PROJECT_PATH_SLUG=sarit-mhu-ped-oauth-provider
DATABASE_URL=postgres://postgres:postgres#postgres:5432/mp_oauth_provider
POSTGRES_DB=poinkdb
GITLAB_USER_EMAIL=sarit#magicboxasia.com
CI_COMMIT_REF_NAME=master
CI_REGISTRY_IMAGE=docker-registry.magicboxasia.com/sarit/mhu_ped_oauth_provider
CI_SERVER_TLS_CA_FILE=/home/gitlab-runner/builds/f304ea76/0/sarit/mhu_ped_oauth_provider.tmp/CI_SERVER_TLS_CA_FILE
CI_RUNNER_ID=42
CI_SERVER=yes
CI_JOB_ID=5514
CI_REPOSITORY_URL=https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx#mbx-git.magicboxasia.com/sarit/mhu_ped_oauth_provider.git
MAIL=/var/mail/gitlab-runner
SHELL=/bin/bash
GITLAB_USER_LOGIN=sarit
CI_RUNNER_REVISION=079aad9e
CI_CONFIG_PATH=.gitlab-ci.yml
CI_PROJECT_NAME=mhu_ped_oauth_provider
POSTGRES_PASSWORD=postgres
POSTGRES_USER=postgres
SHLVL=2
CI_RUNNER_DESCRIPTION=MP OAuth Provider
CI_PROJECT_PATH=sarit/mhu_ped_oauth_provider
LOGNAME=gitlab-runner
XDG_RUNTIME_DIR=/run/user/999
GITLAB_USER_ID=39
CI_BUILD_BEFORE_SHA=e36e718d09f89095972ddf60ebf1fc1931282ae4
CI_BUILD_REF=9030892858b5ca92c7b36a81f573f187e6d14090
PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
CI_PIPELINE_SOURCE=push
CI_PROJECT_NAMESPACE=sarit
CI_PROJECT_DIR=/home/gitlab-runner/builds/f304ea76/0/sarit/mhu_ped_oauth_provider
CI_JOB_STAGE=test
CI_BUILD_REF_SLUG=master
_=/usr/bin/env
$ python -V
Python 2.7.13
I am not sure this is the best practice or not, but I directly set the PATH by this in order to let the gitlab-runner use same as I do when ssh remote to server. But at least it works.
variables:
PATH: "/home/gitlab-runner/.pyenv/plugins/pyenv-virtualenv/shims:/home/gitlab-runner/.pyenv/shims:/home/gitlab-runner/.pyenv/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games"
Similar to setting variables directly in gitlab-ci ...
job:
variables:
MY_CUSTOM_VARIABLE:my_variable
Better place an environment file in your project's main directory within e.g a file .ci/env.
In project directory (same level as .gitlab-ci.yml):
$ mkdir .ci
$ vi .ci/env
Then, in .gitlab-ci.yml copy the file into build context:
before_script:
- . .ci/env
In there you can place your custom or modified variables as in:
Any custom variables
export MY_CUSTOM_VARIABLE=my_variable
Pyenv and pipenv context variables:
export PYENV_ROOT="$HOME/.pyenv"
export PATH=/home/gitlab-runner/.pyenv/shims:/home/gitlab-runner/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
export PIPENV_PYTHON="$PYENV_ROOT/shims/python"
Related
I have something like this in my .gitlab-ci.yml file:
.docker_build_cmd: &docker_build_cmd >-
docker build -t $DOCKER_IMAGE_TAG -f $DOCKER_FILE .
variables:
PROJECT_NAME: best-app-ever
BUILD_VERSION: 4.2.0
DOCKER_IMAGE_TAG: $DOCKER_PROJECT_NAME:$BUILD_VERSION
build:
stage: build
variables:
GRADLE_CMD: app:build
DOCKER_PROJECT_NAME: $PROJECT_NAME-service
DOCKER_FILE: docker/service.dockerfile
script:
- echo $DOCKER_PROJECT_NAME
- echo $DOCKER_IMAGE_TAG
- *docker_build_cmd
but runner says:
$ echo $DOCKER_PROJECT_NAME
best-app-ever
$ echo $DOCKER_IMAGE_TAG
$PROJECT_NAME-service:4.2.0
$ docker build -t $DOCKER_IMAGE_TAG -f $DOCKER_FILE .
invalid argument "$PROJECT_NAME-service:4.2.0" for "-t, --tag" flag: invalid reference format: repository name must be lowercase
Does anyone understand why it happens?
How Gitlab processes the .yml?
And if there is a way to make it work as I intended, I am really want to know.
GitLab SaaS version is 14.3.0-ee.
UPD. After some googling it seems that "variable_inside_variable" feature flag has to be enabled. Will do it tomorrow and see what changes.
GitLab CI "variable inside variable" feature was disabled, so I asked admins to toggle it and everything became fine. Don't think that this could be helpful question for many, because GitLab already enabled the feature globally, but if you have a standalone solution and the same issue, then you know what to do.
I cannot get this to work, even after checking other topics on stackoverflow. My project on gitlab.com/my_company/backend needs a module, found at gitlab.com/my_company/pkg/auth.
Locally, I've setup GOPRIVATE / git's configuration to make it work (and it works), though in Gitlab's CI Pipelines on a merge request, this fails.
Pipeline log / go.mod
I've added some debugging logs just to make sure everything was setup like I thought. Here's a failing pipeline's log :
$ git config --global url."ssh://git#gitlab.com/my_company/".insteadOf "https://gitlab.com/my_company/"
$ git config --global url."git#gitlab.com:".insteadOf "https://gitlab.com/"
$ git config -l | grep instead
url.ssh://git#gitlab.com/my_company/.insteadof=https://gitlab.com/my_company/
url.git#gitlab.com:.insteadof=https://gitlab.com/
$ env | grep GOPRIVATE
GOPRIVATE=gitlab.com/my_company
$ go mod download
go: gitlab.com/my_company/pkg/auth#v1.1.0: reading gitlab.com/my_company/pkg/auth/auth/go.mod at revision auth/v1.1.0: unknown revision auth/v1.1.0
One weird part of this log I've found is :
reading gitlab.com/my_company/pkg/auth/auth/go.mod - why is it repeating auth/auth? It actually happened once before locally, but it was because I wrote "github" instead of "gitlab" :)
The relevant go.mod line just in case :
require (
gitlab.com/my_company/pkg/auth v1.1.0 // indirect
)
Repository tags
Here are the tags setup on the repository gitlab.com/my_company/pkg :
$ git tag -l
auth/v1.0.0
auth/v1.1.0
cache/v1.0.0
cache/v1.0.1
$ git ls-remote --tags
From git#gitlab.my_company/pkg.git
9efcb02d5489adaac9d525dcb496d868d65e856a refs/tags/auth/v1.0.0
13730d4f61df978c6d690fd2678e2ed924808e0c refs/tags/auth/v1.1.0
2b8dff0ec1b737d975290720933180a9b591a1db refs/tags/cache/v1.0.0
9a3e598bbf83bea57b29d8a908b514861ae37b12 refs/tags/cache/v1.0.1
I'm not that familiar with Gitlab CI so I'm out of things to try. Any ideas?
Thank you!
Update: I finally got gitlab-runner installed so I could try running the yml directly, no luck. It still works locally (not a big surprise).
In you project should be file .gitlab-ci.yml and you can add GOPRIVATE variable to your CI and runner will use it for you project.
More details how to add env vars:
https://docs.gitlab.com/ee/ci/variables/#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file
I have a project where I'm using GitHub Actions for the CI / CD. I have a shell script where I would like to inject environment variables from my Actions yml. Here is what I have so far:
- name: docker-push
env:
USER: joesan
SOME_GITHUB_REPO_NAME: github-repo-name
GH_REPO: github.com/$USER/$SOME_GITHUB_REPO_NAME
run: |
echo "Running sbt assembly"
echo $GITHUB_REF
echo "Pushing tag into Docker Registry"
sh ./scripts/tag_deployment.sh
In the tag_deployment.sh, I'm trying to use these variables:
....
....
git clone https://${GH_REPO}
cd "${SOME_GITHUB_REPO_NAME}"
....
....
But I see when the action is run that it does not print what was set!
Cloning into '$SOME_GITHUB_REPO_NAME'...
fatal: unable to access 'https://github.com/$USER/$SOME_GITHUB_REPO_NAME/': The requested URL returned error: 400
How can I inject these environment variables properly into my shell script?
It was not so intuitive to understand from the GitHub documentation. I managed to fix this by moving the environment variables globally like this:
env:
USER: joesan
DEPLOYMENT_REPO_NAME: some-repo-name
And then use it in my some other job definition like this:
- name: docker-push
env:
GH_REPO: github.com/${{ env.USER }}/${{ env.DEPLOYMENT_REPO_NAME }}
I have a Ruby script that I want to run when a pull-request is created. This pull-request validates a series of conditions to be sure the pull-request can be merged. It's a very simple script with no external gems, just standard Ruby.
I'm trying to run this script on a job on a run step. The problem is, I'm not sure the path where the file should be saved.
The script is called: validator.rb. From my local computer I can run the script using:
ruby -r ./validator.rb -e "Validator.new.validate_something 'One parameter'"
This works fine locally but when I push this to GitHub it is failing. I saved my script as .github/workflows/ruby-scripts so my job looks like this:
jobs:
title:
name: "Title"
runs-on: ubuntu-latest
steps:
- run: ruby -r ./ruby-scripts/validator.rb -e "Validator.new.validate_something '${{ github.event.pull_request.title}}'"
And I get:
Run ruby -r ./ruby-scripts/validator.rb -e "Validator.new.validate 'Create README.md'"
/usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
/usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
/usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:29: warning: constant Gem::ConfigMap is deprecated
/usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:30: warning: constant Gem::ConfigMap is deprecated
/usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
/usr/local/lib/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:92:in `require': cannot load such file -- ./ruby-scripts/validator.rb (LoadError)
from /usr/local/lib/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:92:in `require'
##[error]Process completed with exit code 1.
I tried with all the possible combination of paths and it fails each time.
Running pwd and ls returned:
- run: pwd => /home/runner/work/repo-name/repo-name
- run: ls => shell: /bin/bash -e {0}
What is the right way to do this?
As I've mentioned in the comments, the reason why your workflow isn't working is that you forgot the crucial step that checks-out your repository. By default, the workspace is empty unless you check out the repository with the Checkout GitHub Action.
From the GitHub Action's README:
This action checks-out your repository under $GITHUB_WORKSPACE, so your workflow can access it.
Here's an example:
- name: Checkout Repo
uses: actions/checkout#v2
(That's it, really)
(Note: You don't have to specify a name for the step.)
For me, the problem was also the location of the file. The ruby script was running in the root of the repository, not in the same path as the workflow YAML (as I was expecting).
I was running run: ruby -r ./my-file.rb, where my-file.rb was next to the workflow yaml.
I realized this by adding this step:
- run: ruby -e 'p `ls`.split("\n")'
Which printed the output of ls in a ruby array.
I fixed using:
run: ruby ./.github/workflows/my_file.rb
I'm building a workflow on circleci 2.0 and so far jobs are running until it gets to android job.
At the build step ./gradlew assembleRelease it fails stating that an ENV VAR is not set:
Unzipping /home/circleci/.gradle/wrapper/dists/gradle-2.14.1-all/8bnwg5hd3w55iofp58khbp6yv/gradle-2.14.1-all.zip to /home/circleci/.gradle/wrapper/dists/gradle-2.14.1-all/8bnwg5hd3w55iofp58khbp6yv
Set executable permissions for: /home/circleci/.gradle/wrapper/dists/gradle-2.14.1-all/8bnwg5hd3w55iofp58khbp6yv/gradle-2.14.1/bin/gradle
FAILURE: Build failed with an exception.
* What went wrong:
Could not open terminal for stdout: $TERM not set
What I did try according to this post is setting the $TERM variable is a run command prior to the gradle invocation. But the build still fails looking for this variable.
Question:
How can you resolve $TERM not set on gradlew ./assembleRelease on CIrcleCI?
I did verify that I'm using the correct docker image according to this SO post:
https://stackoverflow.com/a/45744987/1829251
Here is the config.yml gist of the android CI Job:
android:
working_directory: ~/repo/android
docker:
- image: circleci/android:api-25-node8-alpha
steps:
- checkout:
path: ~/repo
- restore_cache:
key: jars-{{ checksum "build.gradle" }}-{{ checksum "app/build.gradle" }}
- attach_workspace:
at: ~/repo
- run: ./gradlew androidDepedencies
- run: export TERM=xterm
- run: sudo chmod +x ./gradlew
- run: ./gradlew assembleRelease
- save_cache:
paths:
- ~/.gradle
key: jars-{{ checksum "build.gradle" }}-{{ checksum "app/build.gradle" }}
- store_test_results:
path: ~/repo/android/reports
disclaimer: Developer Evangelist at CircleCI
- run: export TERM=xterm
That line sets the variable $TERM only for that specific shell. Each run step starts a brand new shell.
Your solution of running gradlew in the same step is one possible solution:
- run: export TERM=xterm && ./gradlew androidDepedencies
Another would be to properly export $TERM so that all subsequent steps can see the variable. This would be done like this:
- run: echo 'export TERM=xterm' >> $BASH_ENV
$BASH_ENV contains the path to the Bash file that is sourced at the beginning of every CircleCI step. Here's where this came from: https://circleci.com/docs/2.0/env-vars/#setting-path
I was exporting the ENV VAR incorrectly, using the following fixed the missing $TERM not set erorr in the android build:
- run: export TERM=xterm && ./gradlew androidDepedencies