Cannot push from gitlab-ci.yml - shell

Question

With my colleagues, we work on a C++ library that becomes more and more important each day. We already built continuous integration utilities through the gitlab-ci.yml file that let us:
Build & Test in Debug mode
Build & Test in Release mode
Perform safety checks like memory leaks using Valgrind and checking if there is any clear symbol in our library we don't want inside it
Generate documentation
All kind of stuff that made us choose GitLab !
We would like to profile our whole library and push the benchmarks in a separate project. We have already done something like for out documentation using the SSH key method but we would like to avoid this this time.
We tried a script like this:
test_ci_push:
tags:
- linux
- shell
- light
stage: profiling
allow_failure: false
only:
- new-benchmark-stage
script:
- git clone http://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.mycompany.home/developers/benchmarks.git &> /dev/null
- cd benchmarks
- touch test.dat
- echo "This is a test" > test.dat
- git config --global user.name "${GITLAB_USER_NAME}"
- git config --global user.email "${GITLAB_USER_EMAIL}"
- git add --all
- git commit -m "GitLab Runner Push"
- git push http://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.mycompany.home/developers/benchmarks.git HEAD:master
- cd ..
We also tried a basic git push origin master to push our updated files but each time we got the same answer:
remote: You are not allowed to upload code for this project.
fatal: unable to access 'http://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx#gitlab.mycompany.home/developers/benchmarks.git/': The requested URL returned error: 403
Both projects are under the same site and I have the rights to push in both. Where am I doing anything wrong here ?

Answer 1

The gitlab ci token is more like the deploy key in github.com, so it only has read access to the repository. To actually push you will need to generate a personal access token and use that instead.
First you need to generate the token as shown here in the gitlab documentation. Make sure you check both the read user and api scopes. Also this only works in GitLab 8.15 and above. If you are using an older version and do not wish to upgrade there's an alternative method I could show you but it is more complex and less secure.
In the end your gitlab-ci.yml should look something like this:
test_ci_push:
tags:
- linux
- shell
- light
stage: profiling
allow_failure: false
only:
- new-benchmark-stage
script:
- git clone http://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.mycompany.home/developers/benchmarks.git &> /dev/null
- cd benchmarks
- echo "This is a test" > test.dat
- git config --global user.name "${GITLAB_USER_NAME}"
- git config --global user.email "${GITLAB_USER_EMAIL}"
- git add --all
- git commit -m "GitLab Runner Push"
- git push http://${YOUR_USERNAME}:${PERSONAL_ACCESS_TOKEN}#gitlab.mycompany.home/developers/benchmarks.git HEAD:master
- cd ..

Answer 2

While the previous answers are more or less fine, there are some important gotya's.
before_script:
- git config --global user.name "${GITLAB_USER_NAME}"
- git config --global user.email "${GITLAB_USER_EMAIL}"
script:
- <do things>
- git push "https://${GITLAB_USER_LOGIN}:${CI_GIT_TOKEN}#${CI_REPOSITORY_URL#*#}" "HEAD:${CI_COMMIT_TAG}"
For one, we only need to set the username/email to please git.
Secondly having it in the before script, is not super crucial, but allows for easier reuse when doing 'extend'.
Finally, pushing https is 'fine' but since we're not using a stored ssh key, we should avoid anything that can reveal the token. For one, while gitlab won't print the token in this command, git will happily inform us that the new upstream is set to https://username:thetokeninplaintexthere#url
So there's your token in plain text, so don't use -u to set an upstream.
Also, it's not needed, we are only doing a single push.
Further more, when determining the URL, I found that using the exist CI_REPOSITORY_URL to be the most reliable solution (when moving repo's for example or whatnot). So we just replace the username/token in the URL string.

Answer 3

You could also provide user and password (user with write access) as secret variables and use them.
Example:
before_script:
- git remote set-url origin https://$GIT_CI_USER:$GIT_CI_PASS#$CI_SERVER_HOST/$CI_PROJECT_PATH.git
- git config --global user.email 'myuser#mydomain.com'
- git config --global user.name 'MyUser'
You have to define GIT_CI_USER and GIT_CI_PASS as secret variables (you could always create dedicated user for this purpose).
With this configuration you could normally work with git. I'm using this approach to push the tags after the release (with Axion Release Gradle Pluing - http://axion-release-plugin.readthedocs.io/en/latest/index.html)
Example release job:
release:
stage: release
script:
- git branch
- gradle release -Prelease.disableChecks -Prelease.pushTagsOnly
- git push --tags
only:
- master

Answer 4

I'm using the following GitLab job:
repo_pull_sync:
image: danger89/repo_mirror_pull:latest
rules:
- if: '$CI_PIPELINE_SOURCE == "schedule"'
- if: $REMOTE_URL
- if: $REMOTE_BRANCH
- if: $ACCESS_TOKEN
before_script:
- git config --global user.name "${GITLAB_USER_NAME}"
- git config --global user.email "${GITLAB_USER_EMAIL}"
script:
- git checkout $CI_DEFAULT_BRANCH
- git pull
- git remote remove upstream || true
- git remote add upstream $REMOTE_URL
- git fetch upstream
- git merge upstream/$REMOTE_BRANCH
- git push "https://${GITLAB_USER_LOGIN}:${ACCESS_TOKEN}#${CI_REPOSITORY_URL#*#}" "HEAD:${CI_DEFAULT_BRANCH}"
I'm using my own danger89/repo_mirror_pull docker image based on alpine, check this GitHub repository for more info.
This GitLab job pull upstream changes from the predefined remote repository + branch (see variables below), and merge them locally in CI/CD and pushes them in GitLab again.
Basically I created a repository pull mirror (which is officially not available for free on GitLab CE, only a push mirror is supported in GitLab).
Create in GitLab a Project Access Token first in GitLab. Via: Settings->Access Tokens. Check 'api' as the scope.
Create a new schedule, via: CI/CD->Schedules->New schedule. With the following 3 variables:
REMOTE_URL (example: https://github.com/project/repo.git)
REMOTE_BRANCH (example: master)
ACCESS_TOKEN: (see Access Token in the first step! Example: gplat-234hcand9q289rba89dghqa892agbd89arg2854, )
Save pipeline schedule
Again, see also: https://github.com/danger89/repo_pull_sync_docker_image
Regarding the question, see the git push command above, which allows you to push changes back into GitLab using GitLab (project) access token.