git push changes made during gitlab ci/cd - continuous-integration

I'm trying to do change in a file during a job in a pipeline i'm developing and then commit that change to the master branch of the same project, but I'm having a hard time making it work.
Here's the job:
maven_next_release:
stage: next-version
dependencies:
- maven_test
before_script:
- apt update && apt-get install git perl-base -yrelease-demo.git
- git config --global user.email "${GITLAB_USER_EMAIL}"
- git config --global user.name "${GITLAB_USER_NAME}"
- git fetch
- git checkout master
script:
- cat VERSION
- perl -i -pe 's/\d+\.\d+\.\K(\d+)/ $1+1 /e' VERSION
- echo $(cat VERSION)-SNAPSHOT > VERSION
- cat VERSION
- git add VERSION
- git commit -m "[skip ci]New version $(cat VERSION)"
- git push https://${GIT_USERNAME}:${GIT_PASSWORD}#gitlab.com/myproject/release-demo.git
only:
- tags
except:
- branches
So, everything seems to work except for the push command. Here's the log:
$ git push https://${GIT_USERNAME}:${GIT_PASSWORD}#gitlab.com/myproject/release-demo.git
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://gitlab.com/myproject/release-demo.git/'
I'm really not sure what to do, I read about settting a SSH Key so I don't have to pass user and password, but I'm not sure how to generate a SSH key for the runner.

GitLab CI (unlike GitHub Actions) does not automatically authorize you to push code on check out.
To achieve what you want you need to generate Git Push Token and pass it in secrets to your pipeline.
For a sample - you can refer to my sample helm cd project here - https://gitlab.com/taleodor/sample-helm-cd/
Particularly, search for "GIT_PUSH_TOKEN" in the documentation and then the actual git commit part is in https://gitlab.com/taleodor/sample-helm-cd/-/blob/master/.gitlab-ci.yml in ".git-script" block.

Instead using https you can using ssh_key.
You can add ssh_key inside container in shared runner or private runner gitlabci.

So i resolved my issue:
First at all, I previously created two enviroment varaibles in my ci/cd, GIT_USER and GIT_PASSWORD, I had them as protected variables, so I had to unselect thatt and just mark them as masked.
Secondly I modified my job like this:
maven_next_release:
stage: next-version
dependencies:
- maven_test
before_script:
- apt update && apt-get install git perl-base -y
- git clone http://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.com/myteam/release-demo.git &> /dev/null
- cd release-demo
- git config --global user.email "${GITLAB_USER_EMAIL}"
- git config --global user.name "${GITLAB_USER_NAME}"
script:
- cat VERSION
- perl -i -pe 's/\d+\.\d+\.\K(\d+)/ $1+1 /e' VERSION
- echo $(cat VERSION)-SNAPSHOT > VERSION
- cat VERSION
- git add VERSION
- git commit -m "[skip ci]Version $(cat VERSION)"
- git push "https://${GIT_USERNAME}:${GIT_PASSWORD}#${CI_REPOSITORY_URL#*#}" HEAD:master
only:
- tags
except:
- branches
and with that, my pipeline finally worked and can push changes to master branch.

Related

commit and push generically with Jenkins in a Freestyle project

I'm trying to commit and push the changes to Bitbucket using a Jenkins Job
The job is a Freestsyle Project with "Execute Shell" as a build step.
What I currently did inside the build step of "Execute Shell" is the following:
python run.py
git config --global user.email "name#gmail.com"
git config --global user.name "name"
git add .
git commit -m 'jenkins.job : update...'
Then I push the changes with Git Publisher.
This works as expected in my local computer but is there a way to retrieve the credentials that I used in Source Code Management with Jenkins variables instead of using git config (because I'm not allowed to use git config error: could not lock config file //.gitconfig: Permission denied)
I fixed it using git -c user.name=".." -c user.email="..#.." commit -m ".."
This way I can set temporary configuration data.

Get a private repository from AWS codecommit using HTTPS GRC

I'm trying to import a module located in AWS codecommit. To clone the repository I'm using HTTPS GRC (Git Remote Codecommit) method, which uses Google Suite credentials to access AWS console.
The command I use to clone the repository is:
git clone codecommit::us-west-2://my-module
The remote module's go.mod file contains this:
module git-codecommit.us-west-2.amazonaws.com/my-module.git
I tried to achieve my goal configuring Git like this:
git config --global url."codecommit::us-west-2://".insteadOf "https://git-codecommit.us-west-2.amazonaws.com/"
Setted GOPRIVATE:
go env -w GOPRIVATE=git-codecommit.us-west-2.amazonaws.com/my-module.git
And then getting the repository:
go get -x git-codecommit.us-west-2.amazonaws.com/my-module.git
but I get this output (and the execution gets stuck):
cd.
git ls-remote https://git-codecommit.us-west-2.amazonaws.com/my-module
I would like to mention that when I execute the git ls-remote https://git-codecommit.us-west-2.amazonaws.com/my-module command manually I get the information of the branches and tags without problems.
I checked this topic but in that case SSH protocol is used instead of HTTP GRC. Maybe the only way to import a module from a private repository is via SSH?
Finally found the solution:
Set Git credential helper:
git config --global credential.helper '!aws codecommit credential-helper $#'
git config --global credential.UseHttpPath true
Set GOPRIVATE env var:
go env -w GOPRIVATE=git-codecommit.us-west-2.amazonaws.com
In MacOS, disable keychain for Git:
Comment helper = osxkeychain in the file containing that value (run git config -l --show-origin | grep credential to find the target file)
Run go get:
go get git-codecommit.us-west-2.amazonaws.com/v1/repos/my-module.git

Authorize bash to access GitHub protected branch

This might be a silly question, but here I am after two days of head banging 😔
I'm currently working on a GitHub Actions job, that would at some point git push to a protected branch (master, actually). However, once it gets to this step, the job fails with a "You're not authorized to push" error message:
error code GH006
In my situation, the first time it happened, it meant that the worker (CLI, bash) is not authorized at all. So I went to log it in a dedicated GitHub account, — and this is where I'm stuck for a couple of days now.
So, how do I login to GitHub account via CLI?
The things I've tried:
Pushing to master manually
This works without errors, since the authorization is OK, — but it is obviously not automated, which is what I'm after.
A couple of details:
I was using Windows 10, whereas the job is set up to run on "ubuntu-18.04";
I've removed all GitHub-related logins from Windows Credential Manager before performing git push, and set them up again via GitHub Login dialog window.
Manually pushing under unauthorized credentials fails, as expected.
git remote set-url origin "https://$username:$token#github.com/my/repo"
This didn't seem to give any effect. I've tried both setting URL of the existing remote and removing-than-adding remotes with different URLs, — both approaches seem to work (not work) the same.
None of the configurations below worked:
steps:
- run: git remote set-url origin "https://$username:$token#github.com/my/repo"
- run: git push origin master
steps:
- run: git remote remove origin
- run: git remote add origin "https://$username:$token#github.com/my/repo"
- run: git fetch origin --all # with and without this step
- run: git push origin master
curl -u "$username:$token" https://api.github.com/user
This is suggested in the docs, and it does succeed, but the login does not persist until git push — even if the pushing happens in the same step. I suspect, there might be a cookie-related solution, but I'm not sure how do they work in a non-browser environment. Also, I believe that this API is designed for different purposes.
Both of these configurations failed:
steps:
# separate processes
- run: curl -u "$username:$token" https://api.github.com/user
- run: git push origin master
steps:
# same process
- run: |
curl -u "$username:$token" https://api.github.com/user
git push origin master
actions/checkout#v2 will now configure and persist authentication when setting the token input. You shouldn't need to configure the origin URL manually.
- uses: actions/checkout#v2
with:
token: ${{ secrets.PAT }}
- name: Create a change
run: echo "test" > test.txt
- name: Commit change
run: |
git config --global user.name 'Your Name'
git config --global user.email 'your-username#users.noreply.github.com'
git add -A
git commit -m "Add test file"
git push
According to this comment on the GitHub forums, the PAT must be created from an admin/org owner account. A collaborator with write access is not enough to push to protected branches.

Cannot push from gitlab-ci.yml

With my colleagues, we work on a C++ library that becomes more and more important each day. We already built continuous integration utilities through the gitlab-ci.yml file that let us:
Build & Test in Debug mode
Build & Test in Release mode
Perform safety checks like memory leaks using Valgrind and checking if there is any clear symbol in our library we don't want inside it
Generate documentation
All kind of stuff that made us choose GitLab !
We would like to profile our whole library and push the benchmarks in a separate project. We have already done something like for out documentation using the SSH key method but we would like to avoid this this time.
We tried a script like this:
test_ci_push:
tags:
- linux
- shell
- light
stage: profiling
allow_failure: false
only:
- new-benchmark-stage
script:
- git clone http://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.mycompany.home/developers/benchmarks.git &> /dev/null
- cd benchmarks
- touch test.dat
- echo "This is a test" > test.dat
- git config --global user.name "${GITLAB_USER_NAME}"
- git config --global user.email "${GITLAB_USER_EMAIL}"
- git add --all
- git commit -m "GitLab Runner Push"
- git push http://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.mycompany.home/developers/benchmarks.git HEAD:master
- cd ..
We also tried a basic git push origin master to push our updated files but each time we got the same answer:
remote: You are not allowed to upload code for this project.
fatal: unable to access 'http://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx#gitlab.mycompany.home/developers/benchmarks.git/': The requested URL returned error: 403
Both projects are under the same site and I have the rights to push in both. Where am I doing anything wrong here ?
The gitlab ci token is more like the deploy key in github.com, so it only has read access to the repository. To actually push you will need to generate a personal access token and use that instead.
First you need to generate the token as shown here in the gitlab documentation. Make sure you check both the read user and api scopes. Also this only works in GitLab 8.15 and above. If you are using an older version and do not wish to upgrade there's an alternative method I could show you but it is more complex and less secure.
In the end your gitlab-ci.yml should look something like this:
test_ci_push:
tags:
- linux
- shell
- light
stage: profiling
allow_failure: false
only:
- new-benchmark-stage
script:
- git clone http://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.mycompany.home/developers/benchmarks.git &> /dev/null
- cd benchmarks
- echo "This is a test" > test.dat
- git config --global user.name "${GITLAB_USER_NAME}"
- git config --global user.email "${GITLAB_USER_EMAIL}"
- git add --all
- git commit -m "GitLab Runner Push"
- git push http://${YOUR_USERNAME}:${PERSONAL_ACCESS_TOKEN}#gitlab.mycompany.home/developers/benchmarks.git HEAD:master
- cd ..
While the previous answers are more or less fine, there are some important gotya's.
before_script:
- git config --global user.name "${GITLAB_USER_NAME}"
- git config --global user.email "${GITLAB_USER_EMAIL}"
script:
- <do things>
- git push "https://${GITLAB_USER_LOGIN}:${CI_GIT_TOKEN}#${CI_REPOSITORY_URL#*#}" "HEAD:${CI_COMMIT_TAG}"
For one, we only need to set the username/email to please git.
Secondly having it in the before script, is not super crucial, but allows for easier reuse when doing 'extend'.
Finally, pushing https is 'fine' but since we're not using a stored ssh key, we should avoid anything that can reveal the token. For one, while gitlab won't print the token in this command, git will happily inform us that the new upstream is set to https://username:thetokeninplaintexthere#url
So there's your token in plain text, so don't use -u to set an upstream.
Also, it's not needed, we are only doing a single push.
Further more, when determining the URL, I found that using the exist CI_REPOSITORY_URL to be the most reliable solution (when moving repo's for example or whatnot). So we just replace the username/token in the URL string.
You could also provide user and password (user with write access) as secret variables and use them.
Example:
before_script:
- git remote set-url origin https://$GIT_CI_USER:$GIT_CI_PASS#$CI_SERVER_HOST/$CI_PROJECT_PATH.git
- git config --global user.email 'myuser#mydomain.com'
- git config --global user.name 'MyUser'
You have to define GIT_CI_USER and GIT_CI_PASS as secret variables (you could always create dedicated user for this purpose).
With this configuration you could normally work with git. I'm using this approach to push the tags after the release (with Axion Release Gradle Pluing - http://axion-release-plugin.readthedocs.io/en/latest/index.html)
Example release job:
release:
stage: release
script:
- git branch
- gradle release -Prelease.disableChecks -Prelease.pushTagsOnly
- git push --tags
only:
- master
I'm using the following GitLab job:
repo_pull_sync:
image: danger89/repo_mirror_pull:latest
rules:
- if: '$CI_PIPELINE_SOURCE == "schedule"'
- if: $REMOTE_URL
- if: $REMOTE_BRANCH
- if: $ACCESS_TOKEN
before_script:
- git config --global user.name "${GITLAB_USER_NAME}"
- git config --global user.email "${GITLAB_USER_EMAIL}"
script:
- git checkout $CI_DEFAULT_BRANCH
- git pull
- git remote remove upstream || true
- git remote add upstream $REMOTE_URL
- git fetch upstream
- git merge upstream/$REMOTE_BRANCH
- git push "https://${GITLAB_USER_LOGIN}:${ACCESS_TOKEN}#${CI_REPOSITORY_URL#*#}" "HEAD:${CI_DEFAULT_BRANCH}"
I'm using my own danger89/repo_mirror_pull docker image based on alpine, check this GitHub repository for more info.
This GitLab job pull upstream changes from the predefined remote repository + branch (see variables below), and merge them locally in CI/CD and pushes them in GitLab again.
Basically I created a repository pull mirror (which is officially not available for free on GitLab CE, only a push mirror is supported in GitLab).
Create in GitLab a Project Access Token first in GitLab. Via: Settings->Access Tokens. Check 'api' as the scope.
Create a new schedule, via: CI/CD->Schedules->New schedule. With the following 3 variables:
REMOTE_URL (example: https://github.com/project/repo.git)
REMOTE_BRANCH (example: master)
ACCESS_TOKEN: (see Access Token in the first step! Example: gplat-234hcand9q289rba89dghqa892agbd89arg2854, )
Save pipeline schedule
Again, see also: https://github.com/danger89/repo_pull_sync_docker_image
Regarding the question, see the git push command above, which allows you to push changes back into GitLab using GitLab (project) access token.

Push into master branch from Travis CI

I am switching from Jenkins to Travis CI.
In Jenkins, I did not have to write a script to push my Java/android library to Git master branch.
With Travis, all my research shows that I need to write a custom bash script to execute in after_success.
This is my deploy.sh:
#!/bin/bash
rev=$(git rev-parse --short HEAD)
git config user.name "uname"
git config user.password "password"
git add .
git commit -m "travis commit at ${rev}"
git push origin master
and my .travis.yml:
branches:
only:
- development
language: android
sudo: false
android:
components:
- build-tools-22.0.1
- android-22
script:
- cd appdir
- ./gradlew test
after_success:
- cd ..
- ./deploy.sh
before_cache:
- rm -f $HOME/.gradle/caches/modules-2/modules-2.lock
cache:
directories:
- $HOME/.gradle/caches/
- $HOME/.gradle/wrapper/
Under the script section, I cd from root dir to my appdir and run tests from there (successfully) then in the after_success section, I cd back into root where my deploy.sh is located and call it.
My Travis CI console shows everything is successful but I don't see any changes in my master branch.
Am I doing anything wrong?
Thanks.
My experience is that Git doesn't appear know the branch your build is using and you want to push the current HEAD that has the additional commit.
In Travis CI you may have a log message just showing 'Everything up-to-date' as Git is pushing the same master branch.
You could change your script to add HEAD.
#!/bin/bash
rev=$(git rev-parse --short HEAD)
git config user.name "uname"
git config user.password "password"
git add .
git commit -m "committed at ${rev}"
git push origin HEAD:master
This is a similar answer that explains :- git-pushing-to-github-origin-master-does-nothing.

Resources