In Jenkins, I have a multibranch pipeline job using a Jenkinsfile. I have disabled the default SCM checkout via:
options {skipDefaultCheckout(true)}
so that I can control the checkout for each stage and use the SSH URL from GitHub. I have a working stage that checks out the "main" branch using:
checkout([
$class: 'GitSCM',
branches: [[name: '*/main']],
extensions: [[$class: 'LocalBranch', localBranch: 'main']],
userRemoteConfigs: [[credentialsId: 'XXX', url: 'git#github.com:XXX.git']]
])
For another stage, I would like to checkout and build/test the branch that is triggered (in this case a feature branch called "feature-0001"). I have tried this (note the change for the branches and extensions lines):
checkout([
$class: 'GitSCM',
branches: [[name: env.BRANCH_NAME]],
extensions: [[$class: 'LocalBranch']],
userRemoteConfigs: [[credentialsId: XXX, url: 'git#github.com:XXX.git']]
])
but I get the following error:
Error when executing failure post condition:
java.io.IOException: Cannot retrieve Git metadata for the build
In addition, the build log also has:
git rev-parse "origin/feature-0001^{commit}" # timeout=10
git rev-parse "feature-0001^{commit}" # timeout=10
I'm not sure where the "^{commit}" comes from or if it's causing any issues.
Any guidance would be appreciated, thank you!
P.S. I have also tried many variations of BRANCH_NAME including
$env.BRANCH_NAME, ${env.BRANCH_NAME}, $BRANCH_NAME, ${BRANCH_NAME}
branches: [[name: "refs/heads/${env.BRANCH_NAME}"]] worked.
I was also using the incorrect url/credentials causing further issues.
If you want an unambiguous way to reference your branch name in the SCM checkout step, try (as explained here):
branches: [[name: 'refs/heads/${env.BRANCH_NAME}']]
The ^{commit} revision specification would therefore apply to the ref refs/heads/${env.BRANCH_NAME}
<rev>^{<type>}, e.g. v0.99.8^{commit}
A suffix ^ followed by an object type name enclosed in brace pair means dereference the object at <rev> recursively until an object of type <type> is found or the object cannot be dereferenced anymore (in which case, barf).
For example, if <rev> is a commit-ish, <rev>^{commit} describes the corresponding commit object.
I have n features branch that will MR and merge into a develop branch.
I have a pipeline with 3 stages:
stages:
- feature-push
- develop-mr-retag
- develop-mr-rollout
feature-push runs on any push to a feature branch (not the develop branch we are merging into). It will test, build, and push an app in a docker image tagged with the name of the feature branch.
The latter two stages should run on commits to a branch develop after a merge request is approved and merged (assuming the source branch passed the feature-push stage). It needs to rollout the new image to some k8s pods, and it needs the name of the source branch to find the correct image.
I want to use ${CI_MERGE_REQUEST_SOURCE_BRANCH_NAME} for this, but I don't think that variable exists for pipelines run after a merge, only on merge_requests pipelines. These seem to be triggered before an MR is approved, which I don't want as this is a deployment.
Is this possible or should I find a different approach?
**Edit: ** To clarify, I need to run my docker build before MR, to know it can build successfully. I don't want to just throw away that build if it's the one that gets merged, so that's why I want to build/push before MR, and deploy the previously built image after MR.
I'm looking for the same.
For the moment, I parse the commit title of the mergecommit to extract the source branch, and I check that the branch found really exist. Here is the relevant code :
CI_MERGE_REQUEST_SOURCE_BRANCH_NAME=$(sed -r "s/^Merge branch '(.*)' into .*/\1/i"<<<$CI_COMMIT_TITLE)
if [ $(git ls-remote --heads ${CI_REPOSITORY_URL} $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME | wc -l ) -ne 1 ]; then echo "Can't find source branche ${CI_MERGE_REQUEST_SOURCE_BRANCH_NAME})" && exit 1; fi
But if the default commit title is modified by anybody, this will not work.
I am using following code to checkout or switch the branch within python code,
repo.git.checkout('branch_name')
But when the code later executes is still referring to 'master' branch code.
I am using GitPython version 2.1.11.
import git
repo = git.Repo("/home/user/.emacs.d")
To checkout a branch:
to see available branches
>>> repo.heads
[<git.Head "refs/heads/master">, <git.Head "refs/heads/straight">]
you can use the branch name like this:
>>> repo.heads.straight.checkout()
<git.Head "refs/heads/straight">
the branch changed to straight
If you want to use git directly
>>> repo.git.checkout("master")
"Your branch is up-to-date with 'origin/master'."
the branch changed to master
In the circleci version 1 config, there was the option to specify owner as an option in a deployment. An example from the circleci docs ( https://circleci.com/docs/1.0/configuration/ ) with owner: circleci being the key line:
deployment:
master:
branch: master
owner: circleci
commands:
- ./deploy_master.sh
In version 2 of the config, there is the ability to use filters and tags to specify which branches are built, but I have yet to find (in the docs, or on the interwebs) anything that gives me the same capability.
What I'm trying to achieve is run build and test steps on forks, but only run the deploy steps if the repository owner is the main repo. Quite often people fork using the same branch name - in this case master - so having a build fail due to an inability to deploy is counter-intuitive, especially as I would like to use a protected branch in git and only merge commits based on a successful build in a pull request.
I realise we could move to only running builds based on tags being present, but nothing is stopping somebody with a fork also creating a tag in their fork, which puts us back at square one.
Is anybody aware of how to specify the owner of a repo in the version 2 config?
An example from the version 2 config document ( https://circleci.com/docs/2.0/workflows/ ) in case it helps jog somebodies memory:
workflows:
version: 2
un-tagged-build:
jobs:
- build:
filters:
tags:
ignore: /^v.*/
tagged-build:
jobs:
- build:
filters:
branches:
ignore: /.*/
tags:
only: /^v.*/
disclaimer: Developer Evangelist at CircleCI
That feature is not available on CircleCI 2.0. You can request it here.
As an alternative, you might be able to look for the branch name, say master, as well as the CIRCLE_PR_NUMBER environment variable. If that variable has any value, then it's a fork of master and you shouldn't deploy.
Suppose you have a repository at github.com/someone/repo and you fork it to github.com/you/repo. You want to use your fork instead of the main repo, so you do a
go get github.com/you/repo
Now all the import paths in this repo will be "broken", meaning, if there are multiple packages in the repository that reference each other via absolute URLs, they will reference the source, not the fork.
Is there a better way as cloning it manually into the right path?
git clone git#github.com:you/repo.git $GOPATH/src/github.com/someone/repo
If you are using go modules. You could use replace directive
The replace directive allows you to supply another import path that might
be another module located in VCS (GitHub or elsewhere), or on your
local filesystem with a relative or absolute file path. The new import
path from the replace directive is used without needing to update the
import paths in the actual source code.
So you could do below in your go.mod file
module some-project
go 1.12
require (
github.com/someone/repo v1.20.0
)
replace github.com/someone/repo => github.com/you/repo v3.2.1
where v3.2.1 is tag on your repo. Also can be done through CLI
go mod edit -replace="github.com/someone/repo#v0.0.0=github.com/you/repo#v1.1.1"
To handle pull requests
fork a repository github.com/someone/repo to github.com/you/repo
download original code: go get github.com/someone/repo
be there: cd "$(go env GOPATH)/src"/github.com/someone/repo
enable uploading to your fork: git remote add myfork https://github.com/you/repo.git
upload your changes to your repo: git push myfork
http://blog.campoy.cat/2014/03/github-and-go-forking-pull-requests-and.html
To use a package in your project
https://github.com/golang/go/wiki/PackageManagementTools
One way to solve it is that suggested by Ivan Rave and http://blog.campoy.cat/2014/03/github-and-go-forking-pull-requests-and.html -- the way of forking.
Another one is to workaround the golang behavior. When you go get, golang lays out your directories under same name as in the repository URI, and this is where the trouble begins.
If, instead, you issue your own git clone, you can clone your repository onto your filesystem on a path named after the original repository.
Assuming original repository is in github.com/awsome-org/tool and you fork it onto github.com/awesome-you/tool, you can:
cd $GOPATH
mkdir -p {src,bin,pkg}
mkdir -p src/github.com/awesome-org/
cd src/github.com/awesome-org/
git clone git#github.com:awesome-you/tool.git # OR: git clone https://github.com/awesome-you/tool.git
cd tool/
go get ./...
golang is perfectly happy to continue with this repository and doesn't actually care some upper directory has the name awesome-org while the git remote is awesome-you. All import for awesome-org are resovled via the directory you have just created, which is your local working set.
In more length, please see my blog post: Forking Golang repositories on GitHub and managing the import path
edit: fixed directory path
If your fork is only temporary (ie you intend that it be merged) then just do your development in situ, eg in $GOPATH/src/launchpad.net/goamz.
You then use the features of the version control system (eg git remote) to make the upstream repository your repository rather than the original one.
It makes it harder for other people to use your repository with go get but much easier for it to be integrated upstream.
In fact I have a repository for goamz at lp:~nick-craig-wood/goamz/goamz which I develop for in exactly that way. Maybe the author will merge it one day!
Here's a way to that works for everyone:
Use github to fork to "my/repo" (just an example):
go get github.com/my/repo
cd ~/go/src/github.com/my/repo
git branch enhancement
rm -rf .
go get github.com/golang/tools/cmd/gomvpkg/…
gomvpkg <<oldrepo>> ~/go/src/github.com/my/repo
git commit
Repeat each time when you make the code better:
git commit
git checkout enhancement
git cherry-pick <<commit_id>>
git checkout master
Why? This lets you have your repo that any go get works with. It also lets you maintain & enhance a branch that's good for a pull request. It doesn't bloat git with "vendor", it preserves history, and build tools can make sense of it.
Instead of cloning to a specific location, you can clone wherever you want.
Then, you can run a command like this, to have Go refer to the local version:
go mod edit -replace github.com/owner/repo=../repo
https://golang.org/cmd/go#hdr-Module_maintenance
The answer to this is that if you fork a repo with multiple packages you will need to rename all the relevant import paths. This is largely a good thing since you've forked all of those packages and the import paths should reflect this.
Use vendoring and submodules together
Fork the lib on github (go-mssqldb in this case)
Add a submodule which clones your fork into your vendor folder but has the path of the upstream repo
Update your import statements in your source code to point to the vendor folder, (not including the vendor/ prefix). E.g. vendor/bob/lib => import "bob/lib"
E.g.
cd ~/go/src/github.com/myproj
mygithubuser=timabell
upstreamgithubuser=denisenkom
librepo=go-mssqldb
git submodule add "git#github.com:$mygithubuser/$librepo" "vendor/$upstreamgithubuser/$librepo"
Why
This solves all the problems I've heard about and come across while trying to figure this out myself.
Internal package refs in the lib now work because the path is unchanged from upstream
A fresh checkout of your project works because the submodule system gets it from your fork at the right commit but in the upstream folder path
You don't have to know to manually hack the paths or mess with the go tooling.
More info
https://git-scm.com/book/en/v2/Git-Tools-Submodules
How do I fix the error message "use of an internal package not allowed" when go getting a golang package?
https://github.com/denisenkom/go-mssqldb/issues/406
https://github.com/golang/go/wiki/PackageManagementTools#go15vendorexperiment
The modern answer (go 1.15 and higher, at least).
go mod init github.com/theirs/repo
Make an explicit init arg that is the ORIGINAL package names. If you don't include the repo name, it will assume the one in gopath. But when you use go modules, they no longer care where they are on disk, or where git actually pulls dependencies from.
To automate this process, I wrote a small script. You can find more details on my blog to add a command like "gofork" to your bash.
function gofork() {
if [ $# -ne 2 ] || [ -z "$1" ] || [ -z "$2" ]; then
echo 'Usage: gofork yourFork originalModule'
echo 'Example: gofork github.com/YourName/go-contrib github.com/heirko/go-contrib'
return
fi
echo "Go get fork $1 and replace $2 in GOPATH: $GOPATH"
go get $1
go get $2
currentDir=$PWD
cd $GOPATH/src/$1
remote1=$(git config --get remote.origin.url)
cd $GOPATH/src/$2
remote2=$(git config --get remote.origin.url)
cd $currentDir
rm -rf $GOPATH/src/$2
mv $GOPATH/src/$1 $GOPATH/src/$2
cd $GOPATH/src/$2
git remote add their $remote2
echo Now in $GOPATH/src/$2 origin remote is $remote1
echo And in $GOPATH/src/$2 their remote is $remote2
cd $currentDir
}
export -f gofork
You can use command go get -f to get you a forked repo
in your Gopkg.toml file add these block below
[[constraint]]
name = "github.com/globalsign/mgo"
branch = "master"
source = "github.com/myfork/project2"
So it will use the forked project2 in place of github.com/globalsign/mgo