I am trying to build a go application that makes use of git repositories that are private. I have an ssh key for my github account and have the following in my .gitconfig file:
[url "https://username:token#github.com"]
insteadOf = https://github.com
[url "ssh://git#github.com/"]
insteadOf = https://github.com/
when I execute go test or go build I am asked for the passphrase. I then get a response:
go: found github.com/x/businesses in github.com/x/businesses v0.0.0-yy
go: cmd/main/cmd imports
github.com/x/businesses: github.com/x/businesses#v0.0.0-yy/go.mod:
verifying module: github.com/x/businesses#v0.0.0-yy/go.mod:
reading https://sum.golang.org/lookup/github.com/xx/businesses#v0.0.0-yy: 410 Gone
server response:
not found: github.com/x/businesses#v0.0.0-yy:
invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/*
in tmp/gopath/pkg/mod/cache/vcs/zz:
exit status 128:
fatal: could not read Username for 'https://github.com': terminal prompts disabled
I tried removing the top insteadOf in the .gitconfig for no reason other then to try something.
Your git config should look something like
[url "git#github.com:"] insteadOf = https://github.com/
or similar to what you have.
Then you can go get your pacakge by telling Go that your using a private repo like this:
GOPRIVATE="github.com/your_username_or_org" go get github.com/name_or_org/repo_name
Also, I usually dont put a passprahse on my SSH keys since is kinda annoying to type it in each time but of course is more secure by adding it like someone pointed out in the comments.
Related
The steps I took:
1- Configure .gitconfig
[url "ssh://git#bitbucket.org/"]
insteadOf = https://bitbucket.org/
2- export GOPRIVATE=bitbucket.org/myproject/helpers-go
3- go get
my go.mod file ->
module bitbucket.org/myproject/x/test-service
go 1.16
require (
bitbucket.org/myproject/helpers-go v0.0.2
)
replace bitbucket.org/myproject/helpers-go => bitbucket.org/myproject/helpers-go.git v0.0.2
My ssh key works. I can push/pull each services.
I also have the v0.0.2 tag on my last helpers-go commit.
I am trying to import private repo(helpers-go) into (test-service) and the error I get is :
go: bitbucket.org/myproject/helpers-go#v0.0.2: invalid version control suffix in bitbucket.org/ path
Even if I change the v0.0.2 to something random like v0.0.9 which I don`t have a tag like this, I still get the same error.
Appreciate the help...
it is solved:
1- I was using go 1.16 and there was an issue with bitbucket private repos not being supported for this version. I don`t think it has the same issue with gitlab. Upgrading it to go 1.19
2- Small change here .org/ -> .org:
[url "git#bitbucket.org:"]
insteadOf = https://bitbucket.org/
3- Also making sure the private repo I am importing has a file that ends with .go. My .go files were all in other folders leaving main folder without a .go file.
I'm trying to clone a repo using the following syntax inside a bash script
git clone "https://oauth2:[TOKEN]#github.com/[organization]/$reponame.git $REPOPATH/$reponame"
and I get the following error:
Cloning into 'protos-cusum_hmm-python'...
fatal: unable to access 'https://github.com/[organization]/protos-cusum_hmm-python.git /opt/protolangs/protos-cusum_hmm-python/': The requested URL returned error: 400
when I clone directly from command line git clone https://github.com/[organizaiton]/protos-cusum_hmm-python.git it works fine (presumable because it's using my cached credentials)
Any suggestions?
Edit:
Removed quotations around url (git clone https://oauth2:[TOKEN]#github.com/[organization]/$reponame.git $REPOPATH/$reponame) and now getting
remote: Invalid username or password.
fatal: Authentication failed for 'https://github.com/fluxusinc/protos-cusum_hmm-python.git/'
EDIT2:
When hardcoded with token:
git clone "https://ghp_...O17xckH#github.com/[organization]/"$reponame".git" $REPOPATH/$reponame
it works, so seems like it's a problem with the YAML
- name: Run proto builder and deploy to repos
env:
P_TOKEN: ${{ secrets.REPO_TOKEN }}
run: |
chmod +x "${GITHUB_WORKSPACE}/build.sh"
"$GITHUB_WORKSPACE/build.sh"
shell: bash
much thanks to #torek....
turns out for whatever reason I had to put the secret in the repo rather than the organization and it worked... not sure why that was the case because github is showing it overriding
MORE INFO:
was able to figure out by setting the secret ACTIONS_STEP_DEBUG to true from here.
yielded
##[debug]Evaluating: secrets.REPO_TOKEN_GRPC
##[debug]Evaluating Index:
##[debug]..Evaluating secrets:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'REPO_TOKEN_GRPC'
##[debug]=> null
##[debug]Result: null
showing that the token was empty for whatever reason
I cannot get this to work, even after checking other topics on stackoverflow. My project on gitlab.com/my_company/backend needs a module, found at gitlab.com/my_company/pkg/auth.
Locally, I've setup GOPRIVATE / git's configuration to make it work (and it works), though in Gitlab's CI Pipelines on a merge request, this fails.
Pipeline log / go.mod
I've added some debugging logs just to make sure everything was setup like I thought. Here's a failing pipeline's log :
$ git config --global url."ssh://git#gitlab.com/my_company/".insteadOf "https://gitlab.com/my_company/"
$ git config --global url."git#gitlab.com:".insteadOf "https://gitlab.com/"
$ git config -l | grep instead
url.ssh://git#gitlab.com/my_company/.insteadof=https://gitlab.com/my_company/
url.git#gitlab.com:.insteadof=https://gitlab.com/
$ env | grep GOPRIVATE
GOPRIVATE=gitlab.com/my_company
$ go mod download
go: gitlab.com/my_company/pkg/auth#v1.1.0: reading gitlab.com/my_company/pkg/auth/auth/go.mod at revision auth/v1.1.0: unknown revision auth/v1.1.0
One weird part of this log I've found is :
reading gitlab.com/my_company/pkg/auth/auth/go.mod - why is it repeating auth/auth? It actually happened once before locally, but it was because I wrote "github" instead of "gitlab" :)
The relevant go.mod line just in case :
require (
gitlab.com/my_company/pkg/auth v1.1.0 // indirect
)
Repository tags
Here are the tags setup on the repository gitlab.com/my_company/pkg :
$ git tag -l
auth/v1.0.0
auth/v1.1.0
cache/v1.0.0
cache/v1.0.1
$ git ls-remote --tags
From git#gitlab.my_company/pkg.git
9efcb02d5489adaac9d525dcb496d868d65e856a refs/tags/auth/v1.0.0
13730d4f61df978c6d690fd2678e2ed924808e0c refs/tags/auth/v1.1.0
2b8dff0ec1b737d975290720933180a9b591a1db refs/tags/cache/v1.0.0
9a3e598bbf83bea57b29d8a908b514861ae37b12 refs/tags/cache/v1.0.1
I'm not that familiar with Gitlab CI so I'm out of things to try. Any ideas?
Thank you!
Update: I finally got gitlab-runner installed so I could try running the yml directly, no luck. It still works locally (not a big surprise).
In you project should be file .gitlab-ci.yml and you can add GOPRIVATE variable to your CI and runner will use it for you project.
More details how to add env vars:
https://docs.gitlab.com/ee/ci/variables/#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file
I'm trying to use a private theme/module with a personal access token. I can get this working by adding the following to my global git config.
git config --global url."https://{USER}:{TOKEN}#github.com".insteadOf "https://github.com"
Then running hugo mod get -u it will pull changes as expected.
I don't want this set in my global config and if I set it locally I get an error, because Go doesn't seem to be using the local config.
Set my configurations locally within the root of the site/repository:
git config --local url."https://{USER}:{TOKEN}#github.com".insteadOf "https://github.com"
Then running hugo mod get -u I get the following error:
go get: module github.com/USER/REPOSITORY: git ls-remote -q origin in /var/folders/26/gqnv01_55p964v8yz39d51fw0000gn/T/hugo_cache/modules/filecache/modules/pkg/mod/cache/vcs/b410fc7b91fbc1121b5f6ec2bb2711c27cd172b4084c213e1430a33cde552597: exit status 128:
remote: Repository not found.
fatal: repository 'https://github.com/USER/REPOSITORY/' not found
How can I get Go/Hugo to use my local git config rather than the global?
From the hugo mod source code, hugo will look for a go.mod in your project:
filepath.Walk(dirname, func(path string, info os.FileInfo, err error) error {
if info.IsDir() {
return nil
}
if info.Name() == "go.mod" {
// Found a module.
dir := filepath.Dir(path)
fmt.Println("Update module in", dir)
Check where your go.mod is, and do (in that go.mod parent folder):
git config -l --show-origin --show-scope
That will tell you if your expected local config is actually there or not.
Look for any .git folder which would indicate a nested git repository/submodule, which would ignore your initial git config --local command
An issue like 34513 seems to suggests though that go mod won't take into account the local repository:
The git configuration only affects operations on the underlying git repo.
The error that you're seeing is coming from before that, when the go command is attempting to resolve the repo for the requested package path.
The official documentation only references the global config .gitconfig.
I solved this by adding a directory replacement mapping to the site’s config, instead of modifying the git url. This points to my locally cloned theme and updates the served site whenever I modify the theme.
module:
imports:
path: 'github.com/[USER]/[REPO-NAME]'
replacements: 'github.com/[USER]/[REPO-NAME] -> ../../[REPO-NAME]/'
I am using golang 1.13 .
I have a project that depends on a private gitlab project.
I have the ssh keys for the same.
When I try to retrieve the dependencies for a newly created module, I am getting the following error:
$ go version
go version go1.13 linux/amd64
$ go mod why
go: downloading gitlab.com/mycompany/myproject v0.0.145
verifying gitlab.com/mycompany/myproject#v0.0.145: gitlab.com/mycompany/myproject#v0.0.145: reading https://sum.golang.org/lookup/gitlab.com/mycompany/myproject#v0.0.145: 410 Gone
I have no idea why it is trying to ping sum.golang.org/lookup since it is a private gitlab project.
My ~/.gitconfig contains the following (based on my looking up in google search for similar errors)
# Enforce SSH
[url "ssh://git#github.com/"]
insteadOf = https://github.com/
[url "ssh://git#gitlab.com/"]
insteadOf = https://gitlab.com/
[url "ssh://git#bitbucket.org/"]
insteadOf = https://bitbucket.org/
[url "git#gitlab.com:"]
insteadOf = https://gitlab.com/
The error still persists.
I would expect the package to be downloaded from my private gitlab project repository to the current project.
Is there anything I need to do in my private gitlab project repository to make it ready for 'go get' ?
The private gitlab project repository already contains the go.sum and go.mod for the project as well.
Anything that I am missing ?
edit: 1) The private repo name and the company name contains no asterisks or any other special characters. only alphabets and not even numeric characters.
Answering my own question after looking up,
Setting the GOPRIVATE variable seems to help.
GOPRIVATE=gitlab.com/mycompany/* go mod why
"
The new GOPRIVATE environment variable indicates module paths that are not publicly available. It serves as the default value for the lower-level GONOPROXY and GONOSUMDB variables, which provide finer-grained control over which modules are fetched via proxy and verified using the checksum database.
" from https://golang.org/doc/go1.13
Aliter:
Setting the env variable GONOSUMDB also seems to work.
Specifically, invoking the following command seems to help.
GONOSUMDB=gitlab.com/mycompany/* go mod why
The above env variable prevents the ping to sum.golang.org/lookup for a checksum match. It also prevents leaking the names of private repos to a public checksum db. [ Source - https://docs.gomods.io/configuration/sumdb/ ]
Also - here at
* GONOSUMDB=prefix1,prefix2,prefix3 sets a list of module path prefixes, again possibly containing globs, that should not be looked up using the database.
source: https://go.googlesource.com/proposal/+/master/design/25530-sumdb.md
Related Issues:
https://github.com/golang/go/issues/32291
https://github.com/golang/go/issues/33985
["Go 1.13 has been released, and this issue was filed well after the freeze window. The proposed changes will not happen in 1.13, but don't assume they will necessarily happen in 1.14 either." from issue 33985 above. ]
Basically it failed to verify private repository. However I don't like turning off checksum, but you can easily set GOSUMDB to off before trying to get module. something like this:
GOSUMDB=off go get github.com/mycompany/myproject
ref: https://github.com/golang/go/issues/35164#issuecomment-546503518
A second and better solution is to set GOPRIVATE environment variable that controls which modules the go command considers to be private (not available publicly) and should therefore NOT use the proxy or checksum database. The variable is a comma-separated list of glob patterns (same syntax of Go's path.Match) of module path prefixes. For example,
export GOPRIVATE=*.corp.example.com,rsc.io/private
Or
go env -w GOPRIVATE=github.com/mycompany/*
Last solution you can try is to turn off such checks for all private repositories that you don't want to go public or being verified through sum.golang.org/lookup/github.com/mycompany/...
GONOSUMDB=gitlab.com/mycompany/* go mod why
Note that:
If you have issues fetching modules or repos over https, you may want to add the following to your ~/.gitconfig to make go get/fetch repositories using ssh instead of https
[url "ssh://git#github.com/"]
insteadOf = https://github.com/
Change following go variable's setting and then upgrade your package,
$ export GO111MODULE=on
$ export GOPROXY=direct
$ export GOSUMDB=off
$ go get -u <your dependency package>
I have this scenario too and this works for me.
edit your .git/config and add two lines in it.( I have this in a global .gitconfig in home dir)
[url "ssh://youprivate.com"]
insteadOf = https://yourprivate.com
export GOSUMDB=off
Then everything will OK.