private repo - go 1.13 - `go mod ..` failed: ping "sum.golang.org/lookup" .. verifying package .. 410 gone - go

I am using golang 1.13 .
I have a project that depends on a private gitlab project.
I have the ssh keys for the same.
When I try to retrieve the dependencies for a newly created module, I am getting the following error:
$ go version
go version go1.13 linux/amd64
$ go mod why
go: downloading gitlab.com/mycompany/myproject v0.0.145
verifying gitlab.com/mycompany/myproject#v0.0.145: gitlab.com/mycompany/myproject#v0.0.145: reading https://sum.golang.org/lookup/gitlab.com/mycompany/myproject#v0.0.145: 410 Gone
I have no idea why it is trying to ping sum.golang.org/lookup since it is a private gitlab project.
My ~/.gitconfig contains the following (based on my looking up in google search for similar errors)
# Enforce SSH
[url "ssh://git#github.com/"]
insteadOf = https://github.com/
[url "ssh://git#gitlab.com/"]
insteadOf = https://gitlab.com/
[url "ssh://git#bitbucket.org/"]
insteadOf = https://bitbucket.org/
[url "git#gitlab.com:"]
insteadOf = https://gitlab.com/
The error still persists.
I would expect the package to be downloaded from my private gitlab project repository to the current project.
Is there anything I need to do in my private gitlab project repository to make it ready for 'go get' ?
The private gitlab project repository already contains the go.sum and go.mod for the project as well.
Anything that I am missing ?
edit: 1) The private repo name and the company name contains no asterisks or any other special characters. only alphabets and not even numeric characters.

Answering my own question after looking up,
Setting the GOPRIVATE variable seems to help.
GOPRIVATE=gitlab.com/mycompany/* go mod why
"
The new GOPRIVATE environment variable indicates module paths that are not publicly available. It serves as the default value for the lower-level GONOPROXY and GONOSUMDB variables, which provide finer-grained control over which modules are fetched via proxy and verified using the checksum database.
" from https://golang.org/doc/go1.13
Aliter:
Setting the env variable GONOSUMDB also seems to work.
Specifically, invoking the following command seems to help.
GONOSUMDB=gitlab.com/mycompany/* go mod why
The above env variable prevents the ping to sum.golang.org/lookup for a checksum match. It also prevents leaking the names of private repos to a public checksum db. [ Source - https://docs.gomods.io/configuration/sumdb/ ]
Also - here at
* GONOSUMDB=prefix1,prefix2,prefix3 sets a list of module path prefixes, again possibly containing globs, that should not be looked up using the database.
source: https://go.googlesource.com/proposal/+/master/design/25530-sumdb.md
Related Issues:
https://github.com/golang/go/issues/32291
https://github.com/golang/go/issues/33985
["Go 1.13 has been released, and this issue was filed well after the freeze window. The proposed changes will not happen in 1.13, but don't assume they will necessarily happen in 1.14 either." from issue 33985 above. ]

Basically it failed to verify private repository. However I don't like turning off checksum, but you can easily set GOSUMDB to off before trying to get module. something like this:
GOSUMDB=off go get github.com/mycompany/myproject
ref: https://github.com/golang/go/issues/35164#issuecomment-546503518
A second and better solution is to set GOPRIVATE environment variable that controls which modules the go command considers to be private (not available publicly) and should therefore NOT use the proxy or checksum database. The variable is a comma-separated list of glob patterns (same syntax of Go's path.Match) of module path prefixes. For example,
export GOPRIVATE=*.corp.example.com,rsc.io/private
Or
go env -w GOPRIVATE=github.com/mycompany/*
Last solution you can try is to turn off such checks for all private repositories that you don't want to go public or being verified through sum.golang.org/lookup/github.com/mycompany/...
GONOSUMDB=gitlab.com/mycompany/* go mod why
Note that:
If you have issues fetching modules or repos over https, you may want to add the following to your ~/.gitconfig to make go get/fetch repositories using ssh instead of https
[url "ssh://git#github.com/"]
insteadOf = https://github.com/

Change following go variable's setting and then upgrade your package,
$ export GO111MODULE=on
$ export GOPROXY=direct
$ export GOSUMDB=off
$ go get -u <your dependency package>

I have this scenario too and this works for me.
edit your .git/config and add two lines in it.( I have this in a global .gitconfig in home dir)
[url "ssh://youprivate.com"]
insteadOf = https://yourprivate.com
export GOSUMDB=off
Then everything will OK.

Related

Invalid version control suffix in bitbucket.org/ path when importing private repo in golang

The steps I took:
1- Configure .gitconfig
[url "ssh://git#bitbucket.org/"]
insteadOf = https://bitbucket.org/
2- export GOPRIVATE=bitbucket.org/myproject/helpers-go
3- go get
my go.mod file ->
module bitbucket.org/myproject/x/test-service
go 1.16
require (
bitbucket.org/myproject/helpers-go v0.0.2
)
replace bitbucket.org/myproject/helpers-go => bitbucket.org/myproject/helpers-go.git v0.0.2
My ssh key works. I can push/pull each services.
I also have the v0.0.2 tag on my last helpers-go commit.
I am trying to import private repo(helpers-go) into (test-service) and the error I get is :
go: bitbucket.org/myproject/helpers-go#v0.0.2: invalid version control suffix in bitbucket.org/ path
Even if I change the v0.0.2 to something random like v0.0.9 which I don`t have a tag like this, I still get the same error.
Appreciate the help...
it is solved:
1- I was using go 1.16 and there was an issue with bitbucket private repos not being supported for this version. I don`t think it has the same issue with gitlab. Upgrading it to go 1.19
2- Small change here .org/ -> .org:
[url "git#bitbucket.org:"]
insteadOf = https://bitbucket.org/
3- Also making sure the private repo I am importing has a file that ends with .go. My .go files were all in other folders leaving main folder without a .go file.

Using a private go module on Gitlab as import : "Unknown revision"

I cannot get this to work, even after checking other topics on stackoverflow. My project on gitlab.com/my_company/backend needs a module, found at gitlab.com/my_company/pkg/auth.
Locally, I've setup GOPRIVATE / git's configuration to make it work (and it works), though in Gitlab's CI Pipelines on a merge request, this fails.
Pipeline log / go.mod
I've added some debugging logs just to make sure everything was setup like I thought. Here's a failing pipeline's log :
$ git config --global url."ssh://git#gitlab.com/my_company/".insteadOf "https://gitlab.com/my_company/"
$ git config --global url."git#gitlab.com:".insteadOf "https://gitlab.com/"
$ git config -l | grep instead
url.ssh://git#gitlab.com/my_company/.insteadof=https://gitlab.com/my_company/
url.git#gitlab.com:.insteadof=https://gitlab.com/
$ env | grep GOPRIVATE
GOPRIVATE=gitlab.com/my_company
$ go mod download
go: gitlab.com/my_company/pkg/auth#v1.1.0: reading gitlab.com/my_company/pkg/auth/auth/go.mod at revision auth/v1.1.0: unknown revision auth/v1.1.0
One weird part of this log I've found is :
reading gitlab.com/my_company/pkg/auth/auth/go.mod - why is it repeating auth/auth? It actually happened once before locally, but it was because I wrote "github" instead of "gitlab" :)
The relevant go.mod line just in case :
require (
gitlab.com/my_company/pkg/auth v1.1.0 // indirect
)
Repository tags
Here are the tags setup on the repository gitlab.com/my_company/pkg :
$ git tag -l
auth/v1.0.0
auth/v1.1.0
cache/v1.0.0
cache/v1.0.1
$ git ls-remote --tags
From git#gitlab.my_company/pkg.git
9efcb02d5489adaac9d525dcb496d868d65e856a refs/tags/auth/v1.0.0
13730d4f61df978c6d690fd2678e2ed924808e0c refs/tags/auth/v1.1.0
2b8dff0ec1b737d975290720933180a9b591a1db refs/tags/cache/v1.0.0
9a3e598bbf83bea57b29d8a908b514861ae37b12 refs/tags/cache/v1.0.1
I'm not that familiar with Gitlab CI so I'm out of things to try. Any ideas?
Thank you!
Update: I finally got gitlab-runner installed so I could try running the yml directly, no luck. It still works locally (not a big surprise).
In you project should be file .gitlab-ci.yml and you can add GOPRIVATE variable to your CI and runner will use it for you project.
More details how to add env vars:
https://docs.gitlab.com/ee/ci/variables/#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file

go mod init with private bitbucket repositories, contains disallowed path separator character ':'

Using:
% go version
go version go1.16.4 darwin/amd64
I've got a private bitbucket.org repository with a url like:
git#bitbucket.org:chmorgan/some_repo.git
Per the bitbucket url format you would do 'git clone git#bitbucket.org:chmorgan/some_repo.git' to clone it and cloning works fine.
When I try to make a new module I get:
go mod init git#bitbucket.org:chmorgan/some_repo.git
go: invalid module path "git#bitbucket.org:chmorgan/some_repo.git": contains disallowed path separator character ':'
Note that I've already configured git in ~/.gitconfig:
[url "git#bitbucket.org:"]
insteadOf = https://bitbucket.org/
This used to work with earlier golfing versions, I think it worked with golang 1.12 or 1.13. Anyone know what's up?
The argument to go mod init is the “module path”, which is used as a prefix of the package import path for every package within the module. As a result, the argument to go mod init must be valid as a package import path.
(See https://golang.org/doc/code for more detail.)
The package import path we would normally use for a repo cloned from git#bitbucket.org:chmorgan/some_repo.git is bitbucket.org/chmorgan/some_repo.

Go get is pulling the wrong repository

My module is gitlab.com/getsote/utilities/slogger
My repository is gitlab.com/getsote/utilities/slogger.git
When I run go get gitlab.com/getsote/utilities/slogger, I get the message below.
Scotts-Mac-mini:seeding syacko$ go get gitlab.com/getsote/utilities/slogger
go get gitlab.com/getsote/utilities/slogger: module gitlab.com/getsote/utilities/slogger: git ls-remote -q origin in /Users/syacko/workspace/sotesoft/golang/pkg/mod/cache/vcs/80b3644beae1b986f1c659355360479e2463820660aa328d2edb1e571aba259b: exit status 128:
remote: The project you were looking for could not be found.
fatal: repository 'https://gitlab.com/getsote/utilities.git/' not found
Scotts-Mac-mini:seeding syacko$
The gitlab.com/getsote/utilities.git is a sub-directory and not a repository. I don't understand why go get is going to the utilities as a repository?
==========================
PREVIOUS Updates
Directory Structure:
GOPATH/src/slogger
|----go.mod
|----slogger.go
|----slogger_test.go
go.mod file
module slogger or gitlab.com/getsote/utilities/slogger -> still gets the error below
go 1.14
gitlab.com/getsote/utilities contains repository slogger.git
I have run a test to see if the issue is the number of nodes in the path. So, I create a new repository with no sub-directory and pushed the slogger code. Then ran go get gitlab.com/getsote/slogger which generate a different error message.
GOPATH/gitlab.com/getsote/test-go-mod -> create new directory and added slogger files listed above
gitblab.com/getsote/test-go-mod -> new repository with one less level
Scotts-Mac-mini:test-go-mod syacko$ go get gitlab.com/getsote/test-go-mod
go: downloading gitlab.com/getsote/test-go-mod v0.0.0-20200409023538-794310bf7cf9
go get gitlab.com/getsote/test-go-mod: gitlab.com/getsote/test-go-mod#v0.0.0-20200409023538-794310bf7cf9: verifying module: gitlab.com/getsote/test-go-mod#v0.0.0-20200409023538-794310bf7cf9: reading https://sum.golang.org/lookup/gitlab.com/getsote/test-go-mod#v0.0.0-20200409023538-794310bf7cf9: 410 Gone
server response:
not found: gitlab.com/getsote/test-go-mod#v0.0.0-20200409023538-794310bf7cf9: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /tmp/gopath/pkg/mod/cache/vcs/7753c92c9bd1419156d8120684b7f3707fd207e01a2947ba89e2acfd2ecfb4d0: exit status 128:
fatal: could not read Username for 'https://gitlab.com': terminal prompts disabled
Scotts-Mac-mini:test-go-mod syacko$
This is still getting the status error of 128 for the missing version. Additionally, it is looking in the right location for the code. If this is true, then I just need help with the version missing. Moving to a shorted directory structure is doable.
========================
Newest Update
#praveent > The solution at https://medium.com/cloud-native-the-gathering/go-modules-with-private-git-repositories-dfe795068db4 didn't work for me. So I started from scratch to see how to resolve the issue.
The reason is because for a git repository it assumes that utilities is the repo and not utilities/slogger
There is a way to override this behavior by implementing go get API. But, gitlab is yet to implement the same due to security concerns. You can read more here. Gitlab issue
Update: Add reference to gitlab issue tracking this problem.
So, here is how I got this to work using gitlab.com. I'm not saying other ways will not work, they just didn't for me and my setup. First, since I don't care if the code is available to the public, I created a new group at gitlab.com. This new group is public from the start, so no need to adjust permissions. Then I create a repository called packages and cloned the repository to my local machine with the same directory structure that is in gitlab.com, gitlab.com/soteapps/packages with ~/workspace/soteapps/packages on my machine. Both of these are out side the GOPATH. I'm not sure this matters, but it is working this way, so I'm putting it here.
Under packages, I copied the slogger directory and code.
cp -R slogger ~/workspace/soteapps/packages/.
Edited the go.mod file to match the repository structure, which is in the packages directory. There is no go.mod file in the slogger directory.
module gitlab.com/soteapps/packages
go 1.14
Edited the hello.go import to match the package.
package main
import (
"fmt"
"rsc.io/quote"
"gitlab.com/soteapps/packages/slogger"
)
func main() {
fmt.Println(quote.Hello())
slogger.Info("Test message")
}
Built the program using go build -o hello and then ran it hello with the following results:
Scotts-Mac-mini:hello syacko$ hello
Hello, world.
INFO:2020/04/10 21:11:33 Test message
Scotts-Mac-mini:hello syacko$
Worked! Thank you all that helped. This wouldn't of gotten solved without your help.
Note: This only works for public repositories.

I want to remove my private package from Goproxy

It's a go module question.
My PC is running with go 1.13 and go module mode is on.
This my go env:
GO111MODULE="on"
GOPRIVATE=""
GOPROXY="https://goproxy.io,direct"
At first I wrote a public package for testing and uploaded it to github. And then I "go get" this package for independence. After that I made it private and even deleted it from github (I tried "git clone" and couldn't download it again). But I could "go get" it after I removed it from github. Soon I realized it that maybe this package was cached in goproxy.io or other proxy databases. So I set GOPROXY="direct" and "go get" this package again and failed. This action proved my guess.
Now it's my question:
How to remove this whole package from goproxy.io or other database.
Is there a safe way to use go module, I don't want to upload my private code to other databases by mistake.
I tried to STFW and found nothing. Thank U to all people for reading and answering this question.
If you accidentally published a package / module you intend to be private, then go.dev: About page:
Removing a package
If you would like a package removed, please send an email to go-discovery-feedback#google.com, with the import path or module path that you want to remove.
But as Adrian mentioned in the comments, there is no guarantee that no one downloaded your published packages.
If you want some packages / modules to remain private, you may enumerate them (using glob patterns) in the GOPRIVATE, GONOPROXY, GONOSUMDB environment variables, which are respected by the go tool.
Command go: Environment variables:
GOPRIVATE, GONOPROXY, GONOSUMDB
Comma-separated list of glob patterns (in the syntax of Go's path.Match)
of module path prefixes that should always be fetched directly
or that should not be compared against the checksum database.
See 'go help module-private'.
From command go help module-private:
The GOPRIVATE environment variable controls which modules the go command
considers to be private (not available publicly) and should therefore not use the
proxy or checksum database. The variable is a comma-separated list of
glob patterns (in the syntax of Go's path.Match) of module path prefixes.
These vars can also be set using the go env -w command. You may get more help about it using go help env.
For Question2, I have tried several times today.
If u want to build a private package safely, the most important thing you should do is setting the GOPRIVATE BEFORE u build your package. And GONOPROXY,GONOSUMDB will be automatically set to same as GOPRIVATE.
Now u can write the code and git push to a private remote repo and try to go get it, and u will fail and see some error message like it:
fatal: could not read Username for 'https://github.com': terminal prompts disabled
Confirm the import path was entered correctly.
If this is a private repository, see https://golang.org/doc/faq#git_https for additional information.
The link show us that we should use https link with your username/password or just replace your https link with ssh link if u upload a key to remote ropo. After doing it, u will succeed to go get it from your source address (your package name) but NOT PROXY.
Futhurmore, let's delete all go module cache by using go clean -modcache and set GOPRIVATE="". I will test whether our private package was uploaded to proxy.
case 1:
Set GOPROXY="https://goproxy.io" or other proxy and thenen go get the private package.
u will face a 404 Not Found which show u that u find NOTHING from proxy (it meant nothing was cached/stored in proxy) .
case 2:
Set GOPROXY="direct"(use the direct address of your pacakage name) and then go get the private package.
u will face error message like it:
verifying {PACKAGENAME}: {PACKAGENAME}: initializing sumweb.Conn: checking tree#{ID}: Get https://sum.golang.org/tile/8/2/000.p/11: dial tcp {IP}:{PORT}: i/o timeout.
The go get function will try to check the checksum from golang.org database with your package (I can't link to google server directly so it shows timeout error / I guess u will get 404 if u can link to google server). It shows that if we try to go get a private package with not setting GOPRIVATE, u will fail because u can not pass the check.
Suggestion:
If you want to build your private go package, u should set your GOPRIVATE firstly and make sure it is wide enough to INCLUDE your package name.
Thank U to all people for reading and answering this question.

Resources