I want to remove my private package from Goproxy - go

It's a go module question.
My PC is running with go 1.13 and go module mode is on.
This my go env:
GO111MODULE="on"
GOPRIVATE=""
GOPROXY="https://goproxy.io,direct"
At first I wrote a public package for testing and uploaded it to github. And then I "go get" this package for independence. After that I made it private and even deleted it from github (I tried "git clone" and couldn't download it again). But I could "go get" it after I removed it from github. Soon I realized it that maybe this package was cached in goproxy.io or other proxy databases. So I set GOPROXY="direct" and "go get" this package again and failed. This action proved my guess.
Now it's my question:
How to remove this whole package from goproxy.io or other database.
Is there a safe way to use go module, I don't want to upload my private code to other databases by mistake.
I tried to STFW and found nothing. Thank U to all people for reading and answering this question.

If you accidentally published a package / module you intend to be private, then go.dev: About page:
Removing a package
If you would like a package removed, please send an email to go-discovery-feedback#google.com, with the import path or module path that you want to remove.
But as Adrian mentioned in the comments, there is no guarantee that no one downloaded your published packages.
If you want some packages / modules to remain private, you may enumerate them (using glob patterns) in the GOPRIVATE, GONOPROXY, GONOSUMDB environment variables, which are respected by the go tool.
Command go: Environment variables:
GOPRIVATE, GONOPROXY, GONOSUMDB
Comma-separated list of glob patterns (in the syntax of Go's path.Match)
of module path prefixes that should always be fetched directly
or that should not be compared against the checksum database.
See 'go help module-private'.
From command go help module-private:
The GOPRIVATE environment variable controls which modules the go command
considers to be private (not available publicly) and should therefore not use the
proxy or checksum database. The variable is a comma-separated list of
glob patterns (in the syntax of Go's path.Match) of module path prefixes.
These vars can also be set using the go env -w command. You may get more help about it using go help env.

For Question2, I have tried several times today.
If u want to build a private package safely, the most important thing you should do is setting the GOPRIVATE BEFORE u build your package. And GONOPROXY,GONOSUMDB will be automatically set to same as GOPRIVATE.
Now u can write the code and git push to a private remote repo and try to go get it, and u will fail and see some error message like it:
fatal: could not read Username for 'https://github.com': terminal prompts disabled
Confirm the import path was entered correctly.
If this is a private repository, see https://golang.org/doc/faq#git_https for additional information.
The link show us that we should use https link with your username/password or just replace your https link with ssh link if u upload a key to remote ropo. After doing it, u will succeed to go get it from your source address (your package name) but NOT PROXY.
Futhurmore, let's delete all go module cache by using go clean -modcache and set GOPRIVATE="". I will test whether our private package was uploaded to proxy.
case 1:
Set GOPROXY="https://goproxy.io" or other proxy and thenen go get the private package.
u will face a 404 Not Found which show u that u find NOTHING from proxy (it meant nothing was cached/stored in proxy) .
case 2:
Set GOPROXY="direct"(use the direct address of your pacakage name) and then go get the private package.
u will face error message like it:
verifying {PACKAGENAME}: {PACKAGENAME}: initializing sumweb.Conn: checking tree#{ID}: Get https://sum.golang.org/tile/8/2/000.p/11: dial tcp {IP}:{PORT}: i/o timeout.
The go get function will try to check the checksum from golang.org database with your package (I can't link to google server directly so it shows timeout error / I guess u will get 404 if u can link to google server). It shows that if we try to go get a private package with not setting GOPRIVATE, u will fail because u can not pass the check.
Suggestion:
If you want to build your private go package, u should set your GOPRIVATE firstly and make sure it is wide enough to INCLUDE your package name.
Thank U to all people for reading and answering this question.

Related

Go mod private repo on bitbucket

I have a project to do at my job and we're using Bitbucket. So we have all our repos like this :
bitbucket.org/company/project Nothing new here.
I have created a repository called go-tools, his module name is bitbucket.org/company/go-tools and his path his bitbucket.org/company/go-tools
Following this medium post I could achieve a go mod tidy
package whatever
import (
"bitbucket.org/company/go-tools"
"bitbucket.org/company/go-tools/env"
// and so on ...
)
The problem occurs when I try to replace "bitbucket.org/company" by "company.com" because we would like to have our company name instead bitbucket.
So my module name become company.com/go-tools instead of bitbucket.org/company/go-tools
And my imports become :
package whatever
import (
"company.com/go-tools"
"company.com/go-tools/env"
// and so on ...
)
I have set my GOPRIVATE to use bitbucket and configured git to use bitbucket instead of company.com
git config --global url."https://{username}:{app password}#bitbucket.com/company".insteadOf "https://company.com"
go env -w GOPRIVATE=bitbucket.org/company
And from there I only get a 404 error telling me that my package can't be found.
Did anyone have an idea why ? Am I misunderstanding something ?
NOTE : I also read this
Thanks #adrian for your reply this answer my question for at least a part.
I was more looking for a way of just go get 'company.com/whatever' but this is ok.
So if I understand correctly I need to go get bitbucket.org/company/whatever first and then go mod edit -replace bitbucket.org/company/whatever=company.com/whatever
Thanks

go-swagger restapi/configure_todo_list.go - api.TodoGetHandler undefined error

I am a newbie in go and go-swagger. I am following steps in Simple Server tutorial in goswagger.io.
I am using Ubuntu 18.04, swagger v0.25.0 and go 1.15.6.
Following the same steps, there are a few differences of the files generated. For instance, goswagger.io's has find_todos_okbody.go and get_okbody.go in models but mine does not. Why is that so?
Link to screenshot of my generated files vs
Link to screenshot of generated files by swagger.io
Starting the server as written in the tutorial go install ./cmd/todo-list-server/ gives me the following error. Can anyone please help with this?
# my_folder/swagger-todo-list/restapi
restapi/configure_todo_list.go:41:8: api.TodosGetHandler undefined (type *operations.TodoListAPI has no field or method TodosGetHandler)
restapi/configure_todo_list.go:42:6: api.TodosGetHandler undefined (type *operations.TodoListAPI has no field or method TodosGetHandler)
The first step in goswagger.io todo-list is swagger init spec .... Which directory should I run this command in? I ran it in a newly created folder in my home directory. However, from the page, it shows the path to be ~/go/src/github.com/go-swagger/go-swagger/examples/tutorials/todo-list. I am not sure whether I should use go get ..., git clone ... or create those folders. Can someone advise me?
Thanks.
This is likely the documentation lagging behind the version of the code that you are running. As long as it compiles, the specific files the tool generates isn't so crucial.
This is a compilation error. When you do go install foo it will try to build the foo package as an executable and then move that to your GOPATH/bin directory. It seems that the generated code in restapi/configure_todo_list.go isn't correct for the operations code generated.
All you need to run this tutorial yourself is an empty directory and the swagger tool (not its source code). You run the commands from the root of this empty project. In order not to run into GOPATH problems I would initialise a module with go mod init todo-list-example before doing anything else.
Note that while the todo-list example code exists inside the go-swagger source, it's there just for documenting example usage and output.
What I would advice for #2 is to make sure you're using a properly released version of go-swagger, rather than installing from the latest commit (which happens when you just do a go get), as I have found that to be occasionally unstable.
Next, re-generate the entire server, but make sure you also regenerate restapi/configure_todo_list.go by passing --regenerate-configureapi to your swagger generate call. This file isn't always refreshed because you're meant to modify it to configure your app, and if you changed versions of the tool it may be different and incompatible.
If after that you still get the compilation error, it may be worth submitting a bug report at https://github.com/go-swagger/go-swagger/issues.
Thanks #EzequielMuns. The errors in #2 went away after I ran go get - u -f ./... as stated in
...
For this generation to compile you need to have some packages in your GOPATH:
* github.com/go-openapi/runtime
* github.com/jessevdk/go-flags
You can get these now with: go get -u -f ./...
I think it's an error of swagger code generation. You can do as folloing to fix this:
delete file configure_todo_list.go;
regenerate code.
# swagger generate server -A todo-list -f ./swagger.yml
Then, you can run command go install ./cmd/todo-list-server/, it will succeed.

private repo - go 1.13 - `go mod ..` failed: ping "sum.golang.org/lookup" .. verifying package .. 410 gone

I am using golang 1.13 .
I have a project that depends on a private gitlab project.
I have the ssh keys for the same.
When I try to retrieve the dependencies for a newly created module, I am getting the following error:
$ go version
go version go1.13 linux/amd64
$ go mod why
go: downloading gitlab.com/mycompany/myproject v0.0.145
verifying gitlab.com/mycompany/myproject#v0.0.145: gitlab.com/mycompany/myproject#v0.0.145: reading https://sum.golang.org/lookup/gitlab.com/mycompany/myproject#v0.0.145: 410 Gone
I have no idea why it is trying to ping sum.golang.org/lookup since it is a private gitlab project.
My ~/.gitconfig contains the following (based on my looking up in google search for similar errors)
# Enforce SSH
[url "ssh://git#github.com/"]
insteadOf = https://github.com/
[url "ssh://git#gitlab.com/"]
insteadOf = https://gitlab.com/
[url "ssh://git#bitbucket.org/"]
insteadOf = https://bitbucket.org/
[url "git#gitlab.com:"]
insteadOf = https://gitlab.com/
The error still persists.
I would expect the package to be downloaded from my private gitlab project repository to the current project.
Is there anything I need to do in my private gitlab project repository to make it ready for 'go get' ?
The private gitlab project repository already contains the go.sum and go.mod for the project as well.
Anything that I am missing ?
edit: 1) The private repo name and the company name contains no asterisks or any other special characters. only alphabets and not even numeric characters.
Answering my own question after looking up,
Setting the GOPRIVATE variable seems to help.
GOPRIVATE=gitlab.com/mycompany/* go mod why
"
The new GOPRIVATE environment variable indicates module paths that are not publicly available. It serves as the default value for the lower-level GONOPROXY and GONOSUMDB variables, which provide finer-grained control over which modules are fetched via proxy and verified using the checksum database.
" from https://golang.org/doc/go1.13
Aliter:
Setting the env variable GONOSUMDB also seems to work.
Specifically, invoking the following command seems to help.
GONOSUMDB=gitlab.com/mycompany/* go mod why
The above env variable prevents the ping to sum.golang.org/lookup for a checksum match. It also prevents leaking the names of private repos to a public checksum db. [ Source - https://docs.gomods.io/configuration/sumdb/ ]
Also - here at
* GONOSUMDB=prefix1,prefix2,prefix3 sets a list of module path prefixes, again possibly containing globs, that should not be looked up using the database.
source: https://go.googlesource.com/proposal/+/master/design/25530-sumdb.md
Related Issues:
https://github.com/golang/go/issues/32291
https://github.com/golang/go/issues/33985
["Go 1.13 has been released, and this issue was filed well after the freeze window. The proposed changes will not happen in 1.13, but don't assume they will necessarily happen in 1.14 either." from issue 33985 above. ]
Basically it failed to verify private repository. However I don't like turning off checksum, but you can easily set GOSUMDB to off before trying to get module. something like this:
GOSUMDB=off go get github.com/mycompany/myproject
ref: https://github.com/golang/go/issues/35164#issuecomment-546503518
A second and better solution is to set GOPRIVATE environment variable that controls which modules the go command considers to be private (not available publicly) and should therefore NOT use the proxy or checksum database. The variable is a comma-separated list of glob patterns (same syntax of Go's path.Match) of module path prefixes. For example,
export GOPRIVATE=*.corp.example.com,rsc.io/private
Or
go env -w GOPRIVATE=github.com/mycompany/*
Last solution you can try is to turn off such checks for all private repositories that you don't want to go public or being verified through sum.golang.org/lookup/github.com/mycompany/...
GONOSUMDB=gitlab.com/mycompany/* go mod why
Note that:
If you have issues fetching modules or repos over https, you may want to add the following to your ~/.gitconfig to make go get/fetch repositories using ssh instead of https
[url "ssh://git#github.com/"]
insteadOf = https://github.com/
Change following go variable's setting and then upgrade your package,
$ export GO111MODULE=on
$ export GOPROXY=direct
$ export GOSUMDB=off
$ go get -u <your dependency package>
I have this scenario too and this works for me.
edit your .git/config and add two lines in it.( I have this in a global .gitconfig in home dir)
[url "ssh://youprivate.com"]
insteadOf = https://yourprivate.com
export GOSUMDB=off
Then everything will OK.

go get from a private repo using a non-standard HTTP port

I have a private repository, let's say https://example.com:12345/my_package.git. I want to go get such package using HTTP.
What are the specific steps I must follow to accomplish this?
The first issue I'm facing is that I don't know where I must set the port. In the import section, it complains about invalid import path. In Gopkg.toml, it tries to go to https://example.com instead of https://example.com:12345. Just go get -d -v https://example:12345/my_package.git complains about the : char...
So, how can I do this? Using ether username/pass or token (gitlab), just not SSH.

K8s Go client library fails to find package on go get

We wrote some Go code to talk to our Kubernetes cluster and fetch the IP of a Service exposed. We do it like so:
(import "gopkg.in/kubernetes/kubernetes.v1/pkg/client/restclient")
(import kubectl "gopkg.in/kubernetes/kubernetes.v1/pkg/client/unversioned")
svc, err := c.Services(k8sNS).Get(svcName)
if err != nil {
panic(l.Errorf("Could not retrieve svc details. %s", err.Error()))
}
svcIP := svc.Status.LoadBalancer.Ingress[0].IP
go get works fine, and our script executes when we do go run ... and everybody is happy. Now, as of yesterday (from the time this question is posted) on the same script - go get fails. The error is like so:
[09.07.2016 10:56 AM]$ go get
package k8s.io/kubernetes/pkg/apis/authentication.k8s.io/install: cannot find package "k8s.io/kubernetes/pkg/apis/authentication.k8s.io/install" in any of:
/usr/local/go/src/k8s.io/kubernetes/pkg/apis/authentication.k8s.io/install (from $GOROOT)
/home/ckotha/godir/src/k8s.io/kubernetes/pkg/apis/authentication.k8s.io/install (from $GOPATH)
We have not specifically used authentication package in our code. Are we importing kubernetes libraries correctly? is there another way to do this ?
ls on $GOPATH/k8s.io/kubernetes/pkg/apis/ and found this:
:~/godir/src/k8s.io/kubernetes/pkg/apis
[09.07.2016 10:53 AM]$ ls
abac apps authentication authorization autoscaling batch certificates componentconfig extensions imagepolicy OWNERS policy rbac storage
It looks like a package you imported has changed.
You can update existing repositories:
go get -u
The -u flag instructs get to use the network to update the named
packages and their dependencies. By default, get uses the network to
check out missing packages but does not use it to look for updates to
existing packages.
You do use gopkg.io to pin the version to v1, but I think you want to be more specific, eg, v1.3.6 (EDIT: this won't work because gopkg.in doesn't permit package selectors more specific than the major version.).
Alternatively, a good way to ensure code stays the same is to compile your binary and execute that, instead of using go run.

Resources