Go mod replace and github forks of /vN repositories - go

I've just sent a pull request to http://periph.io/x/devices/v3; it's already been merged, but something about the process was suboptimal:
First I cloned the repository (which is actually http://github.com/periph/devices; the module is named periph.io/x/devices/v3 though), so my fork is automatically called http://github.com/lutzky/devices. I wanted to test a separate piece of code I'm working on with the modified library, call it testclient; adding this to testclient/go.mod works:
replace periph.io/x/devices/v3 v3.6.9 => ../devices
However, this should also work (and would be useful if the pull request took longer to be accepted), and doesn't:
replace periph.io/x/devices/v3 v3.6.9 => github.com/lutzky/devices main
That gives this error:
testclient/go.mod:15: replace github.com/lutzky/devices:
version "v0.0.0-20210508194004-cae0146d8900" invalid:
go.mod has post-v0 module path "periph.io/x/devices/v3" at revision cae0146d8900
On a hunch, I figured I'd create a tag v3.6.9-newfeature and push that to my fork, and point the replace command at that. That doesn't work either:
testclient/go.mod:15: replace github.com/lutzky/devices:
version "v3.6.9-newfeature" invalid:
module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v3
So I can't use a v0 tag because go.mod says it's v3, but I can't use a v3 tag because the URL doesn't have /v3. I don't think I should modify go.mod in my forked repository (for one thing, that'd make the pull request silly). What's the intended way to go about this?

Here's the fix that you are looking for, I tested the following go.mod file and it works,
module test
go 1.14
replace periph.io/x/devices/v3 => github.com/lutzky/devices/v3 v3.6.9-newfeature
require (
periph.io/x/devices/v3 v3.6.9 // indirect
)
I first installed periph.io/x/devices/v3,
go get periph.io/x/devices/v3
Then I inserted the replace in go.mod,
replace periph.io/x/devices/v3 => github.com/lutzky/devices/v3 main
Format: replace <original_module> => <forked_repo> <branch>
Then I ran just go get. After that, the main in the replace instruction got swapped with v3.6.9-newfeature which is the latest tag on the forked repo's main branch, thus giving you the content that you see above.

Thanks to ramaraja-ramanujan#'s answer and a bit more debugging, I found the answer.
TL;DR - the correct answer is:
replace periph.io/x/devices/v3 v3.6.9 => github.com/lutzky/devices/v3 main
// this part was missing ^^^
This will work immediately when building, and go mod tidy (or go get) will change main to a specific tag. There are a couple of interesting side-notes to this:
The go proxy
The v3.6.9-newfeature tag was my own; I had created it as part of trying to get this to work. I've deleted it, and yet go mod tidy kept replacing main with v3.6.9-newfeature - no matter how many caches I cleared! Running strace it turned out that go was contacting http://proxy.golang.org. To disable this, I ran GOPROXY=direct go get. This replaced the line with the pseudo version v3.6.10-0.20210508194004-cae0146d8900. Note that this matches the current main commit in my fork, and neither my fork nor the original repo have v3.6.10 - it appears to have started with the existing v3.6.9 and incremented it.
The /v3 is not a subdirectory
Indeed, the source repo (https://github.com/periph/devices) does not have a v3 subdirectory. I thought what was going on is that the periph.io domain is doing some trickery - indeed going to http://periph.io/x/devices/v3 with a browser does work. However, it would appear that Go's resolution takes the final /v3 in the module path not to indicate, in this case, a subdirectory (but indeed a restriction on tag names). This is apparently allowed, and is discussed in https://golang.org/ref/mod, but it isn't entirely straightforward.

Related

Make dolthub/dolthub-etl-jobs/tree/master/loaders/nvd use newer CVE scheme

I am trying to get some code to run which is here, on GitHub:
https://github.com/dolthub/dolthub-etl-jobs/tree/master/loaders/nvd
Once I've cloned the repo I run the run.sh script and it fails with the below:
./run.sh
1 synchronisation error:
unexpected http response from "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2002.meta" ("404 Not Found"): ""
cloning https://doltremoteapi.dolthub.com/Liquidata/NVD
For this to have a chance of working I need to change wherever this is referenced:
https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-20XX.meta
to:
https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-20XX.meta
The old reference is no longer valid.
However, it's impossible to see where the script is actually calling that URL from. Probably I do not know near enough about how go and GitHub hang together.
If I could figure it out, maybe I could just pull down the code and edit it manually once it was on my host or even create a fork with the new URL in it.
Find where the URL is actually coming from when I call run.sh which errors out almost immediately.
Make a change that reflects the valid one.
TLDR: Replace 4 by 6 in line 44 of main.go.
go.mod from https://github.com/dolthub/dolthub-etl-jobs/tree/master/loaders/nvd
requires github.com/facebookincubator/nvdtools but replaces this by github.com/liquidata-inc/nvdtools which redirects to github.com/dolthub/nvdtools which is an archived repo(!) (cf. https://github.com/dolthub/dolthub-etl-jobs/blob/d858a2433f68d72dc643e26085a5a0c44edbb85c/loaders/nvd/go.mod#L5-L7).
Supported CVE feeds of dolthub/nvdtools are defined here: https://github.com/dolthub/nvdtools/blob/e67111c0fff487cc15cd2ba32668141622cf9c63/providers/nvd/cve.go#L44-L53
cve10jsonGz is 4, cve11jsonGz is 6
main.go sets the CVE feed here: https://github.com/dolthub/dolthub-etl-jobs/blob/d858a2433f68d72dc643e26085a5a0c44edbb85c/loaders/nvd/main.go#L43.
Change it from 4 to 6.
I didn't test the rest of run.sh, but at least the problem you're mentioning in your question should be solved.

Replacing inconvenient package name. Got error: replacement module without version must be directory path (rooted or starting with ./ or ../)

Problem
In go.mod file I wrote:
module github.com/Siiir/vector
go 1.17
require github.com/huandu/go-clone v1.3.2 // indirect
replace clone => github.com/huandu/go-clone[v1.3.2]
It says that I cannot do such a replacement.
I actually solved my problem with the name of imported package.
It is convenient & working without that dash. I found that I can use clone.something to refer to a function.
No need to type go-clone.something.
Anyway, assume that a package name is indeed crazy or inconvenient. How can I replace it?
What I've seen:
I've seen a sibling question:
go modules - replace does not work - replacement module without version must be directory path (rooted or starting with
What I tried:
Working with terminal:
go mod edit -replace=clone=github.com/huandu/go-clone
got: go: -replace=clone=github: unversioned new path must be local directory
manual editing:
Attempts like: replace clone => github.com/huandu/go-clone[v1.3.2]
got: replacement module without version must be directory path (rooted or starting with ./ or ../)
Anyway, assume that a package name is indeed crazy or inconvenient. How can I replace it?
You cannot.
And you should not. The import path is something you write just once in the import declaration and the package name can be changed on a per file level with import nicename "something.you.think/is-totally/inconvenient/and/unacceptable-to/your_taste" .

go-swagger restapi/configure_todo_list.go - api.TodoGetHandler undefined error

I am a newbie in go and go-swagger. I am following steps in Simple Server tutorial in goswagger.io.
I am using Ubuntu 18.04, swagger v0.25.0 and go 1.15.6.
Following the same steps, there are a few differences of the files generated. For instance, goswagger.io's has find_todos_okbody.go and get_okbody.go in models but mine does not. Why is that so?
Link to screenshot of my generated files vs
Link to screenshot of generated files by swagger.io
Starting the server as written in the tutorial go install ./cmd/todo-list-server/ gives me the following error. Can anyone please help with this?
# my_folder/swagger-todo-list/restapi
restapi/configure_todo_list.go:41:8: api.TodosGetHandler undefined (type *operations.TodoListAPI has no field or method TodosGetHandler)
restapi/configure_todo_list.go:42:6: api.TodosGetHandler undefined (type *operations.TodoListAPI has no field or method TodosGetHandler)
The first step in goswagger.io todo-list is swagger init spec .... Which directory should I run this command in? I ran it in a newly created folder in my home directory. However, from the page, it shows the path to be ~/go/src/github.com/go-swagger/go-swagger/examples/tutorials/todo-list. I am not sure whether I should use go get ..., git clone ... or create those folders. Can someone advise me?
Thanks.
This is likely the documentation lagging behind the version of the code that you are running. As long as it compiles, the specific files the tool generates isn't so crucial.
This is a compilation error. When you do go install foo it will try to build the foo package as an executable and then move that to your GOPATH/bin directory. It seems that the generated code in restapi/configure_todo_list.go isn't correct for the operations code generated.
All you need to run this tutorial yourself is an empty directory and the swagger tool (not its source code). You run the commands from the root of this empty project. In order not to run into GOPATH problems I would initialise a module with go mod init todo-list-example before doing anything else.
Note that while the todo-list example code exists inside the go-swagger source, it's there just for documenting example usage and output.
What I would advice for #2 is to make sure you're using a properly released version of go-swagger, rather than installing from the latest commit (which happens when you just do a go get), as I have found that to be occasionally unstable.
Next, re-generate the entire server, but make sure you also regenerate restapi/configure_todo_list.go by passing --regenerate-configureapi to your swagger generate call. This file isn't always refreshed because you're meant to modify it to configure your app, and if you changed versions of the tool it may be different and incompatible.
If after that you still get the compilation error, it may be worth submitting a bug report at https://github.com/go-swagger/go-swagger/issues.
Thanks #EzequielMuns. The errors in #2 went away after I ran go get - u -f ./... as stated in
...
For this generation to compile you need to have some packages in your GOPATH:
* github.com/go-openapi/runtime
* github.com/jessevdk/go-flags
You can get these now with: go get -u -f ./...
I think it's an error of swagger code generation. You can do as folloing to fix this:
delete file configure_todo_list.go;
regenerate code.
# swagger generate server -A todo-list -f ./swagger.yml
Then, you can run command go install ./cmd/todo-list-server/, it will succeed.

I want to remove my private package from Goproxy

It's a go module question.
My PC is running with go 1.13 and go module mode is on.
This my go env:
GO111MODULE="on"
GOPRIVATE=""
GOPROXY="https://goproxy.io,direct"
At first I wrote a public package for testing and uploaded it to github. And then I "go get" this package for independence. After that I made it private and even deleted it from github (I tried "git clone" and couldn't download it again). But I could "go get" it after I removed it from github. Soon I realized it that maybe this package was cached in goproxy.io or other proxy databases. So I set GOPROXY="direct" and "go get" this package again and failed. This action proved my guess.
Now it's my question:
How to remove this whole package from goproxy.io or other database.
Is there a safe way to use go module, I don't want to upload my private code to other databases by mistake.
I tried to STFW and found nothing. Thank U to all people for reading and answering this question.
If you accidentally published a package / module you intend to be private, then go.dev: About page:
Removing a package
If you would like a package removed, please send an email to go-discovery-feedback#google.com, with the import path or module path that you want to remove.
But as Adrian mentioned in the comments, there is no guarantee that no one downloaded your published packages.
If you want some packages / modules to remain private, you may enumerate them (using glob patterns) in the GOPRIVATE, GONOPROXY, GONOSUMDB environment variables, which are respected by the go tool.
Command go: Environment variables:
GOPRIVATE, GONOPROXY, GONOSUMDB
Comma-separated list of glob patterns (in the syntax of Go's path.Match)
of module path prefixes that should always be fetched directly
or that should not be compared against the checksum database.
See 'go help module-private'.
From command go help module-private:
The GOPRIVATE environment variable controls which modules the go command
considers to be private (not available publicly) and should therefore not use the
proxy or checksum database. The variable is a comma-separated list of
glob patterns (in the syntax of Go's path.Match) of module path prefixes.
These vars can also be set using the go env -w command. You may get more help about it using go help env.
For Question2, I have tried several times today.
If u want to build a private package safely, the most important thing you should do is setting the GOPRIVATE BEFORE u build your package. And GONOPROXY,GONOSUMDB will be automatically set to same as GOPRIVATE.
Now u can write the code and git push to a private remote repo and try to go get it, and u will fail and see some error message like it:
fatal: could not read Username for 'https://github.com': terminal prompts disabled
Confirm the import path was entered correctly.
If this is a private repository, see https://golang.org/doc/faq#git_https for additional information.
The link show us that we should use https link with your username/password or just replace your https link with ssh link if u upload a key to remote ropo. After doing it, u will succeed to go get it from your source address (your package name) but NOT PROXY.
Futhurmore, let's delete all go module cache by using go clean -modcache and set GOPRIVATE="". I will test whether our private package was uploaded to proxy.
case 1:
Set GOPROXY="https://goproxy.io" or other proxy and thenen go get the private package.
u will face a 404 Not Found which show u that u find NOTHING from proxy (it meant nothing was cached/stored in proxy) .
case 2:
Set GOPROXY="direct"(use the direct address of your pacakage name) and then go get the private package.
u will face error message like it:
verifying {PACKAGENAME}: {PACKAGENAME}: initializing sumweb.Conn: checking tree#{ID}: Get https://sum.golang.org/tile/8/2/000.p/11: dial tcp {IP}:{PORT}: i/o timeout.
The go get function will try to check the checksum from golang.org database with your package (I can't link to google server directly so it shows timeout error / I guess u will get 404 if u can link to google server). It shows that if we try to go get a private package with not setting GOPRIVATE, u will fail because u can not pass the check.
Suggestion:
If you want to build your private go package, u should set your GOPRIVATE firstly and make sure it is wide enough to INCLUDE your package name.
Thank U to all people for reading and answering this question.

Go install exclude file

I have created a go script that compiles, starts, checks the status, and ends a web service I created (that is also in go). However, I have come to a road block.
With the compile feature I run the following command:
go install .
Which gives the following error:
./script.go:55: main redeclared in this block
previous declaration at ./hello.go:8
Which makes sense as I have two different files, both with the main func and main package. I also tried moving the script to another folder and then changing the command ran to:
go install {path}
Where {path} is equal to the path I want installed/compiled. Which I then got the following error:
exit status 1: can't load package: package /var/www/test.com/go: import "/var/www/test.com/go": cannot import absolute path
So in conclusion I have thought of only one solution (and I am up to hear others if mine isn't the best approach). My idea is to exclude the script file from compiling with the rest of the files, but I am unsure how to.
I did some research and couldn't find an easy way to do it (such as an --exclude flag with the go install command). Does anybody know how to accomplish what I am trying to achieve?
Thank you.
you could give the hello.go a different package name, that should work. Or i am missing something?
Regards
Tim

Resources