I'm trying to understand how to work with Golang and forks. The situation is the following, I'm writing a library project which depends on library github.com/other/some_dependency, which isn't mine.
Because some_dependency is missing some methods that I need, I fork it to github.com/me/some_dependency. However, I can't just do go get github.com/me/some_dependency, the library references itself so it breaks.
In this article they give a possible solution:
go get github.com/other/some_dependency
cd $GOPATH/src/github.com/other/some_dependency
git remote add fork git#github.com:me/some_dependency
git rebase fork/master
Now, this is hacky at best. There is no way from the code of the library to know that the dependency is coming from a different repo. Anyone doing go get of my library wouldn't manage to make it work.
As dep is expected to be the official dependency manager. I've found how to fix the version:
dep ensure -add github.com/foo/bar#v1.0.0
But I cannot find how to set a different remote. Is is possible to do it?
As an example, in Node.js with npm it is dead simple:
npm install git+https://git#github.com/visionmedia/express.git
If you look at the help you will see this:
<import path>[:alt source URL][#<constraint>]
So to add github.com/foo/bar from location github.com/fork/bar you have to add it like this:
dep ensure -add github.com/foo/bar:github.com/fork/bar
The source location will be added as source attribute in the Gopkg.toml.
Gopkg docs for dependency rules constraint and override
Related
This question poses problems that are distinct from those discussed here: How do you read debug VCS version info from a Go 1.18 binary?
I am having trouble getting go build -buildvcs=true to insert correct version information when building executables out of a git repo that has the following structure:
go version go1.18.4 linux/amd64
in project:
go.mod
cmd/exe1/main.go
cmd/exe2/main.go
pkg/pkg1/...
pkg/pkg2/...
(1) If I cd project;go build -buildvcs=true -o /tmp/exe1 cmd/exe1/main.go then the BuildInfo included in the exe includes Deps entries for all the dependencies of all the packages, but there is no embedded Setting with key vcs.revision, and the Dep entry for the module named in go.mod is (devel). I guess this latter issue is related to how to specify versions for modules, which I have not yet looked into, and therefore I assume it's using a default value.
(2) If I cd project/cmd/exe1;go build -buildvcs=true -o /tmp/exe1 (leaving out any relative path specifying what to build) then the BuildInfo included in the exe does NOT include Deps entries but DOES include the vcs.revision
Questions:
Is there any way to get both Deps and vcs.revision into BuildInfo?
Is this directory structure ok? The documentation for this stuff is not in "reference" format, and many important details are spread out throughout all of the tutorials and howtos. Quite frustrating to get to the bottom of these behaviors.
It seems to me go should embed a vcs.revision whenever generating an executable, but before I open a bug, I wanted to get community feedback on whether this is expected behavior when specifying a relative target on the command line. I've seen that that can be an issue in general, with go build.
Any pointers to the right place to read a comprehensive guide about this would be great.
I am getting this error from go mod tidy when I am trying to update my dependancies. I am primarily developing a webhook service to use with cert-manager and I cannot figure out how to resolve this dependency isssue since the packages I am dependent on are created by other developers and I cannot control those "sub dependency".
This is my output:
go.opentelemetry.io/otel/semconv: module go.opentelemetry.io/otel#latest found (v1.9.0), but does not contain package go.opentelemetry.io/otel/semconv
I looked the package up here: https://pkg.go.dev/go.opentelemetry.io/otel/semconv
and the problem for me seems like that the package have been restructured like this:
go.opentelemetry.io/otel/semconv/v1.9.0
as a subdirectory instead of package version.
Is there a way I can manipulate the underlying dependancy of the packages that my service is depending on?
Please leave a comment if you need addictional information.
Take a look at the accepted solution's comments
You may want to use a local copy of the module where you can fix the issue and use it. Steps for that
Clone the module repository git clone https://github.com/open-telemetry/opentelemetry-go.git
If needed, checkout to a branch/tag git checkout branch_name
In the go.mod file of your module, add the following lines
replace (
go.opentelemetry.io => /path/where/cloned/opentelemetry-go
)
Now you should be able to modify code in the cloned opentelemetry-go repo and use it in your module
I have a project where I use go modules. There I need to specify that I depend on a particular fork of a library (call it ), because it has an important patch. When I run go get -u <my_project>, I get a compilation error that clearly means that go took the main repo of , instead of the fork.
After that, I switch to the directory where go downloaded and run go build. Then, go takes the proper version of and compilation is successful.
Could you tell me what could be the reason for that and how to fix it?
Here is specific command to get :
go get -u github.com/planetA/konk
The dependency is "github.com/opencontainers/runc". For this dependency, go.mod contains following:
replace (
github.com/opencontainers/runc => github.com/planeta/runc v1.0.0-rc9.0.20191206160324-51eabe724369
)
require (
github.com/opencontainers/runc v1.0.0-rc9.0.20191206223258-201b06374548
)
Interestingly, in following sequence of commands, second go get does not produce error
go get -u github.com/planetA/konk
cd ~/go/src/github.com/planetA/konk
go build -tags seccomp
go get -u github.com/planetA/konk
I had a similar problem and I resolved this by detaching the fork itself and making a separate repo. Depends on your use case, this is also a much cleaner solution as you don't need to add too many edits to make it work. You can do so by:
Clone the fork repository on your local.
Delete the fork repository from github/gitlab etc. if you want the package name to be the same as the fork if not you can rename the new repo you create.
Create a new repo and push your content onto it
Now go get github.com/myusername/mynewrepo
But if you still want to make it work using the fork here is a great reference.
We are converting our internal codebase from the dep dependency manager to go modules (vgo or built in with go1.11.2). Imagine we have code like this:
$GOPATH/src/mycompany/myprogram/main.go:
package main
import (
"fmt"
lib "mycompany/mylib" )
func main() {
fmt.Println("2+3=", lib.add(2, 3))
}
$GOPATH/src/mycompany/myprogram/go.mod:
module mycompany/myprogram
(it doesn't have any dependencies; our real-world code does).
$GOPATH/src/mycompany/mylib/lib.go:
package mylib
func Add(x int, y int) int {
return x + y
}
I didn't module-ize this code; it doesn't seem to matter whether I do or don't.
These are trivial examples but our internal code follows a similar structure as this worked historically.
Since these directories are on the Gopath, export GO111MODULE=auto still builds as before and this works fine (modules not used because we are on the gopath). However, when I set export GO111MODULE=on I immediately get the error:
build mycompany/myprogram: cannot find module for path mycompany/mylib
So I did some research and I would like to validate my understanding. First let me say our old approach worked, but I am more interested in changing to use go modules as it appears to be where the go project itself is headed. So.
It seems the intention of the golang authors was that "dotless" paths belong to the standard repository only; that is there should be a binding between domain name and project. We don't use go get on our internal project, unsurprisingly. Here is the source specifically:
Dotless paths in general are reserved for the standard library; go get has (to my knowledge) never worked with them, but go get is also the main entry point for working with versioned modules.
Can anyone with more knowledge of golang than me confirm this?
My key assumption is that once go decides to use modules, all dependencies must be modules and the gopath becomes somewhat irrelevant, except as a cache (for downloaded modules). Is this correct?
If this is true, we need to use a private gitlab (in our case) repository on the path. There's an open issue on handling this that I'm aware of so we can implement this if necessary. I'm more interested in the consequences, specifically for iterating in the private repositories. Previously we could develop these libraries locally before committing any changes; now it seems we have a choice:
Accept this remote dependency and iterate. I was hoping to avoid needing to push and pull remotely like this. There are workarounds to needing an internet connection if strictly necessary.
Merge everything into one big git repository.
If it matters, I'm using go version go1.11.2 linux/amd64 and my colleagues are using darwin/amd64. If it helps, my golang is exactly as installed by Fedora's repositories.
So, tl;dr, my question is: are go modules all-or-nothing, in that any dependency must be resolved using the module system (go get, it seems) and the gopath has become redundant? Or is there something about my setup that might trigger this to fail? Is there some way to indicate a dependency should be resolved explicitly from the gopath?
Updates since asking the question:
I can move myprogram out of the gopath. The same issue occurs (mylib has been left in the gopath).
I can run, or not run, go mod init mycompany/mylib in the mylib directory; it makes no difference at all.
I came across Russ Cox's blog post on vgo. My concerns about offline development that I tried not to dive into too far are resolved by $GOPROXY.
I use a workaround with GITHUB_TOKEN to solve this.
Generate GITHUB_TOKEN here https://github.com/settings/tokens
export GITHUB_TOKEN=xxx
git config --global url."https://${GITHUB_TOKEN}:x-oauth-basic#github.com/mycompany".insteadOf "https://github.com/mycompany"
I wrote up a solution for this on Medium: Go Modules with Private Git Repositories.
The way we handle it is basically the same as the answer above from Alex Pliutau, and the blog goes into some more detail with examples for how to set up your git config with tokens from GitHub/GitLab/BitBucket.
The relevant bit for GitLab:
git config --global \
url."https://oauth2:${personal_access_token}#privategitlab.com".insteadOf \
"https://privategitlab.com"
#or
git config --global \
url."https://${user}:${personal_access_token}#privategitlab.com".insteadOf \
"https://privategitlab.com"
I hope it's helpful.
I use ls-remote git command to help resolve private repo tags and go get after it.
$ go env GO111MODULE=on
$ go env GOPRIVATE=yourprivaterepo.com
$ git ls-remote -q https://yourprivaterepo.com/yourproject.git
$ go get
My Problem
Elastic Beats is an open source project for log shippers written in Go. It features several log outputs, including console, Elasticsearch and Redis. I would like to add an output of my own - to AWS Kinesis.
I have cloned the repo to ~/github/beats, and tried building it:
$ cd filebeat; go build main.go
However, it failed due to a missing library which is a part of the project:
main.go:6:2: cannot find package "github.com/elastic/beats/filebeat/cmd" in any of:
/usr/local/go/src/github.com/elastic/beats/filebeat/cmd (from $GOROOT)
/Users/adam/go/src/github.com/elastic/beats/filebeat/cmd (from $GOPATH)
A directory of the project is dependent on a package from the same repo, but instead of looking one directory up the hierarchy it looks in the GOPATH.
So, go get github.com/elastic/beats/filebeat/cmd fetched the code, and now go build main.go works. Changing the code in my GOPATH is reflected in these builds.
This leaves me with an structural inconvenience. Some of my code is at a working directory, and some of it is at my GOPATH and included by the code in my working directory.
I would like to have all my code in a single directory for various reasons, not the least being keeping everything under version control.
What Have I Tried
Mostly searching for the problem. I am quite new to Go, so I might have missed the correct terminology.
My Question
What is the right way to edit the code of an imported library in Go?
One of the recommended ways to work with other's packages is:
Get the sources of the original package:
go get github.com/elastic/beats
As a result you will clone project's git repository to the folder
$GOPATH/src/github.com/elastic/beats
Make some fixes, compile code, fix, compile... When you make go install package will be compiled and installed to your system. When you need merge updates from original repository you can git pull them.
Everything is OK. What's next? How to share your work with others?
Fork project on github, suppose it will be github.com/username/beats
Add this fork as another remote mycopy (or any other name you like) to your local repository
git remote add mycopy git://github.com/username/beats.git
When all is done you can push updated sources to your repo on github
git push mycopy
and then open a pull-request to original sources. This way you can share your work with others. And keep your changes in sync with mainstream.
Previous answers to this question are obsolete when developing projects that using Go Modules.
For projects that using Go Modules, one may use the following command to replace an imported library(eg. example.com/imported/module) with a local module(eg. ./local/module):
go mod edit -replace=example.com/imported/module=./local/module
Or by adding the following line into the go.mod file:
replace example.com/imported/module => ./local/module
Reference Docs: https://golang.org/doc/modules/managing-dependencies#unpublished
A project working copy should be checked out into $GOPATH/src/package/import/path - for example, this project should be checked out into /Users/adam/go/src/github.com/elastic/beats. With the project in the correct location, the go tooling will be able to operate on it normally; otherwise, it will not be able to resolve imports correctly. See go help gopath for more info.