I have this problem:
Module A imports module X v0.1.0
Module B imports module X v0.2.0
I import both A and B in my project.
Golang picks out X v0.2.0 and calls it good. But it is not good. v0.1 and v0.2 are different enough that A breaks and my project doesn't compile.
What options do I have to fix this?
The offical go stance seems to be "developers of X were doing it wrong and should've released a major version after breaking changes". But that doesn't help the current situation.
I can't find discussions of practical solutions.
Further information
The above is a simplification, A and B have a couple more dependencies that also depend on X.
This project needs to be maintainable by a group of people. So updating A and B to new versions is ideally straightforwards.
This is security-critical code so subtle bugs from mismatched dependencies are a concern.
Things I've tried:
Forking A and X, changing the import paths throughout, and updating A's go.mod. It works but makes updates to those modules slow and error prone.
Copy and pasting the required code out of A and X to avoid importing it. Also slow and error prone.
Did a deep dive into creative applications of go mod vendor, but couldn't find a solution.
The ideal solution here is to submit an upstream PR to module A to make it compatible with X v0.2.0, and then update your own module to use the fixed version of module A.
If you need to temporarily slot in a fix to module A while you are waiting for the upstream maintainer to respond to the PR, you can do that using a replace directive. (See https://golang.org/doc/modules/managing-dependencies#tmp_11 for more detail.)
You can, for example, create local copies of the both modules and use replace directives along with a pseudo major versions to import multiple versions of the same repo using different tags:
replace moduleX/v1010 => ./src/moduleX/0.1.0
replace moduleX/v1020 => ./src/moduleX/0.2.0
Then in your project would use either of the versions as needed:
import (
moduleX_010 "moduleX/v1010"
moduleX_020 "moduleX/v1020"
)
Related
Go version: 1.12.9
Here is the structure of a simple demo project:
So we have a module domain, which contains two modules: activity and person.
I would like to use domain with all nested modules in the main file => modules.go.
I know how to import domain in the main go.mod:
module modules
go 1.12
replace modules/domain v0.0.0 => ./domain
require modules/domain v0.0.0
So after that can use code from domain/domain.go, but I cannot access code from activity and person modules.
Yes, I could manually import nested modules, for example:
(main go.mod):
module modules
go 1.12
replace modules/domain v0.0.0 => ./domain
replace modules/domain/activity v0.0.0 => ./domain/activity
require (
modules/domain v0.0.0
modules/domain/activity v0.0.0
)
but I don't want to manually import all submodules.
How to configure modules to import domain with all submodules?
Yes, I could manually import nested modules [...]
but I don't want to manually import all submodules.
How to configure modules to import domain with all submodules ?
You simply cannot do this.
Some background:
There is no real notion of "sub"-module, all modules are equal.
Same for packages: all(*) packages are equal. If you want to use a package you must import it.
The layout in the filesystem does not imply any kind of technical relation between packages (e.g. net/http/cookiejar is as much related to net/http as it is related crypto/md5: not at all).
Being part of a module doesn't mean much: Modules are just sets of packages versioned together and doesn't add any additional relation between these packages.
The rule is dead simple: If you want to import a package you have to import it.
The fact that there are no magical wildcard imports might seem annoying (see below) but prevents unintended imports: Adding a package doesn't magically import it (and execute its init functions!).
In real live having to import all packages is not that annoying because "normal" Go code doesn't use tiny packages. This is a common mistake for someone indoctrinated by Java/C#/PHP/Node "project architectures" with lots of folders and tiny classes: Don't do that in Go. It is not helping. It often even leads to circular import and hurts.
Same for modules. A Go module is probably not what you think it is. A module is a set of packages versioned together. I doubt there is a reason to provide different versions for package person and package domain at the same time. (It is allowed to have several modules in one repository/project, but having more than one is a very rare case, yours clearly isn't).
Best advice (in order of relevance):
Stop making every package its own module.
Stop making tiny packages.
Stop trying to mimick a source code layout ("architecture") you might be used from other languages.
(*) internal and vendored packages are an exception which does not apply to your problem.
go 1.18 has the support for the new multi-workspace feature by go.work files.
for more info, check golang workspace proposal
I faced a similar situation with Go 1.20. Here is How I solved it.
Package
I have used a repository as package in my Go project. This repository is uses a sub-module.
Here is the .gitmodules file inside my repository my-repository/services:
Change
What I was missing:
After the introduction of Workspace, submodules were treated differently. So now I need to create a go.work file. Needed to be placed at same level as go.mod.
go.work
go 1.20
use ./validator/validate
Usage
Now I want to use this package in my main repo. Imported it as,
This solved my issue. You can find the documentation at: Setting up your workspace
In my go.mod file, I have:
require (
// ... editted for brevity
github.com/Liquid-Labs/catalyst-firewrap v2.0.0-prototype.3
// ...
)
When I tried to build another package dependent on this package, I get: invalid module: github.com/Liquid-Labs/catalyst-firewrap should be v0 or v1, not v2 (v2.0.0-prototype.3).
There was a rewrite back before go went modular, and I wasn't tagging stuff back then, so there is no v1... is that the problem? I'm not having any luck finding an explanation of the underlying problem here.
Per the Go modules wiki:
If the module is version v2 or higher, the major version of the module
must be included as a /vN at the end of the module paths used in
go.mod files (e.g., module github.com/my/mod/v2, require github.com/my/mod/v2 v2.0.0)
and in the package import path (e.g.,
import "github.com/my/mod/v2/mypkg").
The same document quotes the rationale from the official Go FAQ:
Packages intended for public use should try to maintain backwards
compatibility as they evolve. The Go 1 compatibility guidelines are a
good reference here: don't remove exported names, encourage tagged
composite literals, and so on. If different functionality is required,
add a new name instead of changing an old one. If a complete break is
required, create a new package with a new import path.
and:
If an old package and a new package have the same import path, the new
package must be backwards compatible with the old package.
I'm working on a library with multiple layers of functionality. I want developers to be able to import only the parts they need, ie mylib-core, mylib-feature1, mylib-feature2, etc. Each lives in its own git repo. I'd also like to provide a simple mylib package which exposes a default set of functionality which is useful for developers new to the library. See d3.js version 4+ for something very similar to what I'm trying to accomplish.
The problems I've run into are
Apparently you can't share a package name between packages. This is a problem because it would be nice to import all the desired repos and then simply have everything available under the mylib name.
I don't see an obvious way to re-export functionality, in order to build the default mylib package.
Are there good solutions or a more idomatic go way to accomplish what I'm shooting for?
Answering your question, there is no idiomatic way of doing what you want. It is common in JavaScript to import a library and export its members without interference. This is not the case in Golang.
I advise you to host your whole library under a single repo, and split the functionality to packages. The Go compiler will only compile and use the imported packages.
And a tip for the future, Go is very different than almost any other language you previously know 😅
I want to make some changes to the Go crypto/tls standard library.
Is making a copy of crypto/tls in the vendor folder a good way to do this?
It almost works, it seems the vendored is copy used when I compile the application (Caddy webserver). Apart from one error I get:
go/src/github.com/user/caddy/caddytls/httphandler.go:40: cannot use "vendor/crypto/tls".Config literal (type *"vendor/crypto/tls".Config) as type *"crypto/tls".Config in field value
Is there a way of casting to get around this one error? Doesn't sound like good practice to me though.
I would have thought that the vendored copy would always be used, but it seems something is still using the standard crypto/tls library? (I think "net/http" is. Do I have to vendor this too?)
I actually had to do that. The most practical way is to copy and modify the package (as well as its internal dependencies) - this includes some import paths. And it is not really vendored (vendoring is basically using unmodified packages, otherwise vendoring tools will not work), it is a fork, under a different name. I would guess that caddy does not need the modified version - if it does, you need to fork and modify caddy as well.
It goes without saying, but be extremely careful when modifying crypto/tls package - I for example has to make a minimal change that does not really modify TLS operation (I needed to be able to derive key material from the session master secret and randoms).
Also, you have to fully realize that this has significant cost - when a new version of go is out, one that potentially has updates for the crypto/tls package or its dependencies, you will have to apply your changes once again, manually. Committing a diff between the vanilla version and your version helps. I don't think this is at all practical for non-trivial changes (mine were quite limited - a new public field in Config, few lines of code in handshakes, a new interface).
I need the same functionality. It seems like the crypto/tls package does not allow the reading of custom TLS extensions added from the client side in the ClientHello payload. It would be great to be able to check for any custom extensions and then marshal them out accordingly.
It is a pity that this is not a separate package as we could then in our go.mod file be able to use replace to specify a custom TLS package.
e.g
replace golang.org/x/crypto/tls => ./tls
Then running go mod vendor.
Where the ./tls is my local version with the changes applied.
In Go, if you reference another package, e.g. something on GitHub, then Go always gets the latest version from the master branch. While this is great for development, I guess it's a problem in production: This way a build is not reproducible.
So, what is the correct way in Go to fix a version of a dependency, and how to handle this efficiently?
A friend pointed me to godep, and this seems fine, but I wanted to know what alternatives are there, and what's good / bad about godep?
Update 2018 with Go 1.11
Dependencies should now be referenced with modules (derived from the vgo project):
Go 1.11 adds preliminary support for a new concept called “modules,” an alternative to GOPATH with integrated support for versioning and package distribution.
Using modules, developers are no longer confined to working inside GOPATH, version dependency information is explicit yet lightweight, and builds are more reliable and reproducible.
See Defining a module. (and the original design proposal)
Update June 2015: first support for vendoring is making its way in Go 1.5!
See c/10923/:
When GO15VENDOREXPERIMENT=1 is in the environment, this CL changes the resolution of import paths according to the Go 1.5 vendor proposal:
If there is a source directory d/vendor, then, when compiling a source file within the subtree rooted at d, import "p" is interpreted as import "d/vendor/p" if that exists.
When there are multiple possible resolutions, the most specific (longest) path wins.
The short form must always be used: no import path can contain “/vendor/” explicitly.
Import comments are ignored in vendored packages.
Update March 2015: the go team is thinking about defining a go dependency management system integrated to the language: the debate is in this thread.
We think it’s time to start addressing the dependency & vendoring issue, especially before too many conflicting tools arise and fragment best practices in the Go ecosystem, unnecessarily complicating tooling. It would be nice if the community could converge on a standard way to vendor.
Our proposal is that the Go project,
officially recommends vendoring into an “internal” directory with import rewriting (not GOPATH modifications) as the canonical way to pin dependencies.
defines a common config file format for dependencies & vendoring
makes no code changes to cmd/go in Go 1.5. External tools such as “godep” or “nut” will implement 1) and 2). We can reevaluate including such a tool in Go 1.6+.
One possible downside of godep is that you can no longer use "go build" or "go test" directly.
You need to precede those commands with godep (or type godep save).
An alternative is glide, which remains compatible with classic go commands.
Manage project-specific GOPATHs
Ease dependency management
Support versioning packages
Support aliasing packages (e.g. for working with github forks)
Remove the need for "vendoring" or munging import statements
Work with all of the go tools
More generally, the article "Know your guarantees, Go edition" is interesting:
It’s also a deliberate choice, where the Go authors chose not to implement a feature when they felt that the trade-offs were no good.
One low-level reason they made this choice is to avoid slow compilation and bloated binaries (which are two sides of the same coin).
Remember, packages depend on other packages. So Foo might depend on Bar 2.1. Foo might also depend on Baz which in turn depends on Bar 1.9, and on down the tree. So that would mean compiling and linking several copies of nearly identical code.
Depending on several versions of the same package also means knowing which version one is calling, whereby the dependency mess seeps into your source code.
Which leads us to the high-level reasoning behind the Go platform punting on this feature: they did not have a logical solution they considered acceptable. It’s not that they don’t understand the problem; it’s that, at the moment, there is not a solution they like. So they choose no feature over over a regressive one.
You handle dependencies like you do handle dependencies in other languages too: You vendor. For Go there is no Nexus which does the vendoring so most just copy external libraries into a "vendor" folder, there are tools helping here. Personally I found all this "fix version" panic a bit exaggerated as it works pretty well without.
You might wanna take a look at http://labix.org/gopkg.in and search golang-nuts for dependency management. I think there is even a whole mailing list devoted to this.