I'm using Go 1.11 modules for multiple repositories at this point. Right now I'm dealing with one that was already at a 1.x version. In combination with moving to go modules, I made some other breaking changes, so it's clear it's time to increment the major version of the repository. This means going to version 2.
Per "go command" documentation:
To preserve import compatibility, the go command requires that modules with major version v2 or later use a module path with that major version as the final element. For example, version v2.0.0 of example.com/m must instead use module path example.com/m/v2, and packages in that module would use that path as their import path prefix, as in example.com/m/v2/sub/pkg. Including the major version number in the module path and import paths in this way is called "semantic import versioning".
Source
Is this as simple as updating the first line of my go.mod file, appending /v2 to the module name? Or should I be creating a v2/ directory in my repository and moving all files into there?
There are two options for how to move to version 2: branches and subdirectories. You can read more about them (with better illustrations) at https://research.swtch.com/vgo-module.
These two options are what make it possible for one version to depend on another. For example, when you implement version 2, you can update version 1 to depend on version 2 (but keep the same v1 API). Then, you only need to have one implementation of the logic for your library. This may or may not work well for you depending on the type of project, the support you want to provide, and the fixes it requires.
Branches
master: A -> B (v1.0.0) -> D (v1.0.1)
\
v2: -> C (v2.0.0)
In this scenario:
You start your project on the master branch,
Make some commits (A and B),
Tag v1.0.0.
You decide to make a breaking change. So, you create a new branch (git checkout -b v2) and make your breaking changes. Your go.mod must now be updated so the module name ends with /v2 (it's essentially a new module!).
You decide to fix a bug in v1, so you go back to that branch, make a new commit, and tag a new v1 version.
When a user requires a particular version of your module, go will look in the two branches for which one provides the right module.
Subdirectories
What if you don't want to develop on branches? You can create a subdirectory for each major version. Version 1 stays at the top level, then new versions move into subdirectories:
go.mod (module example.com/foo)
foo.go
v2/
go.mod (module example.com/foo/v2)
foo.go
When you tag this repo with new versions, v1 will use the top level version. v2 tags will use the v2 subdirectory.
From the Releasing Modules (v2 or Higher) section of the modules wiki:
There are two alternative mechanisms to release a v2 or higher module.
Note that with both techniques, the new module release becomes
available to consumers when the module author pushes the new tags.
Using the example of creating a v3.0.0 release, the two options are:
Major branch: Update the go.mod file to include a /v3 at the end of the module path in the module directive (e.g., module github.com/my/module/v3). Update import statements within the module
to also use /v3 (e.g., import "github.com/my/module/v3/mypkg").
Tag the release with v3.0.0.
Go versions 1.9.7+, 1.10.3+, and
1.11+ are able to properly consume and build a v2+ module created using this approach without requiring updates to consumer code that has not
yet opted in to modules (as described in the the "Semantic Import
Versioning"
section above).
Note: creating a new branch is not required. If instead you have been previously releasing on master, for example, and would prefer to tag v3.0.0 on master, that is a viable option as well. [...]
[...]
Major subdirectory: Create a new v3 subdirectory (e.g., my/module/v3) and place a new go.mod file in that subdirectory.
The module path must end with /v3. Copy or move the code into the
v3 subdirectory. Update import statements within the module to also
use /v3 (e.g., import "github.com/my/module/v3/mypkg"). Tag the
release with v3.0.0.
However, there are additional details in that section that are worth reviewing.
One point covered there that is worth mentioned here is that if you are interested in an automated approach (for example, perhaps you have many files you would need to visit), a good automated solution is https://github.com/marwan-at-work/mod, which can automatically add, remove, or change the required /vN in your *.go code and your go.mod. See this answer for more details.
Related
I'm new in Golang, and I'm trying to develop a go-module and share it with my colleagues while I'm developing it; In JVM/sbt I used to publish my work with a 'SNAPSHOT' postfixed to version value. but How I can achieve the same in with go-modules?
Versions for modules are tagged by using repo tags (e.g. git tag), following semantic versioning (https://semver.org/).
So, any version starting with v0 is treated as unstable and may make breaking changes at any time. Once you release a v1, you cannot make any breaking changes without bumping your major version, which also means you change your module name.
You also have the option of appending +foo to the end of your version to indicate additional information about the version.
I wrote https://blog.golang.org/using-go-modules as an overview of how to get started using modules.
I have a project with mutliple main methods.
When running go build program1/main1.go which has a different set of dependencies than program2/main2.go, my first go build seems to alter my go.mod file and removes the dependencies it thinks it does not need. Yet main2 would need these dependencies.
I've tried using go build ... but that also created a different set of dependencies. Specifically, it seems like all the //indirect dependencies get removed and cause program2 to fail.
Is there a way of running go build or go run without updating the go.mod file? Using go build -mod=readonly program1/main1.go it tells me that it fails because the dependencies need to be updated..
I believe you're looking for submodules. See this walktrhough.
TLDR: you'll want a separate go.mod in each of your tools's cmd dir, and you can use the replace directive to point dependncies from these tools to your local module.
This Go Issue and others linked from it suggest that figuring out "the one right way" to do this is still WIP, though I think your use case is simple enough.
Using submodules is a way to nest multiple Go module projects you can edit.
But Go 1.18 might include the notion of Go workspace, which means you don't need submodules anymore: one Go project can include multiple modules you can edit.
See golang/go issue 45713: "proposal: cmd/go: add a workspace mode " and its design document.
Background
Users often want to make changes across multiple modules: for instance, to introduce a new interface in a package in one module along with a usage of that interface in another module.
Normally, the go command recognizes a single "main" module the user can edit.
Other modules are read-only and are loaded from the module cache.
The go mod replace directive is the exception: it allows users to replace the resolved version of a module with a working version on disk.
But working with the replace directive can often be awkward: each module developer might have working versions at different location on disk, so having the directive in a file that needs to be distributed with the module isn't a good fit for all use cases.
Proposal
This proposal describes a new workspace mode in the go command for editing multiple modules.
The presence of a go.work file in the working directory or a containing directory will put the go command into workspace mode.
The go.work file specifies a set of local modules that comprise a workspace. When invoked in workspace mode, the go command will always select these modules and a consistent set of dependencies.
Main modules: The module the user is working in.
Before this proposal, this is the single module containing the directory where the go command is invoked. This module is used as the starting point when running MVS.
This proposal proposes allowing multiple main modules.
See for instance CL 334934 (CL = Change List)
[dev.cmdgo] cmd/go: add the workspace mode
This change adds the outline of the implementation of the workspace mode.
The go command will now locate go.work files, and read them to determine
which modules are in the workspace.
It will then put those modules in the root of the workspace when building the build list.
It supports building, running, testing, and listing in workspaces.
You can initiate a multiple-module project with go mod initwork
Again, this is not before Go 1.18 (Q1 2022) and will probably be opt-in in Go 1.19 (Q3 2022).
In Golang, we can specify open source libraries on GitHub as dependencies. For example:
import "github.com/RichardKnop/somelibrary"
This will try to look for a branch based on your Go version and default to master if I understand correctly.
So there is no way to import a specific release of a dependency, e.g.:
import "github.com/RichardKnop/somelibrary#v1.4.8"
What is the best practise to manage dependencies in Go then?
I can see two approaches.
I. Version Modules
Is it to create new modules for major versions with breaking changes?
For example, my Go library could define modules v1 and v2 so then you could do:
import "github.com/RichardKnop/somelibrary/v1"
Or:
import "github.com/RichardKnop/somelibrary/v2"
Based on what you need. Any changes made to v1 or v2 would be required not to break any APIs or working functionality.
II. Forking
This would give you a complete control over a version of external dependency your Go code requires.
For example, you could fork github.com/RichardKnop/somelibrary into your own GitHub account and then in your code do:
import "github.com/ForkingUser/somelibrary"
Then you would have to fork all external dependencies which seems a bit overkill. However it would give you total control over versions. You could keep your forks at a version you know is working with your code and only update forks once you have checked that new releases of dependencies do not break anything.
Thoughts?
Feb. 2018: the vendoring approach presented below (in 2015/2016) might end up disappearing if vgo is integrated to the toolchain.
See my answer below.
August 2015 edition: Go 1.5 comes with a built-in (but still experimental) vendoring support. Setting the environment variable GO15VENDOREXPERIMENT will make go build and friends look for packages in ./vendor directory as well as GOPATH. See VonC's answer and the design document for more details.
III. Vendoring
AFAIK, this is the most widely used way of ensuring that your builds are reproducible and predictable. The Go team itself uses vendoring in their repo. The Go team is now discussing the unified dependency manifest file format. From the Go toolchain developers mailing list:
In Google’s internal source tree, we vendor (copy) all our dependencies into our source tree and have at most one copy of any given external library. We have the equivalent of only one GOPATH and rewrite our imports to refer to our vendored copy. For example, Go code inside Google wanting to use “golang.org/x/crypto/openpgp” would instead import it as something like “google/third_party/golang.org/x/crypto/openpgp”.
(...)
Our proposal is that the Go project,
officially recommends vendoring into an “internal” directory with import rewriting (not GOPATH modifications) as the canonical way to pin dependencies.
defines a common config file format for dependencies & vendoring
makes no code changes to cmd/go in Go 1.5. External tools such as “godep” or “nut” will implement 1) and 2). We can reevaluate including such a tool in Go 1.6+.
Note: June 2015, the first support for vendoring appears in Go 1.5!
See c/10923/:
When GO15VENDOREXPERIMENT=1 is in the environment, this CL changes the resolution of import paths according to the Go 1.5 vendor proposal:
If there is a source directory d/vendor, then, when compiling a source file within the subtree rooted at d, import "p" is interpreted as import "d/vendor/p" if that exists.
When there are multiple possible resolutions, the most specific (longest) path wins.
The short form must always be used: no import path can contain “/vendor/” explicitly.
Import comments are ignored in vendored packages.
Update January 2016: Go 1.6 will make vendoring the default.
And as detailed in the article "MOST GO TOOLS NOW WORK WITH GO15VENDOREXPERIMENT":
1.6 brings support for /vendor/ to most tools (like the oracle) out of the box; use the Beta to rebuild them.
issue 12278 has been resolved.
there still is an issue with goimports, and there is a CL that can be cherry-picked.
Update August 2018: this (vgo presented below) is now implemented with Go 1.11 and modules.
Update Feb. 2018, 3 years later.
There is a new approach developed by Russ Cox, from the core Golang development team.
The vgo project.
go get -u golang.org/x/vgo
This proposal:
keeps the best parts of go get,
adds reproducible builds,
adopts semantic versioning,
eliminates vendoring,
deprecates GOPATH in favor of a project-based workflow, and
provides a smooth migration path from dep and its predecessors.
It is based on a new MVS ("Minimal Version Selection") algorithm:
You can see:
the first articles in Go & Versioning
the discussion in this thread
the analysis of that thread by Jon Calhoun
the counter-point by Sam Boyer in "Thoughts on vgo and dep":
He was leading the private initiative of go dep
the first debate (YouTube) between Russ, Sam and Jesse Frazelle!
May 2018: Artifactory proposes a vgo proxy
The recent release of Artifactory 5.11 added support for vgo-compatible Go registries (and go proxy) giving the community a variety of capabilities when developing with Go.
Here are just a few of them:
Local repositories in Artifactory let you set up secure, private Go registries with fine-grained access control to packages according to projects or development teams.
A remote repository in Artifactory is a caching proxy for remote Go resources such as a GitHub project. Accessing a go proxy through Artifactory removes your dependency on the network, or on GitHub since all dependencies needed for your Go builds are cached in Artifactory and are therefore locally available. This also removes the risk of someone mutating or removing a dependency from version control, or worse, force-pushing changes to a remote Git tag thus changing what should be an immutable version which can create a lot of confusion and instability for dependent projects.
A virtual repository aggregates both local and remote Go registries giving you access to all Go resources needed for your builds from a single URL, hiding the complexity of using a combination of both local and remote resources.
also it is possible just describe dependencies in maven, if to use mvn-golang-wrapper, in the case it looks like in maven as below text
<plugin>
<groupId>com.igormaznitsa</groupId>
<artifactId>mvn-golang-wrapper</artifactId>
<version>1.1.0</version>
<executions>
<execution>
<id>golang-get</id>
<goals>
<goal>get</goal>
</goals>
<configuration>
<packages>
<package>github.com/gizak/termui</package>
</packages>
<buildFlags>
<flag>-u</flag>
</buildFlags>
<goVersion>1.6</goVersion>
</configuration>
</execution>
</plugin>
I'm getting my hands dirty with Go, and while I understand and appreciate the principle of simplicity that Go was built upon, I'd like to grasp the rationale behind forgoing a built-in package versioning method in their dependency-fetching tool go get and the import statement.
If I understand correctly, go get and import fetch the package from HEAD and they are unable to refer to a branch or a tag. While there are tools like gopkg.in that circumvent this limitation, the official toolchain:
Forces developers to create separate repos for major (breaking) versions of their products.
It doesn't allow consumers to downgrade between minor or micro versions in case bugs are found in newer ones.
Truth be told, things are not so easy because package versioning would require a strategy to deal with conflicting transitive dependencies, e.g. X depends on A and B, each of which depend on different versions of C.
Coming from a Java background, it does appear that this limitation poses some risks and problems, amongst others:
Product/package evolution and breakage of public APIs of 3rd party deps is unavoidable, therefore versioning must be a first-class citizen in the toolchain IMHO.
The Git-repo-per-version policy is highly inefficient:
The overall Git history of the package is lost or scattered across repos (merges between versions, backports, etc.)
Conflicts with transitive dependencies may still occur, and will go about undetected because the language nor the toolchain impose any semantics to allow detection in the first place.
Enterprise adoption may be hindered and development teams may shy away from the language, given that:
Always dragging in HEAD means that they can't control or freeze their 3rd party deps, leading to a potentially unpredictable end product.
May lack the manpower to keep their product constantly updated and tested with upstream's HEAD (not every company in the world is Google :)).
While I do understand that the latter risk can be – and must be – mitigated with Continuous Integration, it does not solve the underlying root of the problem.
What information am I missing? How do you deal with package upstream changes when deploying Go in an enterprise with limited manpower?
It is being addressed by vendoring which is part of Go 1.5 as an experimental feature, it can be enabled if the go command is run with GO15VENDOREXPERIMENT=1 in its environment, and will be a "full" feature in Go 1.6. Also see Vendor Directories.
The original discussion that led to the Go 1.5 Vedor Experiment can be found here.
The essence of vendoring is that you create a folder named vendor, and you put the exact version of the packages that your code relies on. Code inside the vendor folder is only importable by the code in the directory tree rooted at the parent of vendor, and you can import packages from vendor with an import path as if vendor would be the workspace/src folder (that is, with an import path that omits the prefix up to and including the vendor element).
Example:
/home/user/goworkspace/
src/
mymath/
mymath.go
vendor/
github.com/somebob/math
math.go
In this example github.com/somebob/math is an external package used by mymath package (from mymath.go). It can be used from mymath.go if it is imported like:
import "github.com/somebob/math"
(And not as import mymath/vendor/github.com/somebob/math which would be bad.)
While Go doesn't come with the standard package manager there are enough options to make builds reproducible (even in an enterprise with limited manpower).
Vendoring, which is described in another answer by #icza. This is almost complete equivalent of checking in versioned jar files in Java. This was very common approach with ant build tool before maven became popular. Actually vendoring is much better because you cannot lose source code.
This is slight variation of the first option. Instead of checking in vendored source code you can populate vendor folder during build by checking out predefined versions of the dependencies. There tools (e.g. glide) that automate this process.
Finally you can maintain predefined versions of all 3-rd party libraries in the internal repository and add it to GOPATH. This approach is described in detail in https://blog.gopheracademy.com/advent-2015/go-in-a-monorepo/
Note that incompatible transitive dependencies are not specific to Go. They exist in Java (and most other languages) too, though Java has a mechanism to partially solve this problem by making programs more complex - class loaders. Note that Go will report all incompatibilities at compile time while in Java some of the incompatibilities are triggered only at run time (because of late linking).
Java toolchain doesn't have concept of versions. It is provided by the external tool - maven. I believe by the time Go becomes more mature and popular a similar standard dependency management tool will emerge.
In Golang, we can specify open source libraries on GitHub as dependencies. For example:
import "github.com/RichardKnop/somelibrary"
This will try to look for a branch based on your Go version and default to master if I understand correctly.
So there is no way to import a specific release of a dependency, e.g.:
import "github.com/RichardKnop/somelibrary#v1.4.8"
What is the best practise to manage dependencies in Go then?
I can see two approaches.
I. Version Modules
Is it to create new modules for major versions with breaking changes?
For example, my Go library could define modules v1 and v2 so then you could do:
import "github.com/RichardKnop/somelibrary/v1"
Or:
import "github.com/RichardKnop/somelibrary/v2"
Based on what you need. Any changes made to v1 or v2 would be required not to break any APIs or working functionality.
II. Forking
This would give you a complete control over a version of external dependency your Go code requires.
For example, you could fork github.com/RichardKnop/somelibrary into your own GitHub account and then in your code do:
import "github.com/ForkingUser/somelibrary"
Then you would have to fork all external dependencies which seems a bit overkill. However it would give you total control over versions. You could keep your forks at a version you know is working with your code and only update forks once you have checked that new releases of dependencies do not break anything.
Thoughts?
Feb. 2018: the vendoring approach presented below (in 2015/2016) might end up disappearing if vgo is integrated to the toolchain.
See my answer below.
August 2015 edition: Go 1.5 comes with a built-in (but still experimental) vendoring support. Setting the environment variable GO15VENDOREXPERIMENT will make go build and friends look for packages in ./vendor directory as well as GOPATH. See VonC's answer and the design document for more details.
III. Vendoring
AFAIK, this is the most widely used way of ensuring that your builds are reproducible and predictable. The Go team itself uses vendoring in their repo. The Go team is now discussing the unified dependency manifest file format. From the Go toolchain developers mailing list:
In Google’s internal source tree, we vendor (copy) all our dependencies into our source tree and have at most one copy of any given external library. We have the equivalent of only one GOPATH and rewrite our imports to refer to our vendored copy. For example, Go code inside Google wanting to use “golang.org/x/crypto/openpgp” would instead import it as something like “google/third_party/golang.org/x/crypto/openpgp”.
(...)
Our proposal is that the Go project,
officially recommends vendoring into an “internal” directory with import rewriting (not GOPATH modifications) as the canonical way to pin dependencies.
defines a common config file format for dependencies & vendoring
makes no code changes to cmd/go in Go 1.5. External tools such as “godep” or “nut” will implement 1) and 2). We can reevaluate including such a tool in Go 1.6+.
Note: June 2015, the first support for vendoring appears in Go 1.5!
See c/10923/:
When GO15VENDOREXPERIMENT=1 is in the environment, this CL changes the resolution of import paths according to the Go 1.5 vendor proposal:
If there is a source directory d/vendor, then, when compiling a source file within the subtree rooted at d, import "p" is interpreted as import "d/vendor/p" if that exists.
When there are multiple possible resolutions, the most specific (longest) path wins.
The short form must always be used: no import path can contain “/vendor/” explicitly.
Import comments are ignored in vendored packages.
Update January 2016: Go 1.6 will make vendoring the default.
And as detailed in the article "MOST GO TOOLS NOW WORK WITH GO15VENDOREXPERIMENT":
1.6 brings support for /vendor/ to most tools (like the oracle) out of the box; use the Beta to rebuild them.
issue 12278 has been resolved.
there still is an issue with goimports, and there is a CL that can be cherry-picked.
Update August 2018: this (vgo presented below) is now implemented with Go 1.11 and modules.
Update Feb. 2018, 3 years later.
There is a new approach developed by Russ Cox, from the core Golang development team.
The vgo project.
go get -u golang.org/x/vgo
This proposal:
keeps the best parts of go get,
adds reproducible builds,
adopts semantic versioning,
eliminates vendoring,
deprecates GOPATH in favor of a project-based workflow, and
provides a smooth migration path from dep and its predecessors.
It is based on a new MVS ("Minimal Version Selection") algorithm:
You can see:
the first articles in Go & Versioning
the discussion in this thread
the analysis of that thread by Jon Calhoun
the counter-point by Sam Boyer in "Thoughts on vgo and dep":
He was leading the private initiative of go dep
the first debate (YouTube) between Russ, Sam and Jesse Frazelle!
May 2018: Artifactory proposes a vgo proxy
The recent release of Artifactory 5.11 added support for vgo-compatible Go registries (and go proxy) giving the community a variety of capabilities when developing with Go.
Here are just a few of them:
Local repositories in Artifactory let you set up secure, private Go registries with fine-grained access control to packages according to projects or development teams.
A remote repository in Artifactory is a caching proxy for remote Go resources such as a GitHub project. Accessing a go proxy through Artifactory removes your dependency on the network, or on GitHub since all dependencies needed for your Go builds are cached in Artifactory and are therefore locally available. This also removes the risk of someone mutating or removing a dependency from version control, or worse, force-pushing changes to a remote Git tag thus changing what should be an immutable version which can create a lot of confusion and instability for dependent projects.
A virtual repository aggregates both local and remote Go registries giving you access to all Go resources needed for your builds from a single URL, hiding the complexity of using a combination of both local and remote resources.
also it is possible just describe dependencies in maven, if to use mvn-golang-wrapper, in the case it looks like in maven as below text
<plugin>
<groupId>com.igormaznitsa</groupId>
<artifactId>mvn-golang-wrapper</artifactId>
<version>1.1.0</version>
<executions>
<execution>
<id>golang-get</id>
<goals>
<goal>get</goal>
</goals>
<configuration>
<packages>
<package>github.com/gizak/termui</package>
</packages>
<buildFlags>
<flag>-u</flag>
</buildFlags>
<goVersion>1.6</goVersion>
</configuration>
</execution>
</plugin>