I'm getting my hands dirty with Go, and while I understand and appreciate the principle of simplicity that Go was built upon, I'd like to grasp the rationale behind forgoing a built-in package versioning method in their dependency-fetching tool go get and the import statement.
If I understand correctly, go get and import fetch the package from HEAD and they are unable to refer to a branch or a tag. While there are tools like gopkg.in that circumvent this limitation, the official toolchain:
Forces developers to create separate repos for major (breaking) versions of their products.
It doesn't allow consumers to downgrade between minor or micro versions in case bugs are found in newer ones.
Truth be told, things are not so easy because package versioning would require a strategy to deal with conflicting transitive dependencies, e.g. X depends on A and B, each of which depend on different versions of C.
Coming from a Java background, it does appear that this limitation poses some risks and problems, amongst others:
Product/package evolution and breakage of public APIs of 3rd party deps is unavoidable, therefore versioning must be a first-class citizen in the toolchain IMHO.
The Git-repo-per-version policy is highly inefficient:
The overall Git history of the package is lost or scattered across repos (merges between versions, backports, etc.)
Conflicts with transitive dependencies may still occur, and will go about undetected because the language nor the toolchain impose any semantics to allow detection in the first place.
Enterprise adoption may be hindered and development teams may shy away from the language, given that:
Always dragging in HEAD means that they can't control or freeze their 3rd party deps, leading to a potentially unpredictable end product.
May lack the manpower to keep their product constantly updated and tested with upstream's HEAD (not every company in the world is Google :)).
While I do understand that the latter risk can be – and must be – mitigated with Continuous Integration, it does not solve the underlying root of the problem.
What information am I missing? How do you deal with package upstream changes when deploying Go in an enterprise with limited manpower?
It is being addressed by vendoring which is part of Go 1.5 as an experimental feature, it can be enabled if the go command is run with GO15VENDOREXPERIMENT=1 in its environment, and will be a "full" feature in Go 1.6. Also see Vendor Directories.
The original discussion that led to the Go 1.5 Vedor Experiment can be found here.
The essence of vendoring is that you create a folder named vendor, and you put the exact version of the packages that your code relies on. Code inside the vendor folder is only importable by the code in the directory tree rooted at the parent of vendor, and you can import packages from vendor with an import path as if vendor would be the workspace/src folder (that is, with an import path that omits the prefix up to and including the vendor element).
Example:
/home/user/goworkspace/
src/
mymath/
mymath.go
vendor/
github.com/somebob/math
math.go
In this example github.com/somebob/math is an external package used by mymath package (from mymath.go). It can be used from mymath.go if it is imported like:
import "github.com/somebob/math"
(And not as import mymath/vendor/github.com/somebob/math which would be bad.)
While Go doesn't come with the standard package manager there are enough options to make builds reproducible (even in an enterprise with limited manpower).
Vendoring, which is described in another answer by #icza. This is almost complete equivalent of checking in versioned jar files in Java. This was very common approach with ant build tool before maven became popular. Actually vendoring is much better because you cannot lose source code.
This is slight variation of the first option. Instead of checking in vendored source code you can populate vendor folder during build by checking out predefined versions of the dependencies. There tools (e.g. glide) that automate this process.
Finally you can maintain predefined versions of all 3-rd party libraries in the internal repository and add it to GOPATH. This approach is described in detail in https://blog.gopheracademy.com/advent-2015/go-in-a-monorepo/
Note that incompatible transitive dependencies are not specific to Go. They exist in Java (and most other languages) too, though Java has a mechanism to partially solve this problem by making programs more complex - class loaders. Note that Go will report all incompatibilities at compile time while in Java some of the incompatibilities are triggered only at run time (because of late linking).
Java toolchain doesn't have concept of versions. It is provided by the external tool - maven. I believe by the time Go becomes more mature and popular a similar standard dependency management tool will emerge.
Related
In Golang, we can specify open source libraries on GitHub as dependencies. For example:
import "github.com/RichardKnop/somelibrary"
This will try to look for a branch based on your Go version and default to master if I understand correctly.
So there is no way to import a specific release of a dependency, e.g.:
import "github.com/RichardKnop/somelibrary#v1.4.8"
What is the best practise to manage dependencies in Go then?
I can see two approaches.
I. Version Modules
Is it to create new modules for major versions with breaking changes?
For example, my Go library could define modules v1 and v2 so then you could do:
import "github.com/RichardKnop/somelibrary/v1"
Or:
import "github.com/RichardKnop/somelibrary/v2"
Based on what you need. Any changes made to v1 or v2 would be required not to break any APIs or working functionality.
II. Forking
This would give you a complete control over a version of external dependency your Go code requires.
For example, you could fork github.com/RichardKnop/somelibrary into your own GitHub account and then in your code do:
import "github.com/ForkingUser/somelibrary"
Then you would have to fork all external dependencies which seems a bit overkill. However it would give you total control over versions. You could keep your forks at a version you know is working with your code and only update forks once you have checked that new releases of dependencies do not break anything.
Thoughts?
Feb. 2018: the vendoring approach presented below (in 2015/2016) might end up disappearing if vgo is integrated to the toolchain.
See my answer below.
August 2015 edition: Go 1.5 comes with a built-in (but still experimental) vendoring support. Setting the environment variable GO15VENDOREXPERIMENT will make go build and friends look for packages in ./vendor directory as well as GOPATH. See VonC's answer and the design document for more details.
III. Vendoring
AFAIK, this is the most widely used way of ensuring that your builds are reproducible and predictable. The Go team itself uses vendoring in their repo. The Go team is now discussing the unified dependency manifest file format. From the Go toolchain developers mailing list:
In Google’s internal source tree, we vendor (copy) all our dependencies into our source tree and have at most one copy of any given external library. We have the equivalent of only one GOPATH and rewrite our imports to refer to our vendored copy. For example, Go code inside Google wanting to use “golang.org/x/crypto/openpgp” would instead import it as something like “google/third_party/golang.org/x/crypto/openpgp”.
(...)
Our proposal is that the Go project,
officially recommends vendoring into an “internal” directory with import rewriting (not GOPATH modifications) as the canonical way to pin dependencies.
defines a common config file format for dependencies & vendoring
makes no code changes to cmd/go in Go 1.5. External tools such as “godep” or “nut” will implement 1) and 2). We can reevaluate including such a tool in Go 1.6+.
Note: June 2015, the first support for vendoring appears in Go 1.5!
See c/10923/:
When GO15VENDOREXPERIMENT=1 is in the environment, this CL changes the resolution of import paths according to the Go 1.5 vendor proposal:
If there is a source directory d/vendor, then, when compiling a source file within the subtree rooted at d, import "p" is interpreted as import "d/vendor/p" if that exists.
When there are multiple possible resolutions, the most specific (longest) path wins.
The short form must always be used: no import path can contain “/vendor/” explicitly.
Import comments are ignored in vendored packages.
Update January 2016: Go 1.6 will make vendoring the default.
And as detailed in the article "MOST GO TOOLS NOW WORK WITH GO15VENDOREXPERIMENT":
1.6 brings support for /vendor/ to most tools (like the oracle) out of the box; use the Beta to rebuild them.
issue 12278 has been resolved.
there still is an issue with goimports, and there is a CL that can be cherry-picked.
Update August 2018: this (vgo presented below) is now implemented with Go 1.11 and modules.
Update Feb. 2018, 3 years later.
There is a new approach developed by Russ Cox, from the core Golang development team.
The vgo project.
go get -u golang.org/x/vgo
This proposal:
keeps the best parts of go get,
adds reproducible builds,
adopts semantic versioning,
eliminates vendoring,
deprecates GOPATH in favor of a project-based workflow, and
provides a smooth migration path from dep and its predecessors.
It is based on a new MVS ("Minimal Version Selection") algorithm:
You can see:
the first articles in Go & Versioning
the discussion in this thread
the analysis of that thread by Jon Calhoun
the counter-point by Sam Boyer in "Thoughts on vgo and dep":
He was leading the private initiative of go dep
the first debate (YouTube) between Russ, Sam and Jesse Frazelle!
May 2018: Artifactory proposes a vgo proxy
The recent release of Artifactory 5.11 added support for vgo-compatible Go registries (and go proxy) giving the community a variety of capabilities when developing with Go.
Here are just a few of them:
Local repositories in Artifactory let you set up secure, private Go registries with fine-grained access control to packages according to projects or development teams.
A remote repository in Artifactory is a caching proxy for remote Go resources such as a GitHub project. Accessing a go proxy through Artifactory removes your dependency on the network, or on GitHub since all dependencies needed for your Go builds are cached in Artifactory and are therefore locally available. This also removes the risk of someone mutating or removing a dependency from version control, or worse, force-pushing changes to a remote Git tag thus changing what should be an immutable version which can create a lot of confusion and instability for dependent projects.
A virtual repository aggregates both local and remote Go registries giving you access to all Go resources needed for your builds from a single URL, hiding the complexity of using a combination of both local and remote resources.
also it is possible just describe dependencies in maven, if to use mvn-golang-wrapper, in the case it looks like in maven as below text
<plugin>
<groupId>com.igormaznitsa</groupId>
<artifactId>mvn-golang-wrapper</artifactId>
<version>1.1.0</version>
<executions>
<execution>
<id>golang-get</id>
<goals>
<goal>get</goal>
</goals>
<configuration>
<packages>
<package>github.com/gizak/termui</package>
</packages>
<buildFlags>
<flag>-u</flag>
</buildFlags>
<goVersion>1.6</goVersion>
</configuration>
</execution>
</plugin>
In Golang, we can specify open source libraries on GitHub as dependencies. For example:
import "github.com/RichardKnop/somelibrary"
This will try to look for a branch based on your Go version and default to master if I understand correctly.
So there is no way to import a specific release of a dependency, e.g.:
import "github.com/RichardKnop/somelibrary#v1.4.8"
What is the best practise to manage dependencies in Go then?
I can see two approaches.
I. Version Modules
Is it to create new modules for major versions with breaking changes?
For example, my Go library could define modules v1 and v2 so then you could do:
import "github.com/RichardKnop/somelibrary/v1"
Or:
import "github.com/RichardKnop/somelibrary/v2"
Based on what you need. Any changes made to v1 or v2 would be required not to break any APIs or working functionality.
II. Forking
This would give you a complete control over a version of external dependency your Go code requires.
For example, you could fork github.com/RichardKnop/somelibrary into your own GitHub account and then in your code do:
import "github.com/ForkingUser/somelibrary"
Then you would have to fork all external dependencies which seems a bit overkill. However it would give you total control over versions. You could keep your forks at a version you know is working with your code and only update forks once you have checked that new releases of dependencies do not break anything.
Thoughts?
Feb. 2018: the vendoring approach presented below (in 2015/2016) might end up disappearing if vgo is integrated to the toolchain.
See my answer below.
August 2015 edition: Go 1.5 comes with a built-in (but still experimental) vendoring support. Setting the environment variable GO15VENDOREXPERIMENT will make go build and friends look for packages in ./vendor directory as well as GOPATH. See VonC's answer and the design document for more details.
III. Vendoring
AFAIK, this is the most widely used way of ensuring that your builds are reproducible and predictable. The Go team itself uses vendoring in their repo. The Go team is now discussing the unified dependency manifest file format. From the Go toolchain developers mailing list:
In Google’s internal source tree, we vendor (copy) all our dependencies into our source tree and have at most one copy of any given external library. We have the equivalent of only one GOPATH and rewrite our imports to refer to our vendored copy. For example, Go code inside Google wanting to use “golang.org/x/crypto/openpgp” would instead import it as something like “google/third_party/golang.org/x/crypto/openpgp”.
(...)
Our proposal is that the Go project,
officially recommends vendoring into an “internal” directory with import rewriting (not GOPATH modifications) as the canonical way to pin dependencies.
defines a common config file format for dependencies & vendoring
makes no code changes to cmd/go in Go 1.5. External tools such as “godep” or “nut” will implement 1) and 2). We can reevaluate including such a tool in Go 1.6+.
Note: June 2015, the first support for vendoring appears in Go 1.5!
See c/10923/:
When GO15VENDOREXPERIMENT=1 is in the environment, this CL changes the resolution of import paths according to the Go 1.5 vendor proposal:
If there is a source directory d/vendor, then, when compiling a source file within the subtree rooted at d, import "p" is interpreted as import "d/vendor/p" if that exists.
When there are multiple possible resolutions, the most specific (longest) path wins.
The short form must always be used: no import path can contain “/vendor/” explicitly.
Import comments are ignored in vendored packages.
Update January 2016: Go 1.6 will make vendoring the default.
And as detailed in the article "MOST GO TOOLS NOW WORK WITH GO15VENDOREXPERIMENT":
1.6 brings support for /vendor/ to most tools (like the oracle) out of the box; use the Beta to rebuild them.
issue 12278 has been resolved.
there still is an issue with goimports, and there is a CL that can be cherry-picked.
Update August 2018: this (vgo presented below) is now implemented with Go 1.11 and modules.
Update Feb. 2018, 3 years later.
There is a new approach developed by Russ Cox, from the core Golang development team.
The vgo project.
go get -u golang.org/x/vgo
This proposal:
keeps the best parts of go get,
adds reproducible builds,
adopts semantic versioning,
eliminates vendoring,
deprecates GOPATH in favor of a project-based workflow, and
provides a smooth migration path from dep and its predecessors.
It is based on a new MVS ("Minimal Version Selection") algorithm:
You can see:
the first articles in Go & Versioning
the discussion in this thread
the analysis of that thread by Jon Calhoun
the counter-point by Sam Boyer in "Thoughts on vgo and dep":
He was leading the private initiative of go dep
the first debate (YouTube) between Russ, Sam and Jesse Frazelle!
May 2018: Artifactory proposes a vgo proxy
The recent release of Artifactory 5.11 added support for vgo-compatible Go registries (and go proxy) giving the community a variety of capabilities when developing with Go.
Here are just a few of them:
Local repositories in Artifactory let you set up secure, private Go registries with fine-grained access control to packages according to projects or development teams.
A remote repository in Artifactory is a caching proxy for remote Go resources such as a GitHub project. Accessing a go proxy through Artifactory removes your dependency on the network, or on GitHub since all dependencies needed for your Go builds are cached in Artifactory and are therefore locally available. This also removes the risk of someone mutating or removing a dependency from version control, or worse, force-pushing changes to a remote Git tag thus changing what should be an immutable version which can create a lot of confusion and instability for dependent projects.
A virtual repository aggregates both local and remote Go registries giving you access to all Go resources needed for your builds from a single URL, hiding the complexity of using a combination of both local and remote resources.
also it is possible just describe dependencies in maven, if to use mvn-golang-wrapper, in the case it looks like in maven as below text
<plugin>
<groupId>com.igormaznitsa</groupId>
<artifactId>mvn-golang-wrapper</artifactId>
<version>1.1.0</version>
<executions>
<execution>
<id>golang-get</id>
<goals>
<goal>get</goal>
</goals>
<configuration>
<packages>
<package>github.com/gizak/termui</package>
</packages>
<buildFlags>
<flag>-u</flag>
</buildFlags>
<goVersion>1.6</goVersion>
</configuration>
</execution>
</plugin>
I've read a whole bunch of articles and SO questions on importing 3rd party go packages which all seems straight forward, but what I don't understand is that none that I have read make any references to versioning. In Dartlang there's the pubspec file that defines your package including its version and its dependencies including their required versions. What if I do a go get github.com/gorilla/sessions and write my app then 6 months later I have to clear my directories and re get everything again, in which time that package has been update and broken backwards compatibility with my code that was using the older version?
The official version, from the GO FAQ:
If you're using an externally supplied package and worry that it might change in unexpected ways, the simplest solution is to copy it to your local repository. (This is the approach Google takes internally.) Store the copy under a new import path that identifies it as a local copy.
There are many alternative to that approach, mainly based on declaring the exact version of those projects you are using.
See for instance "Dead Simple Dependencies in Go -- Keep it simple and keep your sanity." (based on emil2k/vend)
The main different options for Go Dependency Management are listed at:
"Go Package Management -- A summary of dependency management in Go"
(And its associate GOPM mailing list)
Update July 2015:
the official vendoring approach from Go team is discussed here.
an alternative go build tool called "gb" is proposed at getgb.io by Dave Cheney.
Update Q4 2017: as mentioned below, go dep is the official tool for pinning version of dependencies (even though that pinning approach is not without criticism: see "The cargo cult of versioning").
It should be merged into the toolchain when Go 1.10 development begins, according to its roadmap.
Update Q2 2018: go dep has been replaced by go mod (modules) in Go 1.11, following works on vgo.
I use dep as a dependency management tool for golang project. Please use the following link dep tool for more info.
dep is a dependency management tool for Go. It requires Go 1.9 or newer to compile.
dep was the "official experiment." The Go toolchain, as of 1.11, has (experimentally) adopted an approach that sharply diverges from dep. As a result, we are continuing development of dep, but gearing work primarily towards the development of an alternative prototype for versioning behavior in the toolchain.
Current status: January 2019
dep is safe for production use.
There are multiple modules in our applications and each of them have their separate version and depend on other modules(external to our organization). They all have a parent POM which has it's own version, independent from the children's version.
When one of those modules change, they're converted to snapshots.
For the following example:
Parent v14.0
- module1 v1.5.0
- dependency1(module2 v15.0.0)
- dependency2(external-jar v12.0.1)
- module2 v15.0.0
- module3 v3.1.0
If there would be a change in module2, then module2's version would become v15.0-SNAPSHOT, then module1 becomes v1.5-SNAPSHOT. The parent remains the same.
The purpose for not having the same version on parent+modules, is that we want to localize the updates made to some modules and not affect the others' versions.
This has been designed like this a long time ago and there are several bash scripts to support the updates, although they're not handling all the cases. In any case, we don't have a one-click release process and we feel we are quite far from it with this approach.
We don't know how to convince the management towards a single version approach on all modules. How do you feel about the above? Did you ever encountered a project using the above structure and how well did it go?
Thank you!
I've had to deal with such situations before. There is an actual benefit from having decentralized versions, especially in cases where your product is made out of a large number of modules and this is because of the following facts:
You don't have to release all of them as a whole, if only a handful have been changed (which, from my observation is almost always the case).
You don't have to create unnecessary tags in your version control for code which hasn't changed since the previous release.
You don't have to waste an excessive amount of time releasing modules which don't need to be released.
You know with certainty which modules have changed in a release, which helps a lot when you need to investigate a complex bug, which seems to be dating back a while.
You can actually release certain modules/aggregators before the actual release date of the complete product, allowing for more testing time and a feeling of completeness for a given part of the product.
You can make feature branch releases much easier and implement a continuous delivery in a better way.
You can re-use the same code across multiple development branches without wondering if that branched version matches the one for your branch (or at least with less confusion).
What we ended up doing was:
Extract a parent or set of parents (with no sub-modules).
Try to use fixed versions for parents as much as possible. This is a bit of a caveat, as you must change all modules that inherit it, but in the end it improves the stability.
Extract each of the modules whose versions are independent of the rest to separate modules.
Extract sets of modules whose versions must always move along together into aggregators.
Create jobs in your CI server that can do releases or manually release these modules.
Use the versions-maven-plugin.
I think it's a lot more mature of a project and company's development principles to use decentralized versions and I must admit that in the beginning I was very reluctant to this approach. You might not realize or understand the benefits immediately, but with some practice and a proper setup, you will start seeing the upsides. I'm not saying there aren't caveats like... for example bumping the version of a parent, or having to know in which modules to bump the version of one of your modules.
From my experience, this module actually works better in the end, once you've become used to working with it.
From my experience: Everywhere we tried this, it eventually failed.
Although Maven supports this approach it is not advisable because of the additional effort.
I try to use the following criteria when choosing whether to use distinct projects or a multimodule structure:
If all projects have the same release cycle, I put them in a common multi module structure. In that case, I give them all the same version and release them together.
If a part of the project is used by different other projects (organizational projects), I always split them of and give them a separate lifecycle and a separate version.
If one part of my project stabilizes, I split it off, and give it a separate lifecycle (Maven refactoring)
To do it different alway results in homebrew solutions that neither scale well, nor are easy to maintain.
Hope that helps.
By default, Go pulls imported dependencies by grabbing the latest version in master (github) or default (mercurial) if it cannot find the dependency on your GOPATH. And while this workflow is quite simple to grasp, it has become somewhat difficult to tightly control. Because all software change incurs some risk, I'd like to reduce the risk of this potential change in a manageable and repeatable way and avoid inadvertently picking up changes of a dependency, especially when running clean builds via CI server or preparing to deploy.
What is the most effective way I can pin (i.e. lock down or capture) a package dependency so I don't find myself unable to reproduce an old package, or even worse, unexpectedly broken when I'm about to release?
---- Update ----
Additional info on the Current State of Go Packaging. While I ended up (as of 7.20.13) capturing dependencies in a 3rd party folder and managing updates (ala Camlistore), I'm still looking for a better way...
Here is a great list of options.
Also, be sure to see the go 1.5 vendor/ experiment to learn about how go might deal with the problem in future versions.
You might find the way Camlistore does it interesting.
See the third party directory and in particular the update.pl and rewrite-imports.sh script. These scripts update the external repositories, change imports if necessary and make sure that a static version of external repositories is checked in with the rest of the camlistore code.
This means that camlistore has a completely repeatable build as it is self contained, but the third party components can be updated under the control of the camlistore developers.
There is a project to help you in managing your dependencies. Check gopack
godep
I started using godep early last year (2014) and have been very happy with it (it met the concerns I mentioned in my original question). I am no longer using custom scripts to manage the vendoring of dependencies as godep just takes care of it. It has been excellent for ensuring that no drift is introduced regardless of timing or a machine's package state. It works with the existing mechanism of go get and introduces the ability to pin (godep save) and restore (godep restore) based on Godeps/godeps.json.
Check it out:
https://github.com/tools/godep
There is no built in tooling for this in go. However you can fork the dependencies yourself either on local disk or in a cloud service and only merge in upstream changes once you've vetted them.
The 3rd party repositories are completely under your control. 'go get' clones tip, you're right, but you're free to checkout any revision of the cloned-by-go-get or cloned-by-you repository. As long as you don't do 'go get -u', nothing touches your 3rd party repositories already sitting at your hard disk.
Effectively, your external, locally cloned, dependencies are always locked down by default.