I come from a ruby background, and I just started learning go. Is there any standard way to install 3rd-party libraries that's comparable to RubyGems?
Since go1.11 released, we have an official go package management tools, the Go Modules.
The difference between go modules and other package management tools is go modules does not rely on $GOPATH. The project must be placed outside of $GOPATH. If your project is already inside a $GOPATH but you wanted to use package management tools, then I suggest to see the old answer below.
Usage:
mkdir testproject
cd testproject
# init project as go module with root package name is testproject
go mod init testproject
# install 3rd party library, it will be stored inside testproject/vendor
go get github.com/labstack/echo
go get github.com/novalagung/gubrak
the go mod init command generates Go.mod file (similar like Gemfile for ruby). You can either install the 3rd party libraries through the usual go get command, or by adding the library metadata into Go.mod file then perform go mod tidy.
More informations about Go Modules: https://blog.golang.org/using-go-modules
Old answer
Go does have package management tool as well, it's called dep.
Usage example:
cd $GOPATH/src
mkdir testproject
cd testproject
# init project
dep init
# install 3rd party library
dep ensure -add github.com/labstack/echo
dep ensure -add github.com/novalagung/gubrak
dep generates Gopkg.toml file (similar like Gemfile for ruby). You can either install the 3rd party libraries through dep ensure -add command, or by adding the library metadata into Gopkg.toml then perform dep ensure.
Btw, there is also few other alternatives other than dep. For more information please take a look at https://github.com/golang/go/wiki/PackageManagementTools.
Related
I have an external package that apart from the usual go get, needs to run make install in its $GOPATH/src directory in order to use it (performs some makefile and git magic).
Trying to use this package with modules means a copy of it is downloaded to the vendor library using go mod vendor. However, this copy is not a git repository so running make install inside the package's vendor folder fails.
Does this mean that the package cannot be used in a module and I have to revert to using GOPATH?
Does this mean that the package cannot be used in a module
Yes.
Contact the author and make him to check in what the makefile does.
I've recently started with Go 1.11 and love the modules. Apart from runtime dependencies I need to work with go modules during the build, e.g. during go generate.
How can I install a specific build dependency (e.g. github.com/aprice/embed/cmd/embed) and run that specific tool from which folder? Is go get the right tool for doing so?
If you get an error
I was not seeing the dependency that I wanted added to the go.mod and I was getting this error:
internal/tools/tools.go:6:5: import "github.com/UnnoTed/fileb0x" is a program, not an importable package
(fileb0x is the thing I'm trying to add)
I'm not 100% clear on the sequence of events that fixed it, but I did all of these things:
Using a "tools" package
I made a tools directory:
mkdir -p internal/tools
I put the tools package inside of it (as mentioned above):
internal/tools/tools.go:
// +build tools
package tools
import (
_ "github.com/UnnoTed/fileb0x"
)
Note that the tag is mostly not important. You could use foo:
// +build foo
However, you cannot use ignore. That's a special predefined tag.
// +build ignore
// NO NO NO NO NO
// `ignore` is a special keyword which (surprise) will cause
// the file to be ignore, even for dependencies
Updating go.mod
The best way is probably to run go mod tidy:
go mod tidy
However, before I did that I ran a number of commands trying to figure out which one would cause it to go into go.mod:
go install github.com/UnnoTed/fileb0x # didn't seem to do the trick
go get
go generate ./...
go build ./...
go install ./...
go mod vendor
Later I did a git reset and rm -rf ~/go/pkg/mod; mkdir ~/go/pkg/mod and found that go mod tidy did well enough on its own.
vendoring
In order to actually take advantage of the modules cache in a project you need to copy-in the source code
go mod vendor
That will grab all dependencies from go.mod
You also need to change nearly all of your go commands to use -mod=vendor in any Makefiles, Dockerfiles or other scripts.
go fmt -mod=vendor ./... # has a bug slated to be fixed in go1.15
go generate -mod=vendor ./...
go build -mod=vendor ./...
That includes go build, go get, go install, and any go run called by go generate (and even the go generate itself)
//go:generate go run -mod=vendor github.com/UnnoTed/fileb0x b0x.toml
package main
// ...
https://github.com/golang/go/issues/25922 proved helpful for me, especially
when using build-only dependencies with modules the main point is version selection (not installing these!)
To avoid installing you can modify your //go:generate directive to something like:
//go:generate go run golang.org/x/tools/cmd/stringer ARGS
There is also the best practices repo: https://github.com/go-modules-by-example/index/blob/master/010_tools/README.md
The convention is to add a file named "tools.go" that is guarded by a build constraint and imports all required tools:
// +build tools
package tools
import (
_ "github.com/aprice/embed/cmd/embed"
)
https://github.com/golang/go/issues/25922#issuecomment-412992431
The tools are then installed as usual in one of
$GOBIN
$GOPATH/bin
$HOME/go/bin
You may also want to follow https://github.com/golang/go/issues/27653, which discusses future explicit support for tools.
tools.go is a great solution if you're building an app or service. But if you're building a library, tools.go still leaks dependencies to things consuming your library (your tools are still there as indirect dependencies, and go mod tidy will pull them in since it considers every possible target). That's not the end of the world since those modules never end up in the actual built binaries of the consumer, but it's still messy.
https://github.com/myitcv/gobin/issues/44 is probably the most promising approach to fixing this long term, but short term I've used a combination of the "internal module" approach explained there along with https://github.com/izumin5210/gex.
First, I install gex globally:
GO111MODULE=off go get github.com/izumin5210/gex/cmd/gex
Then before actually using gex I create a structure like this:
myproject/
\
- go.mod: module github.com/ysamlan/myproject
\
internal/
\
tools/
- go.mod: module github.com/ysamlan/myproject/tools
To install a build-only tool I just cd internal/tools and run gex --add (sometool), which puts that tool in internal/tools/bin. CI scripts and other folks that want to build my stuff locally just need to run cd internal/tools && gex --build to reliably and reproducibly populate the tool binaries, but the top-level go.mod is unchanged.
The key piece there is creating that internal/tools/go.mod file with a different module path than the one the root project uses, and then only running gex from that directory.
I'm using go 1.11 with module support. I understand that the go tool now installs dependencies automatically on build/install. I also understand the reasoning.
I'm using docker to build my binaries. In many other ecosystems its common to copy over your dependency manifest (package.json, requirements.txt, etc) and install dependencies as a separate stage from build. This takes advantage of docker's layer caching, and makes rebuilds much faster since generally code changes vastly outnumber dependency changes.
I was wondering if vgo has any way to do this?
It was an issue #26610, which is fixed now.
So now you can just use:
go mod download
For this to work you need just the go.mod / go.sum files.
For example, here's how to have a cached multistage Docker build: (source)
FROM golang:1.17-alpine as builder
RUN apk --no-cache add ca-certificates git
WORKDIR /build
# Fetch dependencies
COPY go.mod go.sum ./
RUN go mod download
# Build
COPY . ./
RUN CGO_ENABLED=0 go build
# Create final image
FROM alpine
WORKDIR /
COPY --from=builder /build/myapp .
EXPOSE 8080
CMD ["./myapp"]
Also see the article Containerize Your Go Developer Environment – Part 2, which describes how to leverage the Go compiler cache to speed up builds even further.
You may use the go mod vendor command which will create a vendor folder in the main module's root folder, and copy all dependencies into it. After this you may pass the -mod=vendor param to the go tool, and then dependencies from the vendor folder will be used to build / compile / test your app.
So what you may do to speed up your builds:
Run the go mod vendor command to have an actual version of your dependencies.
Save / cache this vendor folder.
During builds, restore this vendor folder, and build / install your app by passing the -mod=vendor argument to the go tool, so no dependencies will be downloaded, but the content of the vendor folder will be used.
Quoting from go help mod:
Modules and vendoring
When using modules, the go command completely ignores vendor directories.
By default, the go command satisfies dependencies by downloading modules
from their sources and using those downloaded copies (after verification,
as described in the previous section). To allow interoperation with older
versions of Go, or to ensure that all files used for a build are stored
together in a single file tree, 'go mod vendor' creates a directory named
vendor in the root directory of the main module and stores there all the
packages from dependency modules that are needed to support builds and
tests of packages in the main module.
To build using the main module's top-level vendor directory to satisfy
dependencies (disabling use of the usual network sources and local
caches), use 'go build -mod=vendor'. Note that only the main module's
top-level vendor directory is used; vendor directories in other locations
are still ignored.
I wanted to re-download all the dependencies using go mod, this is what I did:
Go to your GOROOT
sudo rm -rf pkg/mod/
Go to the directory where the go.mod file exists
go mod download
You can use a package manager, There are many of them like dep, glide, and govendor. dep is newer and is going to be integrated into go toolchain as official dependency management tool.
We also make docker images for go applications and we use dind to make those images and we prepared a CI/CD image with all dependencies preinstalled to make the builds faster. Though, it took a little bit of scripting to glue everything together.
Moreover, layering up the dependencies could result in big size of docker images. I suggest try multi-stage builds which could help making images super lite.
Being very new to Go, I'm trying to import a third party library into a vendor folder. I follow the instructions given by the Go docs, but didn't find anything about third party libraries.
Update (2019)
The Go environment is slowly starting to move away from tools like dep and towards the native Go tooling around modules. While explaining models is outside the scope of this answer, you can look into modules from the following places:
https://blog.golang.org/modules2019
https://github.com/golang/go/wiki/Modules
tldr
Install dep: go get -u github.com/golang/dep/cmd/dep
In your project run: dep init
answer
The easiest way to solve this problem in my opinion is using the dep dependency management tool. This tool is very widely in use and is very easy to use. Here is a typical workflow:
First you should install the dep program.
go get -u github.com/golang/dep/cmd/dep
Now you have access to the dep command. Full documentation can be found here: https://golang.github.io/dep/
This is how you get 3rd party libraries into your vendor directory. In the example below we will use the url router github.com/gorilla/mux.
First, in you code import the libraries like normal.
package main
import "github.com/gorilla/mux"
func main {
r := mux.NewRouter()
r.HandleFunc("/", HomeHandler)
r.HandleFunc("/products", ProductsHandler)
r.HandleFunc("/articles", ArticlesHandler)
http.Handle("/", r)
}
Now all we have to do to get it sed up is run the dep init command. This will look for all of your imports and create a vendor directory for you with all of your needed dependencies. Note that dep automatically analyzes your imports.
Dep in depth
Once you have initialized dep you can start work on your project as normal. When you add a new library you can run the dep ensure command to get the newly added 3rd party libraries in the vendor directory.
In addition, dep gives you the capability to lock down on particular versions of 3rd party libraries. dep init initializes your project with two files: Gopkg.toml and Gopkg.lock. The Gopkg.toml file contains assertions about which dependencies will be at what version. For example, if you wantted the gorilla mux library to stay at version v1.4.0, you could add the following line to your Gopkg.toml:
[[constraint]]
name = "github.com/gorilla/mux"
version = "=v1.4.0"
Dep also has functionality to upgrade dependencies, remove unused dependencies from vendor, and much more. Look at the documentation for more details. https://golang.github.io/dep/
coming from a Node environment I used to install a specific version of a vendor lib into the project folder (node_modules) by telling npm to install that version of that lib from the package.json or even directly from the console, like so:
$ npm install express#4.0.0
Then I used to import that version of that package in my project just with:
var express = require('express');
Now, I want to do the same thing with go. How can I do that?
Is it possible to install a specific version of a package? If so, using a centralized $GOPATH, how can I import one version instead of another?
I would do something like this:
$ go get github.com/wilk/uuid#0.0.1
$ go get github.com/wilk/uuid#0.0.2
But then, how can I make a difference during the import?
Go 1.11 will have a feature called go modules and you can simply add a dependency with a version. Follow these steps:
go mod init .
go mod edit -require github.com/wilk/uuid#0.0.1
go get -v -t ./...
go build
go install
Here's more info on that topic - https://github.com/golang/go/wiki/Modules
Really surprised nobody has mentioned gopkg.in.
gopkg.in is a service that provides a wrapper (redirect) that lets you express versions as repo urls, without actually creating repos. E.g. gopkg.in/yaml.v1 vs gopkg.in/yaml.v2, even though they both live at https://github.com/go-yaml/yaml
gopkg.in/yaml.v1 redirects to https://github.com/go-yaml/yaml/tree/v1
gopkg.in/yaml.v2 redirects to https://github.com/go-yaml/yaml/tree/v2
This isn't perfect if the author is not following proper versioning practices (by incrementing the version number when breaking backwards compatibility), but it does work with branches and tags.
A little cheat sheet on module queries.
To check all existing versions: e.g. go list -m -versions github.com/gorilla/mux
Specific version #v1.2.8
Specific commit #c783230
Specific branch #master
Version prefix #v2
Comparison #>=2.1.5
Latest #latest
E.g. go get github.com/gorilla/mux#v1.7.4
You can use git checkout to get an specific version and build your program using this version.
Example:
export GOPATH=~/
go get github.com/whateveruser/whateverrepo
cd ~/src/github.com/whateveruser/whateverrepo
git tag -l
# supose tag v0.0.2 is correct version
git checkout tags/v0.0.2
go run whateverpackage/main.go
Glide is a really elegant package management for Go especially if you come from Node's npm or Rust's cargo.
It behaves closely to Godep's new vendor feature in 1.6 but is way more easier. Your dependencies and versions are "locked" inside your projectdir/vendor directory without relying on GOPATH.
Install with brew (OS X)
$ brew install glide
Init the glide.yaml file (akin to package.json). This also grabs the existing imported packages in your project from GOPATH and copy then to the project's vendor/ directory.
$ glide init
Get new packages
$ glide get vcs/namespace/package
Update and lock the packages' versions. This creates glide.lock file in your project directory to lock the versions.
$ glide up
I tried glide and been happily using it for my current project.
Nowadays you can just use go get for it. You can fetch your dependency by the version tag, branch or even the commit.
go get github.com/someone/some_module#master
go get github.com/someone/some_module#v1.1.0
go get github.com/someone/some_module#commit_hash
more details here - How to point Go module dependency in go.mod to a latest commit in a repo?
Go get will also install the binary, like it says in the documentation -
Get downloads the packages named by the import paths, along with their dependencies. It then installs the named packages, like 'go install'.
(from https://golang.org/cmd/go/)
Update 18-11-23: From Go 1.11 mod is official experiment. Please see #krish answer.
Update 19-01-01: From Go 1.12 mod is still official experiment.
Starting in Go 1.13, module mode will be the default for all development.
Update 19-10-17: From Go 1.13 mod is official package manager.
https://blog.golang.org/using-go-modules
Old answer:
You can set version by offical dep
dep ensure --add github.com/gorilla/websocket#1.2.0
From Go 1.5 there's the "vendor experiment" that helps you manage dependencies. As of Go 1.6 this is no longer an experiment. Theres also some other options on the Go wiki..
Edit: as mentioned in this answer gopkg.in is a good option for pinning github-depdencies pre-1.5.
dep is the official experiment for dependency management for Go language. It requires Go 1.8 or newer to compile.
To start managing dependencies using dep, run the following command from your project's root directory:
dep init
After execution two files will be generated: Gopkg.toml ("manifest"), Gopkg.lock and necessary packages will be downloaded into vendor directory.
Let's assume that you have the project which uses github.com/gorilla/websocket package. dep will generate following files:
Gopkg.toml
# Gopkg.toml example
#
# Refer to https://github.com/golang/dep/blob/master/docs/Gopkg.toml.md
# for detailed Gopkg.toml documentation.
#
# required = ["github.com/user/thing/cmd/thing"]
# ignored = ["github.com/user/project/pkgX", "bitbucket.org/user/project/pkgA/pkgY"]
#
# [[constraint]]
# name = "github.com/user/project"
# version = "1.0.0"
#
# [[constraint]]
# name = "github.com/user/project2"
# branch = "dev"
# source = "github.com/myfork/project2"
#
# [[override]]
# name = "github.com/x/y"
# version = "2.4.0"
[[constraint]]
name = "github.com/gorilla/websocket"
version = "1.2.0"
Gopkg.lock
# This file is autogenerated, do not edit; changes may be undone by the next 'dep ensure'.
[[projects]]
name = "github.com/gorilla/websocket"
packages = ["."]
revision = "ea4d1f681babbce9545c9c5f3d5194a789c89f5b"
version = "v1.2.0"
[solve-meta]
analyzer-name = "dep"
analyzer-version = 1
inputs-digest = "941e8dbe52e16e8a7dff4068b7ba53ae69a5748b29fbf2bcb5df3a063ac52261"
solver-name = "gps-cdcl"
solver-version = 1
There are commands which help you to update/delete/etc packages, please find more info on official github repo of dep (dependency management tool for Go).
go get is the Go package manager. It works in a completely decentralized way and how package discovery still possible without a central package hosting repository.
Besides locating and downloading packages, the other big role of a package manager is handling multiple versions of the same package. Go takes the most minimal and pragmatic approach of any package manager. There is no such thing as multiple versions of a Go package.
go get always pulls from the HEAD of the default branch in the repository. Always. This has two important implications:
As a package author, you must adhere to the stable HEAD philosophy. Your default branch must always be the stable, released version of your package. You must do work in feature branches and only merge when ready to release.
New major versions of your package must have their own repository. Put simply, each major version of your package (following semantic versioning) would have its own repository and thus its own import path.
e.g. github.com/jpoehls/gophermail-v1 and github.com/jpoehls/gophermail-v2.
As someone building an application in Go, the above philosophy really doesn't have a downside. Every import path is a stable API. There are no version numbers to worry about. Awesome!
For more details : http://zduck.com/2014/go-and-package-versioning/
The approach I've found workable is git's submodule system. Using that you can submodule in a given version of the code and upgrading/downgrading is explicit and recorded - never haphazard.
The folder structure I've taken with this is:
+ myproject
++ src
+++ myproject
+++ github.com
++++ submoduled_project of some kind.
That worked for me
GO111MODULE=on go get -u github.com/segmentio/aws-okta#v0.22.1
There's a go edit -replace command to append a specific commit (even from another forked repository) on top of the current version of a package.
What's cool about this option, is that you don't need to know the exact pseudo version beforehand, just the commit hash id.
For example, I'm using the stable version of package "github.com/onsi/ginkgo v1.8.0".
Now I want - without modifying this line of required package in go.mod - to append a patch from my fork, on top of the ginkgo version:
$ GO111MODULE="on" go mod edit -replace=github.com/onsi/ginkgo=github.com/manosnoam/ginkgo#d6423c2
After the first time you build or test your module, GO will try to pull the new version, and then generate the "replace" line with the correct pseudo version. For example in my case, it will add on the bottom of go.mod:
replace github.com/onsi/ginkgo => github.com/manosnoam/ginkgo v0.0.0-20190902135631-1995eead7451
It might be useful.
Just type this into your command prompt while cd your/package/src/
go get github.com/go-gl/mathgl#v1.0.0
You get specific revision of package in question right into your source code, ready to use in import statement.
The current way to do this is to use go install
https://golang.org/doc/go-get-install-deprecation
Starting in Go 1.17, installing executables with go get is deprecated. go install may be used instead.
go install github.com/someone/some_module
Specific version
go install github.com/someone/some_module#v1.1.0
Specific commit
go install github.com/someone/some_module#commit_hash