I keep running into the issue where I install dependencies locally, it works fine, I push to continuous integration server, and then it breaks because I forgot to godep save ./... the dependency.
How can I run the go command but require vendor imports?
Edit:
I'm using go1.6. I want the command to fail if a 3rd-party dependency does not resolve to vendor. In other words, is there a way to stop resolving dependencies in $GOPATH during tests?
I can't change the environment variable because then none of my project modules can be resolved. How can I force vendor dependencies?
There is no way to prevent builder to scan $GOPATH for packages. It seems, that you use not really good flow for manage dependencies. I recommend you to use glide for a vendoring.
Most recommended workflow:
Keep actual list of dependencies in glide.yaml.
Run glide up after any changes in glide.yaml. It will install all dependencies to vendor directory and generate glide.lock with fixed package versions. Commit glide.lock to VCS. Do not change manually glide.lock.
Do not commit vendor directory to VCS.
Run glide install on your CI or build server to install dependencies by glide.lock to vendor.
Build.
A migration from godep to glide may be done easily, because glide has a command to migrate Godeps.json to glide.yaml.
Related
I'm using go 1.11 with module support. I understand that the go tool now installs dependencies automatically on build/install. I also understand the reasoning.
I'm using docker to build my binaries. In many other ecosystems its common to copy over your dependency manifest (package.json, requirements.txt, etc) and install dependencies as a separate stage from build. This takes advantage of docker's layer caching, and makes rebuilds much faster since generally code changes vastly outnumber dependency changes.
I was wondering if vgo has any way to do this?
It was an issue #26610, which is fixed now.
So now you can just use:
go mod download
For this to work you need just the go.mod / go.sum files.
For example, here's how to have a cached multistage Docker build: (source)
FROM golang:1.17-alpine as builder
RUN apk --no-cache add ca-certificates git
WORKDIR /build
# Fetch dependencies
COPY go.mod go.sum ./
RUN go mod download
# Build
COPY . ./
RUN CGO_ENABLED=0 go build
# Create final image
FROM alpine
WORKDIR /
COPY --from=builder /build/myapp .
EXPOSE 8080
CMD ["./myapp"]
Also see the article Containerize Your Go Developer Environment – Part 2, which describes how to leverage the Go compiler cache to speed up builds even further.
You may use the go mod vendor command which will create a vendor folder in the main module's root folder, and copy all dependencies into it. After this you may pass the -mod=vendor param to the go tool, and then dependencies from the vendor folder will be used to build / compile / test your app.
So what you may do to speed up your builds:
Run the go mod vendor command to have an actual version of your dependencies.
Save / cache this vendor folder.
During builds, restore this vendor folder, and build / install your app by passing the -mod=vendor argument to the go tool, so no dependencies will be downloaded, but the content of the vendor folder will be used.
Quoting from go help mod:
Modules and vendoring
When using modules, the go command completely ignores vendor directories.
By default, the go command satisfies dependencies by downloading modules
from their sources and using those downloaded copies (after verification,
as described in the previous section). To allow interoperation with older
versions of Go, or to ensure that all files used for a build are stored
together in a single file tree, 'go mod vendor' creates a directory named
vendor in the root directory of the main module and stores there all the
packages from dependency modules that are needed to support builds and
tests of packages in the main module.
To build using the main module's top-level vendor directory to satisfy
dependencies (disabling use of the usual network sources and local
caches), use 'go build -mod=vendor'. Note that only the main module's
top-level vendor directory is used; vendor directories in other locations
are still ignored.
I wanted to re-download all the dependencies using go mod, this is what I did:
Go to your GOROOT
sudo rm -rf pkg/mod/
Go to the directory where the go.mod file exists
go mod download
You can use a package manager, There are many of them like dep, glide, and govendor. dep is newer and is going to be integrated into go toolchain as official dependency management tool.
We also make docker images for go applications and we use dind to make those images and we prepared a CI/CD image with all dependencies preinstalled to make the builds faster. Though, it took a little bit of scripting to glue everything together.
Moreover, layering up the dependencies could result in big size of docker images. I suggest try multi-stage builds which could help making images super lite.
When using govendor, what is the difference between go install, govendor install +local and govendor install +vendor,^program?
govendor install +vendor,^program says to build and install all my vendor packages. but what and where will it install to? Will it install my project and vendor's command executables to $GOPATH/bin and my project and vendor's package objects to $GOPATH/pkg?
govendor install +local says to build everything in your repository only. So what does it really mean? Will it create vendor/bin and vendor/pkg?
what about if I run go install in my project? What will this be different to the above two commands?
Go first came into the world with a brand new idea for dependency management and workspace folder structure. There was a strict hierarchy where projects would be located (in $gopath/src/site.com/user/project) and other projects would simply import the latest version of all other projects. The problem with this is that if some upstream project changes the API, then your project will break inexplicably. This is where vendor comes in.
Vendor is a subdirectory in your project that contains everything under $gopath/src that your project imports. The difference is that vendor isn't updated when upstream projects introduce new features and/or fixes. Therefore, you must update it yourself. When go looks for an import (as of the latest version), it will check vendor first and then look for the latest version in your $gopath, preventing builds from inexplicably breaking for no apparent reason.
go install updates your $gopath dependencies to the latest version; the version that all new projects and those without vendor will use.
govendor install +vendor,^program updates your specific project vendor dependencies. This should be done in a separate commit; you should go test; govendor; git commit; go test so that you can check whether updates break your project.
govendor install +local apparently just builds the project.
Also, you should IMHO use godep rather than govendor. It IHMO has better workflow; your $gopath has the latest versions and then you can update your project with godep update. It is also supposed to be standard in golang 1.10.
I have used godep and vendored all my dependencies in vendor/ directory. Go build is working fine as well. However how can I be sure that all my dependencies are vendored?
Is there any command that can make sure of that?
My CI service (Travis is the one I use) lets me know. Because my test build will fail if the deps aren't available.
You should be using a CI service anyway, and then you get that benefit for free.
I use govendor to manage the dependencies, which has a status option. Here's some of the commands with govendor:
init Create the "vendor" folder and the "vendor.json" file.
list List and filter existing dependencies and packages.
add Add packages from $GOPATH.
update Update packages from $GOPATH.
remove Remove packages from the vendor folder.
status Lists any packages missing, out-of-date, or modified locally.
fetch Add new or update vendor folder packages from remote repository.
sync Pull packages into vendor folder from remote repository with revisions
from vendor.json file.
migrate Move packages from a legacy tool to the vendor folder with metadata.
Specifically, you'd do govendor status to check if there are missing packages.
If you decide to use govendor, you can get started by doing:
go get github.com/kardianos/govendor
govendor migrate (which will migrate from godeps to govendor)
Also, you mentioned in a comment that your deploying to Heroky, here's some documentation from them on govendor
The golang Glide packager automatically will install dependencies for a project, however, in doing so it pulls in dependencies from the vendor/, since many projects check in vendor dependencies.
How should one import a glide dependency which includes a vendor/ directory?
As of now, it appears that you can get errors such as :
./scheduler.go:36: cannot use "github.com/jayunit100/my-project/vendor/github.com/spf13/pflag".CommandLine (type *"github.com/jayunit100/my-project/vendor/a/b/spf13/pflag".FlagSet)
as type
*"github.com/jayunit100/my-project/vendor/a/b/vendor/github.com/spf13/pflag".FlagSet
import as normal, and then manually delete vendor/ so there are no redundancies.
import as normal, and use some kind of configuration trick to avoid the vendor/ dependency duplication?
I think this is quite a common issue for people using glide. My team ran into the same issue and had to investigate for a while but eventually found that using the flatten operation solved this for us.
To do this pass the -v flag to your glide operations.
e.g. glide up -v
You can read more about it on the glide docs.
My current solution has been to manually delete vendor/ directories from libraries which are brought in that keep vendor/.
... (Update) glide supports a --strip-vendor option.
I am working on a project that requires some dependencies. I am a little confused by the best practices regarding vendoring. Currently I am using GO15VENDOREXPERIMENT and have copied the dependencies into the vendor directory.
https://github.com/jeffellin/machine-cloudformation/tree/master/vendor/github.com
When I need to update the dependencies I checkout the source code manually from docker-machine and copy it into my vendor directory. This brings in a lot of unnecessary code, tests, etc. Is there any recommendations on automating this? I am used to maven where I can just specify version info in a configuration file.
What about GODEPS? How does that play into the GO15VENDOREXPERIMENT. I would prefer a solution where I don't have to checkin dependencies into my source tree.
There are a few tools that help with vendoring your dependencies. Most of them are able to handle GO15VENDOREXPERIMENT.
For example:
godep
glide