I'll start with a disclaimer here. I'm running go on Ubuntu 18.0.4 on Windows subsystem.
I have a package in which I was able to run "dep init" successfully from the windows command line. I created a new clone (successful clone, so no network connectivity issues) of the repo in my Ubuntu subsystem and ran "dep init."
This is when I run into problems. Dep just hangs after pulling a single package down. Subsequent calls to dep init do not pull more packages. Running "dep init -v" produces the following:
#> dep init -v
Getting direct dependencies...
Checked 14 directories for packages.
Found 3 direct dependencies.
Take a look at the attached screen capture to see that process threads are spun up but are doing nothing (0% CPU and RAM usage).
Screen Capture with HTOP and my dep pkg folder structure
Things I've tried:
Turning it off and on again.
I've completely cleaned out my dep package folder and re-run dep init. This is how I know that there is just a single package that is actually getting pulled.
Made sure that my ssh keys are up-to-date with github.
dep was the "official experiment." The Go toolchain, as of 1.11, has adopted an approach that sharply diverges from dep. As a result, we are continuing development of dep, but gearing work primarily towards the development of an alternative prototype for versioning behaviour in the toolchain.
Go 1.11 included preliminary support for modules and includes support for vendoring.
You should stop using dep and refer to How to Define a Module to migrate your project.
First, navigate to your source tree, and then follow these instructions:
Create the initial module definition and write it to the go.mod file:
$ go mod init
This step converts from any existing dep Gopkg.lock file or from any of the other nine total supported dependency formats, adding require statements to match the existing configuration.
go mod init will often be able to use auxiliary data (such as VCS meta-data) to automatically determine the appropriate module path, but if go mod init states it can not automatically determine the module path, or if you need to otherwise override that path, you can supply the module path as an optional argument to go mod init, for example:
$ go mod init github.com/my/repo
Related
I'm trying to get a good understanding of Go modules and am a bit puzzled by the difference between the go get command and the go mod download command.
"The go get command updates module dependencies in the go.mod file for the main module, then builds and installs packages listed on the command line." https://golang.org/ref/mod#go-get
Whereas the Go mod download is described as :
"The go mod download command downloads the named modules into the module cache. " https://golang.org/ref/mod#go-mod-download
Clearly go get performs some dependency management which go mod download does not, but what is the difference between installing packages with go get and downloading modules to the module cache in go mod download.
Your module's go.mod file records which versions of dependencies it requires. The source code for those dependencies is stored in a local cache.
go get updates the requirements listed in your go.mod file. It also ensures that those requirements are self-consistent, and adds new requirements as needed so that every package imported by the packages you named on the command line is provided by some module in your requirements.
As a side-effect of updating and adding requirements, go get also downloads the modules containing the named packages (and their dependencies) to the local module cache.
In contrast, go mod download does not add new requirements or update existing requirements. (At most, it will ensure that the existing requirements are self-consistent, which can occur if you have hand-edited the go.mod file.) It only downloads either the specific module versions you've requested (if you requested specific versions), or the versions of modules that appear in your requirements.
I'm a bit confused about how Go modules work on installing binaries using go install.
I tried to install (https://github.com/joho/godotenv) binary by executing go install github.com/joho/godotenv/cmd/godotenv and I found out that it adds a record in the go.mod.
I'm lost as we don't use this package in the code and after running go mod tidy it gets deleted (as it is not it the code).
Can someone explain is it expected behaviour of go modules?
Secondly, how I could avoid adding it to the go.mod as we only need to install and execute the binary?
Thanks.
Go version: go version go1.13.4 darwin/amd64
Command go: The go.mod file:
The go command automatically updates go.mod each time it uses the module graph, to make sure go.mod always accurately reflects reality and is properly formatted.
The go tool will update go.mod automatically when it detects dependencies are inaccurate when performing a build.
When you install github.com/joho/godotenv/cmd/godotenv from your module, this install requires at least the package in question being built / installed (and also its dependencies, transitively).
You may safely run go mod tidy to undo the recording of this "one-time" dependency.
In general if you want to disallow the go tool to update the go.mod file, you may use the -mod=readonly flag, but that would fail go install ("can't load package: package xxx: import lookup disabled by -mod=readonly"). You can read more about this here: Go Wiki: Go modules: Can I control when go.mod gets updated and when the go tools use the network to satisfy dependencies?
Alternatively, if you want to avoid this, build / install your tools outside of your module. You may use a "dummy" module for this.
I am getting a stuck while trying to create a Terraform provider. I have been following the advice given on https://www.terraform.io/docs/extend/writing-custom-providers.html but when I go to build my binaries via Go go build -o terraform-provider-example I get several missing packages errors.
So I then work my way down the list of missing packages and use go get ... to get those packages installed in my Go libraries.
I get an error indicating that github.com/hashicorp/hcl/v2 cannot be found. I navigate to the location and sure enough it doesn't exist.
Package not available at install time screen shot
Package not available with go get
So I am getting stuck and unable to build these providers. I have looked for a while now trying to find something which describes how to setup the environment for creating providers but have been unsuccessful so far. Can anyone help get me going?
Terraform Core and Terraform provider development requires using the Go toolchain in the new "modules mode", which in current versions of Go is not the default.
The easiest way to ensure you're working in modules mode is to clone the repository you want to work on outside the $GOPATH/src directory. Development outside of GOPATH is only supported in Modules mode, and so the Go toolchain assumes that you intend to use modules mode if you are working in that way.
One reason why Terraform development requires modules mode (though not the only one) is that it has a dependency on github.com/hashicorp/hcl/v2, which is a module path type that is not supported in the old GOPATH mode because previously the Go toolchain was only able to install from the master branch of a particular remote dependency in a Git repository. The module path github.com/hashicorp/hcl/v2 is the Go Modules way to specify using the second major version of HCL, whereas github.com/hashicorp/hcl is the first major version.
In modules mode, it should be sufficient to just run go build -o terraform-provider-example (or, if you prefer, go install) and it will automatically fetch the dependencies to the local modules cache and use them from there. In modules mode, go get is for changing the dependencies recorded in go.mod rather than for installing existing dependencies.
I'm using Go modules in my project and in my build system (e.g. Travis CI) I'm downloading a command-line utility (written in Go) with go get to assist with my build process, e.g.:
go get github.com/mitchellh/gox
However, this go get causes the file to be added to my go.mod file. This is contaminating the build environment, causing it to be "dirty" (since there are changes to some files tracked in git, in this case go.mod and go.sum), and I use git describe --always --dirty --tag to describe my build, which shows up as "dirty".
Is there a way to "go get" a binary just to download it, without adding it to the go.mod/go.sum?
I've tried setting GOPATH to somewhere else, even then, go get updates the go.mod/go.sum to add this as an // indirect dependency.
dir="$(mktemp -d)"; \
env GOPATH="$dir" go get github.com/mitchellh/gox && \
mv "$dir/bin/gox" "$(go env GOPATH)"/bin/gox
Go 1.16 onwards
Go 1.16 (released February 2021) includes a change that makes it possible to install a binary without affecting go.mod.
Issue 40276 tracks the proposal:
cmd/go: 'go install' should install executables in module mode outside a module
This was implemented in CL 254365. As part of this change, you can run e.g.:
go install golang.org/x/tools/cmd/goimports#latest
to install a binary without affecting go.mod.
To install a specific version, replace #latest with e.g. #v0.1.5.
Hopefully in Go 1.14 there will be a new flag for go get that does exactly what you are asking. This is tracked in issue #30515 "cmd/go: offer a consistent global install command".
Prior to that, you have a few different options.
Go 1.12 and 1.13: change directory
If you are using Go 1.12 or later, the simplest solution is probably to move outside your current module to a directory without a go.mod prior to doing the go get, such as:
$ cd /tmp
$ go get github.com/foo/bar#v1.2.3
$ cd - # return to prior directory
Go 1.11, 1.12, 1.13+: gobin
gobin is a module-aware command to install or run binaries that provides additional flexibility, including the ability to install without altering your current module's go.mod. See the gobin README and FAQ for more details.
Go 1.11: temporary module
If you are using Go 1.11 with modules, the first step is probably to upgrade to Go 1.12 or 1.13 given there are many improvements in modules. If you are required to use Go 1.11 and want to use the #version syntax without updating your current module's go.mod, then one approach is to create a temporary module:
cd $(mktemp -d) && go mod init tempmod && go get github.com/foo/bar#v1.2.3
This is because in Go 1.11, you can't use the #version syntax unless you are in a module, which was relaxed in Go 1.12. This approach has been automated by a simple shell script by #rogpeppe.
Additional Details
In general, the go command in module-module always determines what module it is "in", which is based on the current working directory when you invoke the go command. (You could make an analogy to how make without any args will look for a makefile in the current working directory, or how historically go build without any args will build the current working directory, etc.).
With modules, go get looks for a go.mod file in the current working directory or any of its parents, and go get will use the constraints listed in any go.mod as part of solving for versions, as well as update the go.mod if needed based on doing the go get. That is why your go.mod file is updated if you run go get from within an existing module.
On the other hand, starting with Go 1.12, if you are in a directory that is not part of any module (that is, the directory does not have a go.mod, nor do any of its parents), then there is no go.mod to update, but the go command is still able to operate in module mode and use the #version syntax.
From the Go 1.12 release notes:
When GO111MODULE is set to on, the go command now supports module-aware operations outside of a module directory, provided that those operations do not need to resolve import paths relative to the current directory or explicitly edit the go.mod file. Commands such as go get, go list, and go mod download behave as if in a module with initially-empty requirements. In this mode, go env GOMOD reports the system's null device (/dev/null or NUL).
Per go help build:
The -mod build flag provides additional control over updating and use of go.mod.
If invoked with -mod=readonly, the go command is disallowed from the implicit automatic updating of go.mod
I'm using go 1.11 with module support. I understand that the go tool now installs dependencies automatically on build/install. I also understand the reasoning.
I'm using docker to build my binaries. In many other ecosystems its common to copy over your dependency manifest (package.json, requirements.txt, etc) and install dependencies as a separate stage from build. This takes advantage of docker's layer caching, and makes rebuilds much faster since generally code changes vastly outnumber dependency changes.
I was wondering if vgo has any way to do this?
It was an issue #26610, which is fixed now.
So now you can just use:
go mod download
For this to work you need just the go.mod / go.sum files.
For example, here's how to have a cached multistage Docker build: (source)
FROM golang:1.17-alpine as builder
RUN apk --no-cache add ca-certificates git
WORKDIR /build
# Fetch dependencies
COPY go.mod go.sum ./
RUN go mod download
# Build
COPY . ./
RUN CGO_ENABLED=0 go build
# Create final image
FROM alpine
WORKDIR /
COPY --from=builder /build/myapp .
EXPOSE 8080
CMD ["./myapp"]
Also see the article Containerize Your Go Developer Environment – Part 2, which describes how to leverage the Go compiler cache to speed up builds even further.
You may use the go mod vendor command which will create a vendor folder in the main module's root folder, and copy all dependencies into it. After this you may pass the -mod=vendor param to the go tool, and then dependencies from the vendor folder will be used to build / compile / test your app.
So what you may do to speed up your builds:
Run the go mod vendor command to have an actual version of your dependencies.
Save / cache this vendor folder.
During builds, restore this vendor folder, and build / install your app by passing the -mod=vendor argument to the go tool, so no dependencies will be downloaded, but the content of the vendor folder will be used.
Quoting from go help mod:
Modules and vendoring
When using modules, the go command completely ignores vendor directories.
By default, the go command satisfies dependencies by downloading modules
from their sources and using those downloaded copies (after verification,
as described in the previous section). To allow interoperation with older
versions of Go, or to ensure that all files used for a build are stored
together in a single file tree, 'go mod vendor' creates a directory named
vendor in the root directory of the main module and stores there all the
packages from dependency modules that are needed to support builds and
tests of packages in the main module.
To build using the main module's top-level vendor directory to satisfy
dependencies (disabling use of the usual network sources and local
caches), use 'go build -mod=vendor'. Note that only the main module's
top-level vendor directory is used; vendor directories in other locations
are still ignored.
I wanted to re-download all the dependencies using go mod, this is what I did:
Go to your GOROOT
sudo rm -rf pkg/mod/
Go to the directory where the go.mod file exists
go mod download
You can use a package manager, There are many of them like dep, glide, and govendor. dep is newer and is going to be integrated into go toolchain as official dependency management tool.
We also make docker images for go applications and we use dind to make those images and we prepared a CI/CD image with all dependencies preinstalled to make the builds faster. Though, it took a little bit of scripting to glue everything together.
Moreover, layering up the dependencies could result in big size of docker images. I suggest try multi-stage builds which could help making images super lite.