I am trying to download, make some tweaks and build a golang project from GitHub. The project's instructions are:
go get github.com/<vendor>/<projectName>
cd src/github.com/<vendor>/<projectName>
go build .
That used to work in the past — before enabling Go Modules.
Now I have GO111MODULE=on (go version go1.15.4 linux/amd64). When running the first command, go downloads the project as a module and all its dependencies.
But then there is no src/github.com/<vendor>/<projectName> folder anymore. Moreover, the is no folder named <projectName> anywhere in the system.
Instead, there is folder pkg/mod/github.com/<vendor> which contains the project folder with weird symbols in its name (exclamation marks etc.) and version identifiers.
How do I get the project folder available for making tweaks and builds?
As pointed by #Volker, good old git clone should be used.
It turns out that it should be used instead of go get github.com/<vendor>/<projectName> (no idea why the project vendor recommends that):
git clone git://github.com/<vendor>/<projectName>
cd <projectName>
go get ./...
# do tweaks here
go build .
If your goal is tweaks, the easiest way it use to use go mod vendor.
https://golang.org/ref/mod#go-mod-vendor
The go mod vendor command constructs a directory named vendor in the main module's root directory that contains copies of all packages needed to support builds and tests of packages in the main module
Related
What is the golang command that is equivalent to npm install
npm install downloads all the dependencies listed in the package.json file.
Having said that, what is the command that downloads all the dependencies in the go.mod file?
If you have only a go.mod and you have Go 1.16 or later:
If you just want to run your code, use go build or go run . - your dependencies will be downloaded and built automatically
If you want to save a copy of your dependencies locally, use go mod vendor
Both the above will create a go.sum file (this is maintained by the Go Tools - you can ignore it)
The vendor command will create a vendor folder with a copy of all the source code from your dependencies. Note: If you do use the vendor approach, you will need to run go mod vendor if there is a change in your dependencies, so that a copy is downloaded to the vendor folder. The advantage is your code will build without an internet connection. The disadvantage is you need to keep it up to date.
That should get you started for every day use.
If you'd like to know all about modules, this is a good source.
Setup:
Our project is using golang 1.12.14
We are using go build -mod=vendor
Issue: When new dependencies are added to go.mod the vendor folder isn't updated and people are committing code and forgetting to run go mod vendor to update the folder. My understanding is that since -mod=vendor specifies to use packages from the vendor folder, the go.mod file will have discrepancies from what we are actually using when building the project.
Question: Should go mod vendor be added to a pre-commit hook?
As of Go 1.14, the go command automatically checks for consistency between the vendor directory and the go.mod file whenever the vendor directory is used. In addition, it uses the vendor directory by default if the module specifies go 1.14 or higher (see https://tip.golang.org/doc/go1.14#go-command).
As of today, the oldest supported version of the Go toolchain is Go 1.15.13.
So if you upgrade to a supported version of the Go toolchain, it should not be necessary to run go mod vendor as a pre-commit hook. The go command itself will flag inconsistencies whenever the vendor directory is used.
Say my Go project depends on package example.com/foo. I am using Go 1.12, so the dependency is automatically pulled in by Go modules. That dependency has some problems/bugs, I want to add logs in the source code.
I can find the source code of the dependency on GitHub but I don't know how to make it into my project for debugging purpose.
First fetch all the dependency packages into the vendor folder.
go mod vendor
Then, change the source code in that and build your project by specifying to look into vendor folder.
go build -mod=vendor
or
go run -mod=vendor myapp.go
You can use replace directive:
replace example.com/original/import/path => /your/forked/import/path
Go module fetches packages into $GOPATH/pkg/mod you can change the source code there or using the vendor option of go mod to pull the packages into vendor folder and then start coding there.
I have two projects:
/myproject
/sharedproject
both of them are managed by dep, I have execute go get -u github.com/golang/dep/cmd/dep in order to have latest dep version, and run dep ensure on both projects.
When I run myproject I get following error:
cannot use op (type *"myproject/vendor/github.com/go-openapi/spec".Operation) as type *"sharedproject/vendor/github.com/go-openapi/spec".Operation
What is wrong and how to fix this?
Looks like the situation is that sharedproject vendors the github.com/go-openapi/spec dependency while myproject
gets both sharedproject and github.com/go-openapi/spec dependencies from
GOPATH.
Now when you refer to github.com/go-openapi/spec in sharedproject, it refers to
the package inside the vendor directory, which is technically different from the
same package in GOPATH, even if both have the same content. So when you pass a
variable of type *github.com/go-openapi/spec.Operation from myproject to a
function in sharedproject, the package of the type differs from what's expected and compilation fails.
To solve this, make sure sharedproject is vendored inside myproject. When
you do this, dep ensure will put a copy of sharedproject without its vendor
directory into myproject's vendor directory. After this, both myproject and sharedproject will use the github.com/go-openapi/spec package from myproject's vendor directory.
That does make local development hard if you change sharedproject often and want
to immediately use those changes in myproject (can't use dep till the changes are pushed to the Git remote). I'd work around that by copying
over sharedproject into myproject's vendor directory manually without using dep (excluding the vendor directory of course). Be careful to not commit those manually copied changes to Git though!
I'm using go 1.11 with module support. I understand that the go tool now installs dependencies automatically on build/install. I also understand the reasoning.
I'm using docker to build my binaries. In many other ecosystems its common to copy over your dependency manifest (package.json, requirements.txt, etc) and install dependencies as a separate stage from build. This takes advantage of docker's layer caching, and makes rebuilds much faster since generally code changes vastly outnumber dependency changes.
I was wondering if vgo has any way to do this?
It was an issue #26610, which is fixed now.
So now you can just use:
go mod download
For this to work you need just the go.mod / go.sum files.
For example, here's how to have a cached multistage Docker build: (source)
FROM golang:1.17-alpine as builder
RUN apk --no-cache add ca-certificates git
WORKDIR /build
# Fetch dependencies
COPY go.mod go.sum ./
RUN go mod download
# Build
COPY . ./
RUN CGO_ENABLED=0 go build
# Create final image
FROM alpine
WORKDIR /
COPY --from=builder /build/myapp .
EXPOSE 8080
CMD ["./myapp"]
Also see the article Containerize Your Go Developer Environment – Part 2, which describes how to leverage the Go compiler cache to speed up builds even further.
You may use the go mod vendor command which will create a vendor folder in the main module's root folder, and copy all dependencies into it. After this you may pass the -mod=vendor param to the go tool, and then dependencies from the vendor folder will be used to build / compile / test your app.
So what you may do to speed up your builds:
Run the go mod vendor command to have an actual version of your dependencies.
Save / cache this vendor folder.
During builds, restore this vendor folder, and build / install your app by passing the -mod=vendor argument to the go tool, so no dependencies will be downloaded, but the content of the vendor folder will be used.
Quoting from go help mod:
Modules and vendoring
When using modules, the go command completely ignores vendor directories.
By default, the go command satisfies dependencies by downloading modules
from their sources and using those downloaded copies (after verification,
as described in the previous section). To allow interoperation with older
versions of Go, or to ensure that all files used for a build are stored
together in a single file tree, 'go mod vendor' creates a directory named
vendor in the root directory of the main module and stores there all the
packages from dependency modules that are needed to support builds and
tests of packages in the main module.
To build using the main module's top-level vendor directory to satisfy
dependencies (disabling use of the usual network sources and local
caches), use 'go build -mod=vendor'. Note that only the main module's
top-level vendor directory is used; vendor directories in other locations
are still ignored.
I wanted to re-download all the dependencies using go mod, this is what I did:
Go to your GOROOT
sudo rm -rf pkg/mod/
Go to the directory where the go.mod file exists
go mod download
You can use a package manager, There are many of them like dep, glide, and govendor. dep is newer and is going to be integrated into go toolchain as official dependency management tool.
We also make docker images for go applications and we use dind to make those images and we prepared a CI/CD image with all dependencies preinstalled to make the builds faster. Though, it took a little bit of scripting to glue everything together.
Moreover, layering up the dependencies could result in big size of docker images. I suggest try multi-stage builds which could help making images super lite.