I am using go modules on go1.12 to handle my Go dependencies. Is it best practice to also commit the vendor/ directory into version control?
This is somewhat related to Is it best-practice to commit the `vendor` directory? which asks this question in the case of using dep. With dep, commiting vendor/ is the only way to get truly reproducible builds. What about for go modules?
I'd like to give some arguments in favour of committing vendor, go.mod and go.sum.
I agree with the accepted answer's arguments that it's technically unnecessary and bloats the repo.
But here is a list of contra-arguments:
Building the project doesn't depend on some code being available on Github/Gitlab/... or the Go proxy servers. Open source projects may disappear because of censorship, authors incentives, licensing changes or some other reasons I can't currently think of, which did happen on npm, the JavaScript package manager, and broke many projects. Not in your repo, not your code.
We may have used internal or 3rd party Go modules (private) which may also disappear or become inaccessible, but if they are committed in vendor, they are part of our project. Nothing breaks unexpectedly.
Private Go modules may not follow semantic versioning, which means the Go tools will rely on the latest commit hash when fetching them on-the-fly. Repo history may be rewritten (e.g. rebase) and you, a colleague or your CI job may end up with different code for the dependencies they use.
Committing vendor can improve your code review process. Typically we always commit dependency changes in a separate commit, so they can be easily viewed if you're curious.
Here's an interesting observation related to bloating the repo. If I make code review and a team member has included a new dependency with 300 files (or updated one with 300 files changed), I would be pretty curious to deep dive into that and start a discussion about code quality, the need for this change or alternative Go modules. This may lead to actually decrease your binary size and overall complexity.
If I just see a single new line in go.mod in a new Merge Request, chances are I won't even think about it.
CI/CD jobs which perform compilation and build steps need not waste time and network to download the dependencies every time the CI job is executed. All needed dependencies are local and present (go build -mod vendor)
These are on top of my head, if I remember something else, I'll add it here.
Unless you need to modify the vendored packages, you shouldn't. Go modules already gives you reproducible builds as the go.mod file records the exact versions and commit hashes of your dependencies, which the go tool will respect and follow.
The vendor directory can be recreated by running the go mod vendor command, and it's even ignored by default by go build unless you ask it to use it with the -mod=vendor flag.
Read more details:
Go wiki: How do I use vendoring with modules? Is vendoring going away?
Command go: Modules and vendoring
Command go: Make vendored copies of dependencies
Related
Go dep's dep ensure command will remove packages not currently in use. There's one particular package we use for debugging github.com/sanity-io/litter. The challenge we're facing is if we run dep ensure outside of a debugging session, dep will remove that package.
One solution could be to call that package in some backstage spot in the code that won't bother anyone, thereby showing dep that we are, in fact, using this package. But that sounds ugly, hacky, and could get removed by a future developer on the team.
So, the question is, how to tell dep to keep a package, even if it's not currently in use?
Add to the beginning of Gopkg.toml:
required = ["github.com/sanity-io/litter"]
The Gopkg.toml docs state about required:
Use this for: linters, generators, and other development tools that
Are needed by your project
Aren't imported by your project, directly or transitively
You don't want to put them in your GOPATH, and/or you want to lock the version
Please note that this only pulls in the sources of these dependencies.
It does not install or compile them.
You should use required for your dependency, take a look at documentation
about it. And maybe more useful link about required section.
I am using dep to handle my Go dependencies. Is it best practice to also commit the vendor directory into version control? Or is best practice to always execute dep ensure after checking out a repository?
The dep tool's FAQ answers this:
Should I commit my vendor directory?
It's up to you:
Pros
It's the only way to get truly reproducible builds, as it guards
against upstream renames, deletes and commit history overwrites.
You don't need an extra dep ensure step to sync vendor/ with
Gopkg.lock after most operations, such as go get, cloning, getting
latest, merging, etc.
Cons
Your repo will be bigger, potentially a lot bigger, though prune can help minimize this problem.
PR diffs will include changes for files under vendor/ when Gopkg.lock is modified, however files in vendor/ are hidden by default on GitHub.
imagine what would happen to your project if a dependency was taken offline by the author. Until Go has a central server to hold all packages which are unable to be deleted a lot of people will always see the need to commit the vendor folder
I don’t commit for well available sources. Because on this case many commit messages are bloated with changes in vendors. When I want update then I do it and then commit updated Gopkg.*.
What are some good patterns for using go in concourse-ci tasks. For example, should I build files locally with all dependencies and check-in cross-compiled binaries to the repo? Should I build on concourse prior to running the task?
Examples of what people do here would be great. Public repos of pipelines/tasks even better.
The way I see it there are currently 3 options for handling go builds:
Use vendoring
Explicitly declare the dependencies as concourse resources
Maintain a docker image with the required dependencies already included
All options have pros and cons. The first option is currently my favorite since the responsibility for handling dependencies is up to the project maintainers and there is a very clear way to see which versions/revisions of the dependencies are being used - i.e. just check the vendoring tool config - but it does force you to have all dependency code in the project's repo.
The second option follows the go "philosophy" of always tracking master, but it may lead to slower builds (concourse will need to check every single resource regularly) and may lead to sudden breakage because of changes in dependencies.
The third option allows you to implicitly fix the revision of the dependencies in the docker image, in that respect it's similar to the first, however it requires maintaining docker images (doesn't necessarily mean 1 per project, but it might mean more than one depending on the number of projects that use this option and if there are conflicting dependency versions between them)
Currently, we have all vendored libraries in src/vendor which makes docker-compose build quite fast. Although adding vendored libraries to source control has the disavantage of libraries not being updated and also heavily polluting the diff of pull requests.
Is there a way in between, maybe with caching?
Is there a way in between, maybe with caching?
Yes, several. But don't fight the system/preferred method.
Use $GOPATH/src/MyProject/vendor like you are already doing.
adding vendored libraries to source control has the disavantage of libraries not being updated...
That all depends on your team's management of your repo. If everyone ignores the vendor, ya it will get stale.
Personally I make it a "1st of the month" habit of going through and refreshing all dependencies, running our test suites, and if no errors update for QA integration testing on the dev server and keep an eye on the error logs after release. Tools like godep and gostatus greatly help keep your GOPATH in chrcn with latest, that you can update your vendor folder(s) with quickly.
Just make sure it is a dedicated commit, so it can be reverted in a hurry if an issue creeps up.
also heavily polluting the diff of pull requests
First of all, that's just a process task. I enforce rebasing on all pull requests and reject all merges in all repos. This keeps a very clean git history; but, more to the point, rebasing moves your local commits until after the vendor updates. Shouldn't ever get a conflict unless someone added the same package. Which at that point is easy, just take the latest one and be done.
Sound like there are process issues to work out than worrying about /vendor management.
By default, Go pulls imported dependencies by grabbing the latest version in master (github) or default (mercurial) if it cannot find the dependency on your GOPATH. And while this workflow is quite simple to grasp, it has become somewhat difficult to tightly control. Because all software change incurs some risk, I'd like to reduce the risk of this potential change in a manageable and repeatable way and avoid inadvertently picking up changes of a dependency, especially when running clean builds via CI server or preparing to deploy.
What is the most effective way I can pin (i.e. lock down or capture) a package dependency so I don't find myself unable to reproduce an old package, or even worse, unexpectedly broken when I'm about to release?
---- Update ----
Additional info on the Current State of Go Packaging. While I ended up (as of 7.20.13) capturing dependencies in a 3rd party folder and managing updates (ala Camlistore), I'm still looking for a better way...
Here is a great list of options.
Also, be sure to see the go 1.5 vendor/ experiment to learn about how go might deal with the problem in future versions.
You might find the way Camlistore does it interesting.
See the third party directory and in particular the update.pl and rewrite-imports.sh script. These scripts update the external repositories, change imports if necessary and make sure that a static version of external repositories is checked in with the rest of the camlistore code.
This means that camlistore has a completely repeatable build as it is self contained, but the third party components can be updated under the control of the camlistore developers.
There is a project to help you in managing your dependencies. Check gopack
godep
I started using godep early last year (2014) and have been very happy with it (it met the concerns I mentioned in my original question). I am no longer using custom scripts to manage the vendoring of dependencies as godep just takes care of it. It has been excellent for ensuring that no drift is introduced regardless of timing or a machine's package state. It works with the existing mechanism of go get and introduces the ability to pin (godep save) and restore (godep restore) based on Godeps/godeps.json.
Check it out:
https://github.com/tools/godep
There is no built in tooling for this in go. However you can fork the dependencies yourself either on local disk or in a cloud service and only merge in upstream changes once you've vetted them.
The 3rd party repositories are completely under your control. 'go get' clones tip, you're right, but you're free to checkout any revision of the cloned-by-go-get or cloned-by-you repository. As long as you don't do 'go get -u', nothing touches your 3rd party repositories already sitting at your hard disk.
Effectively, your external, locally cloned, dependencies are always locked down by default.