Go dep's dep ensure command will remove packages not currently in use. There's one particular package we use for debugging github.com/sanity-io/litter. The challenge we're facing is if we run dep ensure outside of a debugging session, dep will remove that package.
One solution could be to call that package in some backstage spot in the code that won't bother anyone, thereby showing dep that we are, in fact, using this package. But that sounds ugly, hacky, and could get removed by a future developer on the team.
So, the question is, how to tell dep to keep a package, even if it's not currently in use?
Add to the beginning of Gopkg.toml:
required = ["github.com/sanity-io/litter"]
The Gopkg.toml docs state about required:
Use this for: linters, generators, and other development tools that
Are needed by your project
Aren't imported by your project, directly or transitively
You don't want to put them in your GOPATH, and/or you want to lock the version
Please note that this only pulls in the sources of these dependencies.
It does not install or compile them.
You should use required for your dependency, take a look at documentation
about it. And maybe more useful link about required section.
Related
I am using go modules on go1.12 to handle my Go dependencies. Is it best practice to also commit the vendor/ directory into version control?
This is somewhat related to Is it best-practice to commit the `vendor` directory? which asks this question in the case of using dep. With dep, commiting vendor/ is the only way to get truly reproducible builds. What about for go modules?
I'd like to give some arguments in favour of committing vendor, go.mod and go.sum.
I agree with the accepted answer's arguments that it's technically unnecessary and bloats the repo.
But here is a list of contra-arguments:
Building the project doesn't depend on some code being available on Github/Gitlab/... or the Go proxy servers. Open source projects may disappear because of censorship, authors incentives, licensing changes or some other reasons I can't currently think of, which did happen on npm, the JavaScript package manager, and broke many projects. Not in your repo, not your code.
We may have used internal or 3rd party Go modules (private) which may also disappear or become inaccessible, but if they are committed in vendor, they are part of our project. Nothing breaks unexpectedly.
Private Go modules may not follow semantic versioning, which means the Go tools will rely on the latest commit hash when fetching them on-the-fly. Repo history may be rewritten (e.g. rebase) and you, a colleague or your CI job may end up with different code for the dependencies they use.
Committing vendor can improve your code review process. Typically we always commit dependency changes in a separate commit, so they can be easily viewed if you're curious.
Here's an interesting observation related to bloating the repo. If I make code review and a team member has included a new dependency with 300 files (or updated one with 300 files changed), I would be pretty curious to deep dive into that and start a discussion about code quality, the need for this change or alternative Go modules. This may lead to actually decrease your binary size and overall complexity.
If I just see a single new line in go.mod in a new Merge Request, chances are I won't even think about it.
CI/CD jobs which perform compilation and build steps need not waste time and network to download the dependencies every time the CI job is executed. All needed dependencies are local and present (go build -mod vendor)
These are on top of my head, if I remember something else, I'll add it here.
Unless you need to modify the vendored packages, you shouldn't. Go modules already gives you reproducible builds as the go.mod file records the exact versions and commit hashes of your dependencies, which the go tool will respect and follow.
The vendor directory can be recreated by running the go mod vendor command, and it's even ignored by default by go build unless you ask it to use it with the -mod=vendor flag.
Read more details:
Go wiki: How do I use vendoring with modules? Is vendoring going away?
Command go: Modules and vendoring
Command go: Make vendored copies of dependencies
I have a project with mutliple main methods.
When running go build program1/main1.go which has a different set of dependencies than program2/main2.go, my first go build seems to alter my go.mod file and removes the dependencies it thinks it does not need. Yet main2 would need these dependencies.
I've tried using go build ... but that also created a different set of dependencies. Specifically, it seems like all the //indirect dependencies get removed and cause program2 to fail.
Is there a way of running go build or go run without updating the go.mod file? Using go build -mod=readonly program1/main1.go it tells me that it fails because the dependencies need to be updated..
I believe you're looking for submodules. See this walktrhough.
TLDR: you'll want a separate go.mod in each of your tools's cmd dir, and you can use the replace directive to point dependncies from these tools to your local module.
This Go Issue and others linked from it suggest that figuring out "the one right way" to do this is still WIP, though I think your use case is simple enough.
Using submodules is a way to nest multiple Go module projects you can edit.
But Go 1.18 might include the notion of Go workspace, which means you don't need submodules anymore: one Go project can include multiple modules you can edit.
See golang/go issue 45713: "proposal: cmd/go: add a workspace mode " and its design document.
Background
Users often want to make changes across multiple modules: for instance, to introduce a new interface in a package in one module along with a usage of that interface in another module.
Normally, the go command recognizes a single "main" module the user can edit.
Other modules are read-only and are loaded from the module cache.
The go mod replace directive is the exception: it allows users to replace the resolved version of a module with a working version on disk.
But working with the replace directive can often be awkward: each module developer might have working versions at different location on disk, so having the directive in a file that needs to be distributed with the module isn't a good fit for all use cases.
Proposal
This proposal describes a new workspace mode in the go command for editing multiple modules.
The presence of a go.work file in the working directory or a containing directory will put the go command into workspace mode.
The go.work file specifies a set of local modules that comprise a workspace. When invoked in workspace mode, the go command will always select these modules and a consistent set of dependencies.
Main modules: The module the user is working in.
Before this proposal, this is the single module containing the directory where the go command is invoked. This module is used as the starting point when running MVS.
This proposal proposes allowing multiple main modules.
See for instance CL 334934 (CL = Change List)
[dev.cmdgo] cmd/go: add the workspace mode
This change adds the outline of the implementation of the workspace mode.
The go command will now locate go.work files, and read them to determine
which modules are in the workspace.
It will then put those modules in the root of the workspace when building the build list.
It supports building, running, testing, and listing in workspaces.
You can initiate a multiple-module project with go mod initwork
Again, this is not before Go 1.18 (Q1 2022) and will probably be opt-in in Go 1.19 (Q3 2022).
I am using GoDep to resolve a project dependencies.
My problem is that repositories for dependencies maight be removed and my project wouldn't build.
I am trying to find any solution to store dependencies at Artifactory or another solution.
Please advice.
Regards.
Okay so GoDeps may be the standard way of doing this, but I usually found it a bit complicated. In my opinion, use a Makefile which sets a custom GoPath and just include dependencies with your code (remove their .git folder). This way the version freezes and no one needs to do a godep restore or something similar.
You can make recipes like make deploy that builds your code, runs GoFmt, cleans the pkg files, installs it to your custom GoPath bin/ and then you just go and run the binary.
You can have another one like make install that will install any missing dependencies.
I've managed to create a watch using this on my Makefile to keep on looking for changes on a linux based system using inotify-tools and call rebuild.
Internally all commands will be using standard go commands but you'll get rid of the GoDeps and maintaining JSON. To upgrade a dependency, it may be a bit of a problem as you'd have to manually copy the whole directory into your custom path and remove the .git/ folder.
Our company uses this method and seems to work quite nice for us.
Plus this method basically gets you away the $GOPATH/src/github.com/repoName/ kind of paths.
If i seem unclear, let me know, I'll add a gist on github.
I've read a whole bunch of articles and SO questions on importing 3rd party go packages which all seems straight forward, but what I don't understand is that none that I have read make any references to versioning. In Dartlang there's the pubspec file that defines your package including its version and its dependencies including their required versions. What if I do a go get github.com/gorilla/sessions and write my app then 6 months later I have to clear my directories and re get everything again, in which time that package has been update and broken backwards compatibility with my code that was using the older version?
The official version, from the GO FAQ:
If you're using an externally supplied package and worry that it might change in unexpected ways, the simplest solution is to copy it to your local repository. (This is the approach Google takes internally.) Store the copy under a new import path that identifies it as a local copy.
There are many alternative to that approach, mainly based on declaring the exact version of those projects you are using.
See for instance "Dead Simple Dependencies in Go -- Keep it simple and keep your sanity." (based on emil2k/vend)
The main different options for Go Dependency Management are listed at:
"Go Package Management -- A summary of dependency management in Go"
(And its associate GOPM mailing list)
Update July 2015:
the official vendoring approach from Go team is discussed here.
an alternative go build tool called "gb" is proposed at getgb.io by Dave Cheney.
Update Q4 2017: as mentioned below, go dep is the official tool for pinning version of dependencies (even though that pinning approach is not without criticism: see "The cargo cult of versioning").
It should be merged into the toolchain when Go 1.10 development begins, according to its roadmap.
Update Q2 2018: go dep has been replaced by go mod (modules) in Go 1.11, following works on vgo.
I use dep as a dependency management tool for golang project. Please use the following link dep tool for more info.
dep is a dependency management tool for Go. It requires Go 1.9 or newer to compile.
dep was the "official experiment." The Go toolchain, as of 1.11, has (experimentally) adopted an approach that sharply diverges from dep. As a result, we are continuing development of dep, but gearing work primarily towards the development of an alternative prototype for versioning behavior in the toolchain.
Current status: January 2019
dep is safe for production use.
By default, Go pulls imported dependencies by grabbing the latest version in master (github) or default (mercurial) if it cannot find the dependency on your GOPATH. And while this workflow is quite simple to grasp, it has become somewhat difficult to tightly control. Because all software change incurs some risk, I'd like to reduce the risk of this potential change in a manageable and repeatable way and avoid inadvertently picking up changes of a dependency, especially when running clean builds via CI server or preparing to deploy.
What is the most effective way I can pin (i.e. lock down or capture) a package dependency so I don't find myself unable to reproduce an old package, or even worse, unexpectedly broken when I'm about to release?
---- Update ----
Additional info on the Current State of Go Packaging. While I ended up (as of 7.20.13) capturing dependencies in a 3rd party folder and managing updates (ala Camlistore), I'm still looking for a better way...
Here is a great list of options.
Also, be sure to see the go 1.5 vendor/ experiment to learn about how go might deal with the problem in future versions.
You might find the way Camlistore does it interesting.
See the third party directory and in particular the update.pl and rewrite-imports.sh script. These scripts update the external repositories, change imports if necessary and make sure that a static version of external repositories is checked in with the rest of the camlistore code.
This means that camlistore has a completely repeatable build as it is self contained, but the third party components can be updated under the control of the camlistore developers.
There is a project to help you in managing your dependencies. Check gopack
godep
I started using godep early last year (2014) and have been very happy with it (it met the concerns I mentioned in my original question). I am no longer using custom scripts to manage the vendoring of dependencies as godep just takes care of it. It has been excellent for ensuring that no drift is introduced regardless of timing or a machine's package state. It works with the existing mechanism of go get and introduces the ability to pin (godep save) and restore (godep restore) based on Godeps/godeps.json.
Check it out:
https://github.com/tools/godep
There is no built in tooling for this in go. However you can fork the dependencies yourself either on local disk or in a cloud service and only merge in upstream changes once you've vetted them.
The 3rd party repositories are completely under your control. 'go get' clones tip, you're right, but you're free to checkout any revision of the cloned-by-go-get or cloned-by-you repository. As long as you don't do 'go get -u', nothing touches your 3rd party repositories already sitting at your hard disk.
Effectively, your external, locally cloned, dependencies are always locked down by default.