Intro: dep is a tool to manage dependencies of Go projects. bazel is a build tool that produces stable, reproducable builds. There is a bazel rule set for Go projects, including an automatic build-file generator, gazelle, that generates Bazel build files following the conventions that go build would apply, using the dependencies declared in the bazel WORKSPACE.
The WORKSPACE, though, I have to create by hand or by help of a little hackish helper tool, wtool. dep, on the other hand, helps track dependencies, is able to download them into a vendor/ directory and locking specific versions.
To me it appears that the Gopkg.lock file that dep uses to specify versions of dependencies (usually by their VCS commit hash) would be easily translated into a bazel WORKSPACE file. I have successfully done so manually in a company-internal project as well.
Is there an existing link between bazel and its Go rules and dep that I could have used instead of doing this manually?
(Additionally: Is anybody else doing this? Should I be doing this at all?)
There's a pretty new, and actively-being-developed project for this: https://github.com/scele/rules_go_dep.
There's no existing tool for this, but it should be pretty straightforward to convert entries in the lock file to WORKSPACE.
We're thinking of a command to the Gazelle build file generator to help with this (not just for dep, but other tools as well). The main plan is for Gazelle to automatically add repositories that are missing, but that will take more time to implement.
Related
I can't really grasp the purpose of having a vendor folder. Based on what I learned, it seems the vendor folder is only beneficial if you're trying to make your repo compatible with golang versions earlier than 1.11. We are running golang 1.12.14.
When I brought this up to my coworker he said:
Please use vendor with modules - go doesn't have a global artifactory. this is, currently, the best option to make sure you have hermetic builds and your code doesn't break when somebody changes something in their repo.
I thought this is what Go modules does? I asked this question and a commenter is saying I shouldn't use vendor? Does it make sense to add `go mod vendor` to a pre-commit hook?
Go modules bring the guarantee that you will be able to build your packages deterministically by locking down the dependencies into a go.sum. That being said, the promise to deterministically build your project only stands if your dependencies are still accessible in the future. You don't know if this is going to be the case.
Vendoring on the other hand, with or without Go modules, brings stronger guarantees as it enables to commit the dependencies next to the code. Thus even if the remote repository is no longer accessible (deleted, renamed, etc), you will still be able to build your project.
Another alternative is to use Go modules along with a proxy. You can find more information in the official documentation. You can also look at some OSS implementations like gomods/athens or goproxy/goproxy. If you don't feel like setting up and maintaining your own proxy, some commercial offers are available on the market.
So should you go mod vendor each time you commit? Well it's ultimately up to you dependending on the kind of guarantees you want. But yes leveraging a proxy or vendoring your dependencies help getting closer to reproducable builds.
Note: with Go 1.17, go mod vendor (from 1.17 Go commands) might be easier to use:
vendor contents
If the main module specifies go 1.17 or higher, go mod vendor now annotates vendor/modules.txt with the go version indicated by each vendored module in its own go.mod file.
The annotated version is used when building the module's packages from vendored source code.
If the main module specifies go 1.17 or higher, go mod vendor now omits go.mod and go.sum files for vendored dependencies, which can otherwise interfere with the ability of the go command to identify the correct module root when invoked within the vendor tree.
Vendor Folder is a great way to organize and manage third-party dependencies in your project. It is especially useful when your code relies on external libraries or frameworks.
Benefits of having a Vendor Folder:
It helps to reduce dependencies conflicts.
It allows you to keep a separate version of each library / framework installed in your project.
It helps to keep the project structure clean and organized.
It makes it easy to update, install, and remove any dependencies with minimal effort.
It makes it easier to switch between different versions of a library or framework.
I'm "all in" on Go Modules. Mostly, I prefer the experience. In Go development, I've -- perhaps like many others -- treated dependencies as if I worked in a mono repo, each of my projects had its own GOPATH and I'd often clone from scratch and pull all then-latest versions of dependencies.
Using Modules, I think I'm breaking the best practice:
For per-commit builds, my projects' go.mod file would contain only primary -- and often only one -- explicit dependencies. Effectively, I don't commit go.mod and leave my build process to generate it and then the build. My thinking being that, apart from e.g. specific platforms that I'm using, where my familiarity with them means I'm confident in pinning to a specific version, for other dependencies, I'd rather maintain currency and get #vLatest.
If I get to building releases, I'd then go mod tidy and commit the go.mod to source control for the basis of the build.
Besides:
potentially breaking builds (which is acceptable for currency);
the absence of go.sum and package hashes (which I'm not independently verifying but trusting, e.g. golang.proxy.org); and
the repetition of pulling dependencies which is unavoidable anyway with my build process,
Is this approach bad?
For building releases, dependency immutability and build reproducibility are critical to software releases. Relying on go mod tidy to create the go.mod assumes the module git tag is immutable and is always available which is not the case. To ensure that the module tag is persistent and immutable, a go module repository is recommended. Refer to Go1.11 documentation for a list of "always on" module repositories and enterprise proxies. A short video on "Go Module and Dependency Management - GoCenter and Project Athens" talks about immutable dependency management..
I have started learning go (1.7.4) and have a project which currently produces two executables. I have a directory structure as below following the standard go layout:
GOPATH=`pwd`
bin
src/
src/<project1>
src/<project1>/vendor
src/<project1>/glide.yaml
src/<project2>
src/<project2>/vendor
src/<project2>/glide.yaml
pkg/
Project 1 and project 2 share a lot of dependencies.
Is there a way to share the vendor directory between project1 and project2 and still pin the versions to ensure reproducible builds?
I don't want to duplicate the glide.yaml and vendor directories for each project as it bloats the build and violates DRY.
The pkg directory is the obvious the way to do this but unlike vendor I don't have a dependency manager tool like glide to ensure a specific version is used (see also my related question).
A possibly related issue is how this project is organised. I believe in go it would be more conventional for each project sub-directory to map to a single github repository. However, for my project I want to build at least two executables. I realise you can do this by using different package names but it confuses go and glide. I wrestled with getting this to work under a single project and decided/discovered it was easier to use the standard go layout and work two levels up. For example, an advantage is that "go build" etc. in the subdirectories just works without having to name the package. I can also have my build, test and package machinery at the top level operate on all projects and keep my go environment separate from any others.
The programs are not complex enough to warrant separate git repositories (even as submodules). If there is a pattern that makes this work it might render my original question moot.
It should be possible to have a shared vendor directory. The way I am doing it involves Go 1.11 and the new Go feature called modules. But I am pretty sure it should work with vendor and tools like glide and dep. To use dep/glide your directory structure might looks like this
- src
- projects
- project1
- project2
- vendor
- Glide.yaml
And you can build it either from the projects folder using go build -o p1 project1/*.go or from individual project folder using go build
The same structure, but outside of GOPATH will work for Go 1.11 modules. You would have to set the GO111MODULE variable to "on" or "auto". Mind you that go modules store dependencies in some other location and download them automatically during the build process when needed.
Note: glide github page recommends switching to dep as the more official tool
Edit: Just tested it with dep. It works for me.
I recommend look at new vendoring system - https://github.com/golang/go/wiki/Modules
It allows you to fix versions of packages used:
module github.com/my/thing
require (
github.com/some/dependency v1.2.3
github.com/another/dependency/v4 v4.0.0
)
Is Go supported by Gradle? How to start?
I am starting my research, but so far I only find 1 plugin on https://plugins.gradle.org/search?term=go
https://github.com/echocat/gradle-golang-plugin
Generally, gradle or maven would not be needed, because:
go build is enough.
the dependencies are managed by go itself.
You see some project using Makefile (to link go build and go test and go vet).
Try Gogradle https://github.com/blindpirate/gogradle, a full featured plugin for Golang.
The reason why we need it is that Golang lacks of automatic build mechanism, package versioning and many other stuff which is supported by Gogradle.
I have many projects that use a bunch of exact same class.
Is there a way to add a script to Xcode, so, each time i compile, he go to a network folder and update is files from there... If newer. (i do this step manually, but could be great to automate it)
Thanks
You could add a "run script" build phase to copy over files before compiling if that's really what you want to do. That would catch updates for you but I don't think it would help you if new files are added (though copying them into a location your project has a folder reference rather than a group pointing to might work).
That said I think there's a better solution. It sounds like you're reinventing a process for managing project dependencies when you could use existing tools. I would publish those shared classes as a library and add it to each project using CocoaPods and a reference to the library's git repository. That way you just need to run a pod install to get the latest version of your library. A good dependency manager gives you a clear understanding of which version of your dependencies you're currently using, control over when to update them, handles installing dependencies of your dependencies, and will avoid link errors from multiple static libraries attempting to each include a copy of the same common dependency.