I have a project with mutliple main methods.
When running go build program1/main1.go which has a different set of dependencies than program2/main2.go, my first go build seems to alter my go.mod file and removes the dependencies it thinks it does not need. Yet main2 would need these dependencies.
I've tried using go build ... but that also created a different set of dependencies. Specifically, it seems like all the //indirect dependencies get removed and cause program2 to fail.
Is there a way of running go build or go run without updating the go.mod file? Using go build -mod=readonly program1/main1.go it tells me that it fails because the dependencies need to be updated..
I believe you're looking for submodules. See this walktrhough.
TLDR: you'll want a separate go.mod in each of your tools's cmd dir, and you can use the replace directive to point dependncies from these tools to your local module.
This Go Issue and others linked from it suggest that figuring out "the one right way" to do this is still WIP, though I think your use case is simple enough.
Using submodules is a way to nest multiple Go module projects you can edit.
But Go 1.18 might include the notion of Go workspace, which means you don't need submodules anymore: one Go project can include multiple modules you can edit.
See golang/go issue 45713: "proposal: cmd/go: add a workspace mode " and its design document.
Background
Users often want to make changes across multiple modules: for instance, to introduce a new interface in a package in one module along with a usage of that interface in another module.
Normally, the go command recognizes a single "main" module the user can edit.
Other modules are read-only and are loaded from the module cache.
The go mod replace directive is the exception: it allows users to replace the resolved version of a module with a working version on disk.
But working with the replace directive can often be awkward: each module developer might have working versions at different location on disk, so having the directive in a file that needs to be distributed with the module isn't a good fit for all use cases.
Proposal
This proposal describes a new workspace mode in the go command for editing multiple modules.
The presence of a go.work file in the working directory or a containing directory will put the go command into workspace mode.
The go.work file specifies a set of local modules that comprise a workspace. When invoked in workspace mode, the go command will always select these modules and a consistent set of dependencies.
Main modules: The module the user is working in.
Before this proposal, this is the single module containing the directory where the go command is invoked. This module is used as the starting point when running MVS.
This proposal proposes allowing multiple main modules.
See for instance CL 334934 (CL = Change List)
[dev.cmdgo] cmd/go: add the workspace mode
This change adds the outline of the implementation of the workspace mode.
The go command will now locate go.work files, and read them to determine
which modules are in the workspace.
It will then put those modules in the root of the workspace when building the build list.
It supports building, running, testing, and listing in workspaces.
You can initiate a multiple-module project with go mod initwork
Again, this is not before Go 1.18 (Q1 2022) and will probably be opt-in in Go 1.19 (Q3 2022).
Related
I am using go modules on go1.12 to handle my Go dependencies. Is it best practice to also commit the vendor/ directory into version control?
This is somewhat related to Is it best-practice to commit the `vendor` directory? which asks this question in the case of using dep. With dep, commiting vendor/ is the only way to get truly reproducible builds. What about for go modules?
I'd like to give some arguments in favour of committing vendor, go.mod and go.sum.
I agree with the accepted answer's arguments that it's technically unnecessary and bloats the repo.
But here is a list of contra-arguments:
Building the project doesn't depend on some code being available on Github/Gitlab/... or the Go proxy servers. Open source projects may disappear because of censorship, authors incentives, licensing changes or some other reasons I can't currently think of, which did happen on npm, the JavaScript package manager, and broke many projects. Not in your repo, not your code.
We may have used internal or 3rd party Go modules (private) which may also disappear or become inaccessible, but if they are committed in vendor, they are part of our project. Nothing breaks unexpectedly.
Private Go modules may not follow semantic versioning, which means the Go tools will rely on the latest commit hash when fetching them on-the-fly. Repo history may be rewritten (e.g. rebase) and you, a colleague or your CI job may end up with different code for the dependencies they use.
Committing vendor can improve your code review process. Typically we always commit dependency changes in a separate commit, so they can be easily viewed if you're curious.
Here's an interesting observation related to bloating the repo. If I make code review and a team member has included a new dependency with 300 files (or updated one with 300 files changed), I would be pretty curious to deep dive into that and start a discussion about code quality, the need for this change or alternative Go modules. This may lead to actually decrease your binary size and overall complexity.
If I just see a single new line in go.mod in a new Merge Request, chances are I won't even think about it.
CI/CD jobs which perform compilation and build steps need not waste time and network to download the dependencies every time the CI job is executed. All needed dependencies are local and present (go build -mod vendor)
These are on top of my head, if I remember something else, I'll add it here.
Unless you need to modify the vendored packages, you shouldn't. Go modules already gives you reproducible builds as the go.mod file records the exact versions and commit hashes of your dependencies, which the go tool will respect and follow.
The vendor directory can be recreated by running the go mod vendor command, and it's even ignored by default by go build unless you ask it to use it with the -mod=vendor flag.
Read more details:
Go wiki: How do I use vendoring with modules? Is vendoring going away?
Command go: Modules and vendoring
Command go: Make vendored copies of dependencies
I'm using Go 1.11 modules for multiple repositories at this point. Right now I'm dealing with one that was already at a 1.x version. In combination with moving to go modules, I made some other breaking changes, so it's clear it's time to increment the major version of the repository. This means going to version 2.
Per "go command" documentation:
To preserve import compatibility, the go command requires that modules with major version v2 or later use a module path with that major version as the final element. For example, version v2.0.0 of example.com/m must instead use module path example.com/m/v2, and packages in that module would use that path as their import path prefix, as in example.com/m/v2/sub/pkg. Including the major version number in the module path and import paths in this way is called "semantic import versioning".
Source
Is this as simple as updating the first line of my go.mod file, appending /v2 to the module name? Or should I be creating a v2/ directory in my repository and moving all files into there?
There are two options for how to move to version 2: branches and subdirectories. You can read more about them (with better illustrations) at https://research.swtch.com/vgo-module.
These two options are what make it possible for one version to depend on another. For example, when you implement version 2, you can update version 1 to depend on version 2 (but keep the same v1 API). Then, you only need to have one implementation of the logic for your library. This may or may not work well for you depending on the type of project, the support you want to provide, and the fixes it requires.
Branches
master: A -> B (v1.0.0) -> D (v1.0.1)
\
v2: -> C (v2.0.0)
In this scenario:
You start your project on the master branch,
Make some commits (A and B),
Tag v1.0.0.
You decide to make a breaking change. So, you create a new branch (git checkout -b v2) and make your breaking changes. Your go.mod must now be updated so the module name ends with /v2 (it's essentially a new module!).
You decide to fix a bug in v1, so you go back to that branch, make a new commit, and tag a new v1 version.
When a user requires a particular version of your module, go will look in the two branches for which one provides the right module.
Subdirectories
What if you don't want to develop on branches? You can create a subdirectory for each major version. Version 1 stays at the top level, then new versions move into subdirectories:
go.mod (module example.com/foo)
foo.go
v2/
go.mod (module example.com/foo/v2)
foo.go
When you tag this repo with new versions, v1 will use the top level version. v2 tags will use the v2 subdirectory.
From the Releasing Modules (v2 or Higher) section of the modules wiki:
There are two alternative mechanisms to release a v2 or higher module.
Note that with both techniques, the new module release becomes
available to consumers when the module author pushes the new tags.
Using the example of creating a v3.0.0 release, the two options are:
Major branch: Update the go.mod file to include a /v3 at the end of the module path in the module directive (e.g., module github.com/my/module/v3). Update import statements within the module
to also use /v3 (e.g., import "github.com/my/module/v3/mypkg").
Tag the release with v3.0.0.
Go versions 1.9.7+, 1.10.3+, and
1.11+ are able to properly consume and build a v2+ module created using this approach without requiring updates to consumer code that has not
yet opted in to modules (as described in the the "Semantic Import
Versioning"
section above).
Note: creating a new branch is not required. If instead you have been previously releasing on master, for example, and would prefer to tag v3.0.0 on master, that is a viable option as well. [...]
[...]
Major subdirectory: Create a new v3 subdirectory (e.g., my/module/v3) and place a new go.mod file in that subdirectory.
The module path must end with /v3. Copy or move the code into the
v3 subdirectory. Update import statements within the module to also
use /v3 (e.g., import "github.com/my/module/v3/mypkg"). Tag the
release with v3.0.0.
However, there are additional details in that section that are worth reviewing.
One point covered there that is worth mentioned here is that if you are interested in an automated approach (for example, perhaps you have many files you would need to visit), a good automated solution is https://github.com/marwan-at-work/mod, which can automatically add, remove, or change the required /vN in your *.go code and your go.mod. See this answer for more details.
When a private agent build starts in VSTS, it gets assigned a directory, e.g. C:\vstsagent_work\1\s
Is there a way to set this to a different path? On other CI servers, like Jenkins, I can define a custom workspace for a job. I'm dealing with a huge monorepo and have dozens of build definitions around the same repository. It makes sense (to me anyway) to share a single directory on the build agent computer.
The benefit to me is that my builds can use pre-built components from upstream repositories, if they have already been built.
Thanks for any help
VSTS build always creates a working directory per build definition. This leaves you two options:
Create a single build definition and use conditionals on steps to skip certain steps in order to only run what is needed. This allows you to use the standard steps and may require a powershell script to figure out which steps to run and which ones to skip. Set variables from powershell using the special logging commands.
Disable the get sources step and add a step that manually fetches sources. You'll need to clean the working directory, checkout the right commit, basically replicating the actions in the get sources step manually. It may require some fidgeting to get all the behavior correctly for normal build, pull request builds etc. That way you can take full control over the location where sources are checked out.
I'd also recommend you investigate the 2017 project formats that use the new <packageReference> in the project files to fetch packages. The new system supports configuring a version range which can always fetch the latest available version of packages. It's a better long-term solution.
No, it isn’t available in VSTS build system.
You can change working directory of agent (C:\vstsagent_work) (Re-configure it and specify another working folder), but it won’t uses the same source folder for different build definitions, the folder would be 1, 2, 3 ….
I'm new to Golang. As I understand, when you want to create a new Go project, we just need to create a directory. Then we point the environment variable GOPATH to this directory. Inside this directory, we create three subdirectories pkg, src and bin. Then when we execute go get ..., the third-party package will be installed in the pkg subdirectory. Later if I want to create another Go project, I create a new dir called project2 and point GOPATH to project2. At this time go get ... will download third-party package in the pkg subdirectory of project2. My question is, whether Go has a central repository? If not, the same package will be downloaded twice if they are used in two different projects. Is that true?
I guess now there is https://gocenter.jfrog.com/
More info in this blog https://jfrog.com/blog/go-at-full-speed-with-gocenter
There is no central repository of go packages. Go always is looking for packages either in GOPATH or GOROOT. go get simply downloads packages using git or mercurial. I recommend you to read
https://golang.org/doc/code.html
and https://peter.bourgon.org/go-best-practices-2016/#repository-structure
GOPATH simply tells go compiler where to search for src, pkg directories.
Later if I want to create another Go project, I create a new dir called project2 and point GOPATH to project2
…
My question is, whether Go has a central repository? If not, the same package will be downloaded twice if they are used in two different projects. Is that true?
No, there is no central repository for Go code. However, it is also not true that the packages will always be downloaded twice.
The misconception here is that GOPATH points to an individual project: it does not. Instead, GOPATH points to an environment where all of your packages live; it is where go get will download packages, and where go build will look for packages when building.
Instead of changing GOPATH for every project, you should set GOPATH once and put all of your projects in $GOPATH/src/ (your projects don't contain an src/ directory, they go in the src/ directory).
So for example, the entire tree might look like:
$GOPATH/src/bitbucket.org/ (or GitHub, or your website, or whatever)
├── YourProject
└── AnotherProject
Update
It is worth noting that this answer is no longer correct. Now that Go Modules are the normal versioning mechanism for Go code and $GOPATH is being phased out, a central proxy has been setup that routes all requests for packages through Google servers where the various tagged versions of the package can be cached. A separate checksum database keeps hashes for every package that are audit-able and can help you detect if a package author has changed an already released tag. All of this isn't a central repository in the same sense that PyPi (in the Python world) or NPM (for JavaScript) are a repo: the packages are still fetched from their source control, but because all packages are routed through the proxy by default it serves a similar purpose. For more information see https://proxy.golang.org/
Recently, a new site that collects information about Go packages has emerged:
https://go.dev/.
go.dev is the hub for Go users providing centralized and curated resources from across the Go ecosystem.
It is an official companion website to golang.org. It does not qualify for a repository, such as cpan, nmpjs, nuget or crates. For external packages, it simply links to their respective Github pages.
Go.dev is currently in MVP status. We’re proud of what we’ve built and excited to share it with the community. We hope you find value and joy in using go.dev. Go.dev only has a small portion of features we intend to build, and we are actively seeking feedback
But as is written it the about page, it is still in early development. Maybe one day (hopefully) it shall become a fully featured code repository.
According to the Go documentation they would like you to have a workspace that you should put all their projects in.1 However, as far as I can tell, this all falls apart as soon as you want to make a project that does not use Go exclusively.
Take a project where it is made up of many micoservices for example. Lets say that it is structured like this:
app/
authentication/ (Using rust)
users/ (Using NodeJS)
posts/ (Using Go)
Only one part of the app would be written in Go, and that part is nested in a subdirectory of the app. How would I apply the Go workspace philosophy to this situation?
https://golang.org/doc/code.html#Workspaces
Using a different GOPATH per project is a very good and simple approach. In my experience this also works better than vendor since you can also install binaries and keep them on different versions.
vg is a simple tool that helps managing workspaces, it integrates with your shell and detects automatically workspaces when you cd them.
Disclaimer: I am one of the authors of the tool.
As of Go 1.11, go now has modules. Amongst other things, modules enable you to have isolated source trees (with any number of packages and their own dependencies) outside of your $GOPATH.
You create a new module by running go mod init <module name> (you must be outside of $GOPATH/src to do this). This will create a go.mod file in the current folder, and any go command you run in that folder (or any folder beneath) will use that folder as your project root.
You can read more about using go modules as workspaces in this post: https://aliceh75.github.io/using-modules-for-workspaces-in-golang (disclaimer: I wrote it), and you can read more about Go modules on the Go Modules Wiki:
https://github.com/golang/go/wiki/Modules
You can put app/ in $GOPATH/src. Then whenever you're ready to build, you specify the path of your source files, relative to where they are in GOPATH.
For example:
if your app source is in $GOPATH/src/app/ and your .go files are in $GOPATH/src/app/posts/ then you can build a source (lets say posts.go in app/posts/) with go build $GOPATH/src/app/posts/posts.go or better go build posts/posts.go with app/ as your current working directory.
just set GOPATH according to your go files:
GOPATH=$PROJECT_PATH/app/posts
then put your source codes under
$PROJECT_PATH/app/posts/src/package