Install a Go binary, keep dependencies / packages - go

When I build a Go binary, I usually do something like this:
go build -ldflags "-X main.basedir somevar" -o mybuilddir/bin/myfile mypackage/main"
this builds the binary and places it in a custom directory. But this doesn't keep the "intermediate" package files beneath pkg/, which would speed up the next compilation runs.
The solution would be go install, but I cannot specify an output directory. It seems to be possible to set the binary directory with GOBIN, but I am unable to specify the name of the executable (always main).
What is a possible solution to this problem?
Custom destionation directory
Custom name (not main)
Keep intermediate generated package files (.a)
This is the src directory of GOPATH:
GOPATH/src$ tree
.
└── mypackage
├── packagea
│   └── packagea.go
├── packageb
│   └── packageb.go
└── main
   └── mypackage.go
With go install, the package files (.a) are created in $GOPATH/pkg, with go build, I can't find the .a files anywhere.

Update Nov. 2017: Go 1.10 (Q1 2018) will add caching for go build and go install: see "go build rebuilds unnecessarily".
Original answer (2014)
With go install, the package files (.a) are created in $GOPATH/pkg, with go build, I can't find the .a files anywhere.
As mentioned in "How does the go build command work ?"
The final step is to pack the object file into an archive file, .a, which the linker and the compiler consume.
Because we invoked go build on a package, the result is discarded as $WORK is deleted after the build completes.
If we invoke go install -x two additional lines appear in the output
mkdir -p /home/dfc/go/pkg/linux_arm/crypto/
cp $WORK/crypto/hmac.a /home/dfc/go/pkg/linux_arm/crypto/hmac.a
This demonstrates the difference between go build and install;
build builds,
install builds then installs the result to be used by other builds.

Related

GO Monorepo: List of Files that need to be re-built using go build

I have a monorepo setup for my go project. I would love it if I could find a way to use go build (or similar internal tool) to get a list of targets that need to be re-built.
Here is an example of what I am looking for:
...
├── pkg //shared code across mono repo
│ └── math
│ └── common_operations.go
└── cmd // individual packages to be built
├── package1
│ └── main.go
└── package2
└── main.go
The package1 program calls a subtract function from the math shared library. The package2 program calls an add function.
If I change the package1 code, only the package1 target is listed
If I change the package2 code, only the package2 target is listed
If I change the add function in the shared library, only the package2 target is listed
If I change the subtract function in the shared library, only the package1 target is listed
If I change all the functions in the shared library, both package1 and package2 rebuilds.
I would be perfectly happy to use use the internal build package and get the list programatically. I am just am unfamiliar with it.
What I have tried:
Bazel has an option for this but I would prefer to avoid using bazel if at all possible.
the bazel command: bazel build cmd/some-target --check_up_to_date returns error code 0 if it is up to date, otherwise it returns error code 1.
Which is technically a solution, but my need, as you might have inferred, is ci/cd based. And I want to avoid integrating Bazel into that process as much as possible.
Not really sure of the use case here, are you OK with actually compiling the packages as well? In that case maybe go build -v can do the job for you. From go help build:
-v
print the names of packages as they are compiled.
My desire was to find something similar to a go build option that would basically spit out a true or false if a target package was up-to-date. I would be happy if someone out there found a tool ( Earthly I am looking at you right now) that could solve this.
The closest thing to a solution I could find to solve my issue was to run this command:
go build -n cmd/some-target
And if the output is:
touch some-target
Then, it must be up-to-date. If the output is a long list of commands, then it isn't.
If it is not up-to-date. You could then get the package name using:
go build -v
To get the name of the package and move it to the next stage of the CI process (building target, testing target, building image, etc).
Obviously it is a little hacky and requires self-rolling a solution and specifics would likely need to be changed on your exact needs. It also requires, as #zacho314 mentioned, saving the state of the go build cache, but most modern CI technologies have a solution for that. I am sure I will do something similar to this for now.

How to install plugins

TL;DR: Where do .so files end up when you install plugins with go install -buildmode=plugin?
I have a project that uses plugins. The layout is something like this:
myproject/
├── main.go
└── modules
├── bar
│   └── main.go
└── foo
└── main.go
When I run go install the binary gets installed OK.
But I would also like to run go install for each of my modules and have them available to the main binary everywhere on the system.
If I run go install -buildmode=plugin from inside a module folder (say, modules/foo) the command runs to completion but I can't find the resulting file anywhere.
Installing a normal package ends up in:
GOPATH/pkg/<goos>_<goarch>_dynlink/path/to/parent/folder/packagename.a
Installing main packages end up in:
GOPATH/bin/foldername
(where foldername is the parent folder of the main package you install, it'll get an .exe extension on windows).
When you "go install" a plugin (using -buildmode=plugin), that ends up in
GOPATH/pkg/<goos>_<goarch>_dynlink/path/to/parent/folder/foldername.a

go install creates directory os_arch - select different output directory

I have this folder structure for my fib package:
$ tree
.
└── src
└── fib
├── fib
│   └── main.go
├── fib.go
└── fib_test.go
(main.go is in package main, fib(_test).go is in package fib)
GOPATH is set to $PWD/src, GOBIN is set to $PWD/bin. When I run go install fib/fib, I get a file called fib in the directory bin (this is what I expect):
$ tree bin/
bin/
└── fib
But when I set GOOS or GOARCH, the directory in the form GOOS_GOARCH is created:
$ GOARCH=386 GOOS=windows go install fib/fib
$ tree bin/
bin/
└── windows_386
└── fib.exe
This is not what I want. I'd like to have the file fib.exe in the bin directory, not in the sub directory bin/windows_386.
(How) is this possible?
That doesn't seem possible, as illustrated in issue 6201.
GOARCH sets the kind of binary to build.
You might be cross-compiling: GOARCH might be arm.
You definitely don't want to run the arm tool on an x86 system.
The host system type is GOHOSTARCH.
To install the api tool (or any tools) you need to use
GOARCH=$(go env GOHOSTARCH) go install .../api
and then plain 'go tool' will find them.
In any case (GOARCH or GOHOSTARCH), the go command will install in a fixed location that you cannot change.
The phrase "I (don't) want" is incompatible with the go tool; the go tool works how it works. You can a) copy the file to where you want it to be after installing it with the go tool or b) compile it yourself, e.g. by invoking 6g manually (here you can specify the output). If you are unhappy with how the go tool works, just switch to a build tool of your liking, e.g. plain old Makefiles. Note that the go tool helps you there too, e.g. by invoking the compiler via go tool 6g

What does go install do?

The docs say nothing about what build vs install does
My expectation was that it's like make install; i.e. it takes the compiled stuff and puts in its final location (/usr/local/bin/my_new_toy or whatever) but it seems that it puts things in GOROOT/bin
Can I tell go to do a make install - i.e. put things elsewhere? Or do I just write a makefile (please tell me no)?
go build vs go install:
go build just compiles the executable file and moves it to the destination.
go install does a little bit more. It moves the executable file to
$GOPATH/bin and caches all non-main packages which are imported to
$GOPATH/pkg. The cache will be used during the next compilation provided the
source did not change yet.
A package tree after go build and go install:
.
├── bin
│ └── hello # by go install
└── src
└── hello
├── hello # by go build
└── hello.go
More detailed information.
If you want binary files to go to a specific location, you can use the environment variable GOBIN :
The bin/ directory holds compiled commands. Each command is named for
its source directory, but only the final element, not the entire path.
That is, the command with source in DIR/src/foo/quux is installed into
DIR/bin/quux, not DIR/bin/foo/quux. The foo/ is stripped so that you
can add DIR/bin to your PATH to get at the installed commands. If the
GOBIN environment variable is set, commands are installed to the
directory it names instead of DIR/bin.
Source : http://golang.org/cmd/go/#hdr-GOPATH_environment_variable
GOBIN=/usr/local/bin/ go install
If you want per-project bin/ directory then you can simply append your project path to GOPATH, however you must have your code under $project-path/src/ and go install will put all the binaries in $project-path/bin.
export GOPATH=/dir1:/dir2:/dir3
If GOBIN is not set, binaries from /dir1/src end up in /dir1/bin,
binaries from /dir2/src end up in /dir2/bin, and so on (and binaries
from $GOROOT/src end up in $GOROOT/bin).
Source : https://groups.google.com/forum/#!topic/golang-nuts/-mN8R_Fx-7M
And you can also just use (thanks JimB):
go build -o /path/binary-name

Correct directory structure for a Go Project?

I'm relatively new to Go and I've recently created a project that's going up on GitHub. I've tried to follow guides but there's a pressing question of why my binaries end up in src/?
My layout is like this:
ssm/ - Name of project
LICENSE
README.md
src/ - Source Code
files.go - All my source code is here.
src - The compiled binary ends up here
bin/ - Binaries
I set my $GOPATH to ~/Documents/Programming/Go/. From my gopath, I can't type go build ssm because it cannot find package. If I cd into the directory, it complains it can't load package: package .: no Go source files.
I have to actually go into src and compile there, which means the binary isn't in bin/.
What am I doing wrong?
See https://code.google.com/p/go-wiki/wiki/GithubCodeLayout
To be compatible with go get, your project's package name needs to be fully-qualified under the github.com domain:
$GOPATH/
src/github.com/<user>/ssm/
.git
LICENSE
README.md
files.go
bin/
Note that the base of the git repository (.git) is not the same as the $GOPATH.
Also, go build <package> will output a compiled executable to the current directory. If you want the exe to go to bin/, use go install <package> instead.
Your go code you can kept in a workspace. A workspace contains many source files (git, hg, svm, etc.). The Go tool understand the layout. We don't require Makefile or build.xml here. The basic directory layout is everything. If in any case if you are going to change the file layout, accordingly you need to change the build.
This is the general structure you can follow,
$GOPATH/
src/
github.com/username/repo/
mypkg/
mysrc1.go
mysrc2.go
cmd/mycmd/
main.go
bin/
mycmd
And, this is the standard workspace
$GOPATH/
bin/fixhub # installed binary
pkg/darwin_amd64/ # compiled archives
code.google.com/p/goauth2/oauth.a
github.com/...
src/ # source repositories
code.google.com/p/goauth2/
.hg
oauth # used by package go-github
...
github.com/
golang/lint/... # used by package fixhub
.git
google/go-github/... # used by package fixhub
.git
dsymonds/fixhub/
.git
client.go
cmd/fixhub/fixhub.go # package main
go get fetch many repositories whereas go install builds a binary out of them. It's convenient and easy to go for go development quickly. And, everyone in the go community follows the same. This puts src, bin and pkg into the home directory. And, $HOME/bin is already in our path before creating our workspace.

Resources