go build and go run are very slow on a tiny program I have (cgo invocations in particular). I'd like go to cache the binary so that it only rebuilds when the source is newer. I would use a simple Makefile with a % rule, but the language designers claim that go's build support doesn't need Makefiles.
Is there another alternative I've overlooked? Does the go community prefer another build system, maybe hash-based instead, for caching and reusing build products?
go build and go install will soon (Go 1.10, Q1 2018) be much faster: see this thread and this draft document.
The go command now maintains a cache of built packages and other small metadata (CL 68116 and CL 75473). The cache defaults to the operating system-defined user cache directory but can be moved by setting $GOCACHE.
Run "go env GOCACHE" to see the current effective setting. Right now the go command never deletes anything from the cache. If the cache gets too big, run "go clean -cache" instead of deleting the directory. That command will preserve the cache's log.txt file. In a few weeks I'll ask people to post their log.txt files to a Github issue so that we can evaluate cache size management approaches.
The main effect of the build cache is that commands like "go test" and "go build" run fast and do incremental builds always, reusing past build steps as aggressively as possible.
You do not have to use "go test -i" or "go build -i" or "go install" just to get fast incremental builds. We will not have to teach new users those workarounds anymore. Everything will just be fast.
Note that go install won't installs dependencies of the named packages: see "What does go build build?".
I wrote a tool that happens to solve this as a side effect. go build alone will not check if the executable it's producing is already up to date. go install does, and if you tweak it to install to a location of your choice, then you'll get the desired result, similar to go build.
You can see the behaviour you describe by doing something like this:
$ go get -d github.com/anacrolix/missinggo/cmd/nop
$ time go run "$GOPATH"/src/github.com/anacrolix/missinggo/cmd/nop/*.go
real 0m0.176s
user 0m0.142s
sys 0m0.048s
That's on a warm run. go run will link on every invocation, just as go build would. Note that github.com/anacrolix/missinggo/cmd/nop is an program that does absolutely nothing.
Here's invoking the same package, using my tool, godo:
$ time godo github.com/anacrolix/missinggo/cmd/nop
real 0m0.073s
user 0m0.029s
sys 0m0.033s
For larger programs, the difference should be more pronounced.
So in summary, your standard tooling option is to use go install, or an alternative like godo.
Related
I always build with -i flag to install packages and .a files are installed in GOPATH/pkg directory.
GOCACHE directory shown by go env GOCACHE seems to store cache files as well.
What's difference between them?
And what I wanna know is both of them should be saved if I want to make build time faster?
TLDR; The cache folder is internal to the go tool and its working should be opaque to the user, and its purpose is to speed up builds and tests. For example if you use a version control system (such as git), switching between branches or versions, the GOPATH/pkg may only contain package files of one version. The go cache folder may contain (partially) compiled packages of multiple branches and versions, speeding up future builds when you switch between branches and versions.
The cache folder was introduced in Go 1.10:
The go build command now maintains a cache of recently built packages, separate from the installed packages in $GOROOT/pkg or $GOPATH/pkg. The effect of the cache should be to speed builds that do not explicitly install packages or when switching between different copies of source code (for example, when changing back and forth between different branches in a version control system). The old advice to add the -i flag for speed, as in go build -i or go test -i, is no longer necessary: builds run just as fast without -i. For more details, see go help cache.
So you don't need to use -i anymore for fast builds.
Some quotes from the output of go help cache:
The go command caches build outputs for reuse in future builds.
The default location for cache data is a subdirectory named go-build
in the standard user cache directory for the current operating system.
Setting the GOCACHE environment variable overrides this default,
and running 'go env GOCACHE' prints the current cache directory.
The go command periodically deletes cached data that has not been
used recently. Running 'go clean -cache' deletes all cached data.
The build cache correctly accounts for changes to Go source files,
compilers, compiler options, and so on: cleaning the cache explicitly
should not be necessary in typical use. However, the build cache
does not detect changes to C libraries imported with cgo.
If you have made changes to the C libraries on your system, you
will need to clean the cache explicitly or else use the -a build flag
(see 'go help build') to force rebuilding of packages that
depend on the updated C libraries.
The go command also caches successful package test results.
See 'go help test' for details. Running 'go clean -testcache' removes
all cached test results (but not cached build results).
The cache folder is also used to store test results, so in some circumstances, the cached results may be presented without running the tests again.
Your questions is self-answering:
$ ls $(go env GOCACHE)
$ cat $(go env GOCACHE)/README
and
$ ls $(go env GOPATH)/pkg
As you can see - there is nothing similar between them:
GOPATH/pkg - compiled packages that remains static between builds. Those files are not "cashe" files actually
GOCACHE - collections of build artefacts that constantly changes between builds
More elaborate answer could be done by examining sources of go build
I'm running a project using Go Modules with 1.11.4 on Ubuntu, running in WSL.
My problem is that I'm having getting incremental builds to work as I expect. Perhaps this is due to me misunderstanding how it's supposed to, but I'd be glad if someone could clarify if that is the case.
Just as an example, if I do go build ./... then everything is built, as expected.
If I now do go build ./... again without any changes, my expectation was that due to the incremental builds, this time nothing would be built. But it builds everything again. I tried doing go build -i ./... (even though my understanding is that -i isn't needed anymore from 1.10), but the result is the same. This has been puzzling me for some time, as after reading the documentation I indeed expected the go build command to produce incremental builds.
The other day I realized that if instead I do go install ./... first, and then go install ./... again a second time, the second time around nothing is built, as I would expect. If I change a single module and run go install ./... again, then only that module is rebuilt, again what I would expect. So this gives me incremental builds.
So my questions are
1) Did I misunderstand go build ./... and how it handles incremental builds? Do I need to use go install instead?
2) Typically, we build the modules one by one, using the -o flag to specify an output path. Using go install instead, there is no -o option to specify an output path. Is there anything I can do to achieve something similar to -o using go install?
Thanks!
I have a little problem, where every time I run 'go run >filename<' after making changes to my problem, it takes many seconds to start executing.
I tried it with a simple program like this:
package main
import "fmt"
func main() {
fmt.Println("Output")
}
And it took about 18 seconds to print out the result.
And ideas on what could cause this problem?
I am on windows by the way.
Thanks in Advance
Found this while experiencing an identical issue with freshly-compiled golang binaries on MacOS (OSX) Catalina.
In short, the OS now scans a new (to it) binary for malware, it usually does it on first start, but I found that it was doing it 3-5 times in a row, after which the binary got whitelisted and started as normal. Obviously, once you make changes to the code and re-compile the scans start happening again. The scan would take upwards of 20 seconds which ruined golangs fast iteration cycle.
Solution for me was as follows:
sudo spctl developer-mode enable-terminal
Then go to Preferences -> Security & Privacy -> Privacy
In the list to the left you will now have "Developer Tools" section, which will have OSX built-in Terminal listed. Check the box to enable it, and/or add anything else you may be using to develop (iTerm, VS Code, etc)
When running binaries from those applications the scans stop and things go back to normal.
$ go run command always complies the code into a temporary binary file and then executes it, every time it is run.
To go around this you could do $ go build -i main.go which will compile dependencies separately as .a files (i'm guessing this is the part that takes the longest because it takes time to build the dependencies) and then execute it with $ ./main, and each execution should be faster that $ go run.
You could also run a $ go get -u ./... to update all your deps and building with the -x flag will show you if the toolchain is finding incompatible versions.
$ go install builds the command in a temporary directory then moves it to $GOPATH/bin so you can execute it without the path $ main.
The last two commands require a rebuild/reinstall if the code has changes.
If anyone still facing this issue on windows. you can fix it in 2 ways. it works 100%.
open virus & threat protection in windows security in settings
way 1:
turn off real-time protection
way 2:
in virus and threat protection settings(click on manage settings) go to exclusions and add the following three folders
C:\Users\anves\AppData\Local\go-build
C:\Users\anves\AppData\Local\Temp
The folder where go is installed.
I tried to run the program in command prompt #Go lang- But when i type run "go run hello.go" command i am gettin
CreateFile hello.go:The system cannot find the file specified
Please help to to compile and run the above marked program, Thanks in advance
As you can see from the output of running the dir command
earlier up in your shell session, there is indeed no file named "hello.go"
in the C:\GOCODE\testproject directory.
When you execute the
go run hello.go
command, the go tool tries to find the file named "hello.go" in the current
directory (because the name of that file is relative, so it's being
searched in current working directory). There's no such file,
and that's what go run tells you.
Unfortunately, from the outlook of your shell session, it appears there
are more problems with your setup.
And there are problems with your approach to Go.
First, while it looks like you're following
this guide (and this is the right thing
to do, actually), you misread it.
What it tried to tell you is that you should create the "src"
directory (and then the "hello" directory to contain your test project)
in the so-called workspace, and a list of workspaces known to Go
is stored in the GOPATH environment variable.
As you can see from the go env output, Go thinks you have a single
workspace located in C:\Users\Sitaram\Go.
Now it worth reiterating that—contrary to many (if not most) "mainstream"
languages,—Go is not "project-based"; instead, it requires all your code
to be organized in those workspaces, and it wants to know where these
workspaces are.
By default—if you did not explicitly set the GOPATH environment
variable,—it assumes your single workspace is located in the directory
named "go" placed in your "home folder".
And that's what you see in the go env output.
Now you have two options:
Set the GOPATH env. variable for your user to C:\GOCODE
then start another shell—so that it "sees" that variable and allows
the go tool to also see it and use).
Run go env to verify GOPATH contains C:\GOCODE.
Then follow the rest of the tutorial document:
Make sure there is the "src" folder directly under the C:\GOCODE.
Create your project folder directory under "src".
Let's say, it will be named "hello".
Under "hello", create that "hello.go" file.
Now cd C:\GOCODE\src\hello and then go build — you will have
the hello.exe created there.
Don't mess with GOPATH and just repeat the steps 2-4 from above
in the default workspace—C:\Users\Sitaram\go.
I'd go with the second variant because that inexplicable affection
of certain Windows users for polluting the C:\ with random personal
data is really an anti-pattern; have your personal belongings in your home
folder! Windows has gone a long way getting that right; and almost all
Windows software is finally there—understanding that paradigm. So why deviate?
Second, please unlearn go run.
I'm not sure the Go developers actually regret implementing it,
but people do really misinterpret what this tool is for.
It's for one-off throw-away "scripts".
Real development is done using go install and, sometimes, go build.
In most cases your normal development routine you use go install
exclusively — as it caches the results of compilation of all the packages
your project depends on. go build does not do this, and go run does
not even preserve the result of the compilation of your project itself.
Please read this quick reference card for more info.
After go get gpackage check if there is a yourproject.exe in your bin directory to compile your github package with your project.
If not, you have to do cd src/yourproject and type go install and hit enter.
Try execute the folowing command e check if corrected:
go env -w GOOS=windows
After upgrading from Go 1.2.1 to 1.3 (Windows 7 64 bit) "go build" execution time has increased from around 4 to over 45 seconds. There were no other changes except the go version update. Switching off the virus scanner seems to have no effect. Any clues?
You probably have dependencies that are being recompiled each time. Try go install -a mypackage to rebuild all dependencies.
Removing $GOPATH/pkg also helps to ensure you don't have old object files around.
Building with the -x flag will show you if the toolchain is finding incompatible versions.
I have the exact same problem, running this command solves it:
go get -u -v github.com/mattn/go-sqlite3
Another tip: http://kokizzu.blogspot.co.id/2016/06/solution-for-golang-slow-compile.html
Using go1.6,
Simply run go build -i.
It will compile all the dependencies and store them at $GOPATH/pkg/*/* as .a files.
Later when you run go run main.go, everything is much faster.
What s really great is that if you use vendored dependencies (IE: a vendor folder in your project), deps are built appropriately within $GOPATH/pkg/**/yourproject/vendor/**
So you don t have to go get install/get/whatever and have a mix of vendor / global dependencies.
I suspect you got to re-build .a files after deps update (glide update or smthg like this), but i did not test that yet.
After Go 1.10, you'd just need to type go build. You'd not need to type: go build -i.
From the draft Go 1.10 document, here.
Build & Install
The go build command now detects out-of-date packages purely based on the content of source files, specified build flags, and metadata stored in the compiled packages. Modification times are no longer consulted or relevant. The old advice to add -a to force a rebuild in cases where the modification times were misleading for one reason or another (for example, changes in build flags) is no longer necessary: builds now always detect when packages must be rebuilt. (If you observe otherwise, please file a bug.)
...
The go build command now maintains a cache of recently built packages, separate from the installed packages in $GOROOT/pkg or $GOPATH/pkg. The effect of the cache should be to speed builds that do not explicitly install packages or when switching between different copies of source code (for example, when changing back and forth between different branches in a version control system). The old advice to add the -i flag for speed, as in go build -i or go test -i, is no longer necessary: builds run just as fast without -i. For more details, see go help cache.
I just experienced the same problem - updating from 1.4 to 1.5. It seems that the olds versions are somehow incompatible or are being rebuild every time as go build -x shows. Executing go get -v invalidates all packages or refetches them, I am not quite sure and go build -x shows quite less output.
You can build sqlite3 like this:
cd ./vendor/github.com/mattn/go-sqlite3/
go install
After that your project will b built much faster.
If you try as all other said but still not work, I suggest removing the directory of $GOPATH such as :
sudo rm -rf $GOPATH
cd yourproject
go get -d
go get -u -v github.com/mattn/go-sqlite3