Golang - How to display modules version from inside of code - go

I'm writing two binaries, and both of them use two libraries (we can call them libA and libB).
Each lib is in a dedicated git repo, with git-tags to declare versions.
For example, libA is at v1.0.9 and libB is v0.0.12.
Both binaries have CLI flags, and I would like to add a debug flag to display lib versions like that:
> ./prog -d
Used libraries:
- libA, v1.0.9
- libB, v0.0.12
I don't know how to do that.
The only way I see to set variable from "outside" is to use ldflags (go build -ldflags="-X 'main.Version=v1.0.0'" for example). But this way don't seems scalable, how to add a libC? It also imply to manage tags two times, one time for git, and one time in goreleaser.yml or makefile.
Can you help me to find a solution?

The Go tool includes module and dependency information in the executable binary. You may use runtime/debug.ReadBuildInfo() to acquire it. It returns you a list of dependencies, including module path and version. Each module / dependency is described by a value of type debug.Module which contains these info:
type Module struct {
Path string // module path
Version string // module version
Sum string // checksum
Replace *Module // replaced by this module
}
For example:
package main
import (
"fmt"
"log"
"runtime/debug"
"github.com/icza/bitio"
)
func main() {
_ = bitio.NewReader
bi, ok := debug.ReadBuildInfo()
if !ok {
log.Printf("Failed to read build info")
return
}
for _, dep := range bi.Deps {
fmt.Printf("Dep: %+v\n", dep)
}
}
This outputs (try it on the Go Playground):
Dep: &{Path:github.com/icza/bitio Version:v1.0.0 Sum:h1:squ/m1SHyFeCA6+6Gyol1AxV9nmPPlJFT8c2vKdj3U8= Replace:<nil>}
Also see related question: How to get Go detailed build logs, with all used packages in GOPATH and "go module" mode?

Related

golang reproducable binary - changing buildId

Currently trying to work out why building twice from same source creates different buildid and more specifically different actionID part.
I have vendored the dependencies with the following main.go
package main
import (
"context"
"fmt"
awsConfig "github.com/aws/aws-sdk-go-v2/config"
)
func main() {
awsCfg, err := awsConfig.LoadDefaultConfig(context.Background())
if err != nil {
fmt.Println(err)
}
_ = awsCfg
fmt.Println("hello world")
}
go build -o bootstrap -mod=vendor -trimpath -buildvcs=false cmd/main.go
go clean --cache
go build -o bootstrap2 -mod=vendor -trimpath -buildvcs=false cmd/main.go
will produce the following difference respectively:
4PSJ2COQu-WEdQUCy9GF/bEj2KquhCFU77YWE3s8m/M-Eq01RmuZNuSlF4NvKh/Sw9yCoyvM-OHIpXpdNG5
1B4wdixPXI9YqxSglbgJ/bEj2KquhCFU77YWE3s8m/M-Eq01RmuZNuSlF4NvKh/Sw9yCoyvM-OHIpXpdNG5
This is causing updates to a lambda function has the produced hash from aws-cdk differs between builds on circleci even if nothing within the actual function has changed. I understand we could pass ldflags=-buildid= as a build argument (??) but want to understand what could be causing this change in actionId
This change is not consistent and sometimes produces the same buildId
go version go1.19 darwin/amd64
edit:
The following issue 33772 states this was resolved when using -trimpath

Compilation error only building a module, not main package

I'm making a Go program and have created a module to divide it. Here is my working tree (the minimal directory is in $GOPATH/src/):
minimal/
├── main.go
└── ui
├── go.mod
├── go.sum
└── ui.go
In the ui module I have the following:
package ui
import "github.com/satori/go.uuid"
func SomeFunction() {
id, err := uuid.NewV4()
if err == nil {
print(id.String())
} else {
print(err)
}
}
The main package uses this module as follows
package main
import "minimal/ui"
func main() {
ui.SomeFunction()
}
Here is the go.mod file:
module minimal/ui
go 1.14
require github.com/satori/go.uuid v1.2.0
When running go build in the main package folder, everything works and the binary is generated. However, when building only the ui module, it gives the following compilation error:
ui$ go build
# minimal/ui
./ui.go:6:10: assignment mismatch: 2 variables but uuid.NewV4 returns 1 values
You can check https://github.com/satori/go.uuid to see that the function returns two values. What has me puzzled is that building the main package works, but the module doesn't. Why is that?
github.com/satori/go.uuid documentation appears to support GOPATH builds. go module builds, however, produce inconsistent results.
Take the simple API usage from it's README.md:
package main
import "github.com/satori/go.uuid"
func main() {
u, err := uuid.NewV4()
_, _ = u, err
}
And try to compile it with go modules:
go mod init
go build
go: finding module for package github.com/satori/go.uuid
go: found github.com/satori/go.uuid in github.com/satori/go.uuid v1.2.0
./main.go:6:9: assignment mismatch: 2 variables but uuid.NewV4 returns 1 values
it pulled tagged v1.2.0 and it failed to compile.
So now pull the latest commit:
go get github.com/satori/go.uuid#master
go: github.com/satori/go.uuid master => v1.2.1-0.20181028125025-b2ce2384e17b
Now it compiles:
go build && echo $?
0
So what is happening here?
github.com/satori/go.uuid is tagged with a mature v1 release, so all dot releases should be feature enhancements with no breaking changes.
The latest tagged version v1.2.0 returns only one value from uuid.NewV4().
The latest commit (which go modules infers a pseudo dot release version of v1.2.1-0.20181028125025-b2ce2384e17b) has code that matches the README.md.
If this repo ever becomes tagged as v1.2.1 - this would be a breaking change, as it changes the signature of a function published in a previous v1 release. This would be a violation of semver rules.
Conclusion:
The documentation matches a GOPATH (i.e. non-go modules) build. GOPATH builds always pull the latest commit.
From a go modules perspective, while one may be able to coerce a working build, it does not look like it's supported. Yes, the git repo uses semver tags, but there is no go.mod.
I would not trust this package with a go modules build. Perhaps consider using github.com/google/uuid which does.
The uuid package has different versions. In the latest version, function NewV4() returns just one variable but in the previous versions, it returns two variables which one of them is an error.
You create a go.mod file in the subdirectory of your project, So when you compile your project in the main directory, go-compiler uses one version, and in your ui directory it uses another version to compile. You should just edit the version of uuid package in your go.mod file.

Golang: multiple definition of CGO ported package

I have 2 project, the first, name as A, there is a submodule a imported sqlite3(github.com/mattn/go-sqlite3). Another B project import A's submodule a, and in another submodule b, it also import the same sqlite3.
Both A and B put there imports under vendor dir(managed by govendor). My Golang version is go version go1.12 linux/amd64.
While build B (go build main.go), throwing following errors(too many, part of them):
/usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/tmp/go-link-281256755/000029.o: In function `callbackTrampoline':
/tmp/go-build/_cgo_export.c:25: multiple definition of `callbackTrampoline'
/tmp/go-link-281256755/000005.o:/tmp/go-build/_cgo_export.c:25: first defined here
/tmp/go-link-281256755/000029.o: In function `stepTrampoline':
...
/home/xxx/go/src/gitlab.xxxxxxxxx.com/xxxxxxxxx-tools/A/vendor/github.com/mattn/go-sqlite3/sqlite3.go:129: multiple definition of `_sqlite3_result_text'
/tmp/go-link-281256755/000009.o:/home/xxx/go/src/gitlab.xxxxxxxxx.com/xxxxxxxxx-tools/A/vendor/github.com/mattn/go-sqlite3/sqlite3.go:129: first defined here
/tmp/go-link-281256755/000033.o: In function `_sqlite3_result_blob':
...
But building A works well. To testing the error, I started following demo, also with vendor inited by govendor, and build ok.
package main
import (
"database/sql"
"fmt"
"gitlab.xxxxxxxxx.com/xxxxxxxxxxxxxxx/A/a"
_ "github.com/mattn/go-sqlite3"
)
func main() {
fmt.Println(a.ModuleVariable) // use submodule `a` just like B is doing
_, _ = sql.Open(`sqlite3`, `test.db`) // use sqlite too
}
I think the compiler first compile A's sqlite3, objects are created under /tmp/go-link-281256755/000005.o (but no this dir after building), then compile B's importing of sqlite3 and also create a object contains the same-name function, then the compiler find 2 same-name symbols, the linking failed.
How to fix these situation? Is there any golang env settings to avoid these?
After I remove the sqlite3 package under vendor both of A and B, they both use the sqlite3 under ~/go/src/github.com/mattn/go-sqlite3/, they all build ok. But I can't do these, due to project A's deploy platform, I must put all dependencies under vendor, is there any another option to use multiple import with the same package?
For the cgo's issue of linking error "multiple definition of ...", the (work-around) solution is dependent on the nature of the linked C codes:
If two Go packages link to the same C codes (libraries), you should pass option --allow-multiple-definition to the linker (see ld man page), either via command options
go build --ldflags '-extldflags "-Wl,--allow-multiple-definition"',
or via #cgo directive in Go source of the packages linking to C codes:
//#cgo LDFLAGS: -Wl,--allow-multiple-definition
import "C"
If two Go packages link to different C codes containing some functions & variables with the same names, you should refactor those C codes:
Make sure to put keyword static to all declarations whose usage is within that C object only (not intended to be linked to Go nor to other C objects).
Find some way to do name mangling or put those duplicated identifiers into different namespaces (like in C++). It would be better if cgo supported some mechanism to do automatic namespacing using Go package name, but until now (2020) you must do it yourself. The C preprocessor's "token pasting" operator ## may help in this namespacing task. Eg.
//File: my_package1.h
#define NS(id) my_package1_ ## id
void NS(my_function1)(int);
void NS(my_function2)(float);
char NS(my_shared_var);
If you have any C function definition in the Go source, like in this question, you must move those definitions into a separate C source file under the same package folder, leaving only declarations in the Go source.

How do Go plugin dependencies work?

Go 1.8 supports Go plugins.
I have created two plugins as follows.
As I understand, the plugin exposes only the functions and variables in the main package. i.e. plugin.Lookup() will fail for non-main variable/function.
But I wanted to test if a plugin can internally invoke a method from another plugin, similar to how a C++ library can invoke another library.
So I tested as follows:
plugin1 github.com/vimal/testplugin
$ cat myplugin.go
package main
import "C"
import "fmt"
import help "github.com/vimal/testplugin1/plug"
func init() {
fmt.Printf("main.init invoked\n")
}
// TestPlugin
func TestPlugin() string {
return help.Help()
}
plugin2 github.com/vimal/testplugin1
$ cat myplugin.go
package main
import "C"
func HelperFunc() string {
return "help"
}
$ cat plug/helper.go
package help
func Help() string {
return "help234"
}
Idea here is that plugin1 invokes an internal, non-main function of plugin2.
main program
Main program loads a number of plugins given as arguments, and invokes TestPlugin() from the last plugin.
test 1:
Build both plugins, and load both plugins, and invoke TestPlugin(), the output contains "help234", i.e. the inner function is invoked. This can be understood, since both plugins are loaded, one plugin can invoke another plugin's inner code.
test 2:
Load only plugin1, and invoke TestPlugin(), the output contains "help234", i.e. the inner function is invoked. The same output is observed as in test1. Perhaps this time the method is found from GOPATH.
test 3:
Rename the folder "github.com/vimal/testplugin1" to "github.com/vimal/junk1", delete the plugin2, and load only plugin1, and invoke TestPlugin(). the output still contains "help234", i.e. the inner function is invoked.
I am not able to understand how test3 produces the same output. Does plugin1 contain plugin2 code also? How can I understand the Go plugin dependency on other Go plugins?
Go version : go version go1.8rc3 linux/amd64
You're not doing exactly what you think you are.
Your plugin1 imports and uses a package, namely github.com/vimal/testplugin1/plug. This is not "equal" to plugin2!
What happens here is that when you build plugin1, all its dependencies are built into the plugin file, including the .../testplugin1/plug package. And when you load plugin1, all its dependencies are also loaded from the plugin file, including the plug package. After this, it's not surprising that it works no matter the loaded status of plugin2. These 2 plugins are independent from each another.
The -buildmode=plugin instructs the compiler that you want to build a plugin and not a standalone app, but it doesn't mean dependencies must not be included. They have to be, because the plugin cannot have any guarantee what Go app will load it, and what packages that Go app will have. Because a runnable app also only contains packages even from the standard library that the app itself references explicitly.
The only way to guarantee that the plugin will have everything it needs and so that it will work if it also contains all its dependencies, including those from the standard library. (This is the reason why building simple plugins generate relatively big files, similarly to building simple Go executables resulting in big files.)
Few things that don't need to be added to plugins include the Go runtime for example, because a running Go app that will load the plugin will already have a Go runtime running. (Note that you can only load a plugin from an app compiled with the same version of Go.) But beyond that, the plugin has to contain everything it needs.
Go is a statically linked language. Once a Go app or plugin is compiled, they do not rely nor check the value of GOPATH, it is only used by the Go tool during building them.
Deeper insight
It's possible that your main app and a plugin refers to the same package ("same" by import path). In such cases only one "instance" of the package will be used.
This can be tested if this commonly referred package has "state", e.g a global variable. Let's assume a common, shared package called mymath:
package mymath
var S string
func SetS(s string) {
S = s
}
And a plugin called pg that uses it:
package main
import (
"C"
"mymath"
"fmt"
)
func Start() {
fmt.Println("pg:mymath.S", mymath.S)
mymath.SetS("pghi")
fmt.Println("pg:mymath.S", mymath.S)
}
And the main app that uses mymath and loads pg (which uses it):
package main
import (
"plugin"
"mymath"
"fmt"
)
func main() {
fmt.Println("mymath.S", mymath.S)
mymath.SetS("hi")
fmt.Println("mymath.S", mymath.S)
p, err := plugin.Open("../pg/pg.so")
if err != nil {
panic(err)
}
start, err := p.Lookup("Start")
if err != nil {
panic(err)
}
start.(func())()
fmt.Println("mymath.S", mymath.S)
}
Building the plugin:
cd pg
go build -buildmode=plugin
Running the main app, the output is:
mymath.S
mymath.S hi
pg:mymath.S hi
pg:mymath.S pghi
mymath.S pghi
Analysis: first the main app plays with mymath.S, sets it to "hi" eventually. Then comes the plugin, which prints it (we see the value set by the main app "hi"), then changes it to "pghi". Then comes again the main app and prints mymath.S, and again, sees the last value set by the plugin: "pghi".
So there is only one "instance" of mymath. Now if you go ahead and change mymath, e.g. rename myMath.SetS() to mymath.SetS2(), and you update the call in the main app (to mymath.SetS2("hi")), and without rebuilding the plugin, just running the main app, you get the following output:
mymath.S
mymath.S hi
panic: plugin.Open: plugin was built with a different version of package mymath
goroutine 1 [running]:
main.main()
<GOPATH>/src/play/play.go:16 +0x4b5
exit status 2
As you can see, when building the main app and the plugin, the package version (which is most likely a hash) is recorded, which must match if their import paths match in the main app and in the plugin.
(Note that you will also get the above error if you don't change the exported identifiers (and signatures) of the used mymath package, only the implementation; e.g. func SetS(s string) { S = s + "+" }.)

Why can't I import pkg "builtin"?

cat test.go
package main
import "builtin"
func main() {
return
}
go run test.go
can't find import: "builtin"
I'm just curious because the file exists and is properly packaged. But can't be imported like other packages.
/usr/local/go/src/pkg/builtin/builtin.go
You don't need to import it. Is imported by default.
From http://golang.org/pkg/builtin:
Package builtin provides documentation for Go's predeclared identifiers. The items documented here are not actually in package builtin but their descriptions here allow godoc to present documentation for the language's special identifiers.
from golang.org/pkg/builtin)
If you take a look at the content of http://golang.org/src/pkg/builtin/builtin.go
You will notice that there are only declarations
// The copy built-in function copies elements from a source slice into a
// destination slice. (As a special case, it also will copy bytes from a
// string to a slice of bytes.) The source and destination may overlap. Copy
// returns the number of elements copied, which will be the minimum of
// len(src) and len(dst).
func copy(dst, src []Type) int
and as #Anonymous says the compiler skips it:
http://golang.org/src/cmd/go/build.go?#L558
if p.Standard {
switch p.ImportPath {
case "builtin", "unsafe":
// Fake packages - nothing to build.
return a
}
// gccgo standard library is "fake" too.
if _, ok := buildToolchain.(gccgoToolchain); ok {
// the target name is needed for cgo.
a.target = p.target
return a
}
}
When you import a package, the compiler (or at least, the gc compiler), searches for the already compiled package.
You can see this code in the source: http://golang.org/src/cmd/gc/lex.c?#L578
In particular, it doesn't search for .go files: these are assumed to be already built. This is a big win for go compared to, for example, C++, because each package can be compiled once, and code that depends on it can use the already-compiled version.
So why doesn't "builtin" ever get built, even though it's there as a package? Well, it's special-cased to be ignored in the part of the code that builds dependencies before building a source file: http://golang.org/src/cmd/go/build.go?#L558

Resources