Golang: multiple definition of CGO ported package - go

I have 2 project, the first, name as A, there is a submodule a imported sqlite3(github.com/mattn/go-sqlite3). Another B project import A's submodule a, and in another submodule b, it also import the same sqlite3.
Both A and B put there imports under vendor dir(managed by govendor). My Golang version is go version go1.12 linux/amd64.
While build B (go build main.go), throwing following errors(too many, part of them):
/usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/tmp/go-link-281256755/000029.o: In function `callbackTrampoline':
/tmp/go-build/_cgo_export.c:25: multiple definition of `callbackTrampoline'
/tmp/go-link-281256755/000005.o:/tmp/go-build/_cgo_export.c:25: first defined here
/tmp/go-link-281256755/000029.o: In function `stepTrampoline':
...
/home/xxx/go/src/gitlab.xxxxxxxxx.com/xxxxxxxxx-tools/A/vendor/github.com/mattn/go-sqlite3/sqlite3.go:129: multiple definition of `_sqlite3_result_text'
/tmp/go-link-281256755/000009.o:/home/xxx/go/src/gitlab.xxxxxxxxx.com/xxxxxxxxx-tools/A/vendor/github.com/mattn/go-sqlite3/sqlite3.go:129: first defined here
/tmp/go-link-281256755/000033.o: In function `_sqlite3_result_blob':
...
But building A works well. To testing the error, I started following demo, also with vendor inited by govendor, and build ok.
package main
import (
"database/sql"
"fmt"
"gitlab.xxxxxxxxx.com/xxxxxxxxxxxxxxx/A/a"
_ "github.com/mattn/go-sqlite3"
)
func main() {
fmt.Println(a.ModuleVariable) // use submodule `a` just like B is doing
_, _ = sql.Open(`sqlite3`, `test.db`) // use sqlite too
}
I think the compiler first compile A's sqlite3, objects are created under /tmp/go-link-281256755/000005.o (but no this dir after building), then compile B's importing of sqlite3 and also create a object contains the same-name function, then the compiler find 2 same-name symbols, the linking failed.
How to fix these situation? Is there any golang env settings to avoid these?
After I remove the sqlite3 package under vendor both of A and B, they both use the sqlite3 under ~/go/src/github.com/mattn/go-sqlite3/, they all build ok. But I can't do these, due to project A's deploy platform, I must put all dependencies under vendor, is there any another option to use multiple import with the same package?

For the cgo's issue of linking error "multiple definition of ...", the (work-around) solution is dependent on the nature of the linked C codes:
If two Go packages link to the same C codes (libraries), you should pass option --allow-multiple-definition to the linker (see ld man page), either via command options
go build --ldflags '-extldflags "-Wl,--allow-multiple-definition"',
or via #cgo directive in Go source of the packages linking to C codes:
//#cgo LDFLAGS: -Wl,--allow-multiple-definition
import "C"
If two Go packages link to different C codes containing some functions & variables with the same names, you should refactor those C codes:
Make sure to put keyword static to all declarations whose usage is within that C object only (not intended to be linked to Go nor to other C objects).
Find some way to do name mangling or put those duplicated identifiers into different namespaces (like in C++). It would be better if cgo supported some mechanism to do automatic namespacing using Go package name, but until now (2020) you must do it yourself. The C preprocessor's "token pasting" operator ## may help in this namespacing task. Eg.
//File: my_package1.h
#define NS(id) my_package1_ ## id
void NS(my_function1)(int);
void NS(my_function2)(float);
char NS(my_shared_var);
If you have any C function definition in the Go source, like in this question, you must move those definitions into a separate C source file under the same package folder, leaving only declarations in the Go source.

Related

Golang - Duplicate Symbol when multiple libraries with same C libraries

When I try to build a Go program that has multiple libraries that import the same C libraries via CGO, I get duplicate symbol errors.
Here is some sample code
package main
import (
_ "github.com/karalabe/usb"
_ "github.com/trezor/trezord-go/usb"
)
func main() {
}
When I try to build this, I get errors like:
/Users/tlainevool/.asdf/installs/golang/1.14.2/go/pkg/tool/darwin_amd64/link: running clang failed: exit status 1
duplicate symbol _hid_read_timeout in:
/var/folders/5f/r8k9s56d5wbfpx5sj9d739pwnv1297/T/go-link-572074318/000002.o
/var/folders/5f/r8k9s56d5wbfpx5sj9d739pwnv1297/T/go-link-572074318/000016.o
duplicate symbol _hid_get_feature_report in:
/var/folders/5f/r8k9s56d5wbfpx5sj9d739pwnv1297/T/go-link-572074318/000002.o
/var/folders/5f/r8k9s56d5wbfpx5sj9d739pwnv1297/T/go-link-572074318/000016.o
This is just a toy example, my actual code uses an internal library and another external one that both reference the same third-party C libraries, but I believe the solution would be the same.
Is there any way to get clang to recognize the symbols from the two different libraries as the same ones? Or have them be in different namespaces or something?

Simple CGO example fails compilation with "duplicate symbol" error

I want to make a simple example of calling C code from Go with CGO. But for some reason I can't achieve desired. Compilation fails with the following error:
go build main.go
# awesomeProject1/print
duplicate symbol '_do_print' in:
$WORK/b002/_x002.o
$WORK/b002/_x003.o
ld: 1 duplicate symbol for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
The code:
// print/print.c
#include <stdio.h>
void do_print(char * x){
printf("%s", x);
}
// print/print.go
package print
// #include <print.c>
import "C"
func DoPrint() {
C.do_print(C.CString("Hello!"))
}
// main.go
package main
import "awesomeProject1/print"
func main() {
print.DoPrint()
}
If I make do_print function static it compiles but I wouldn't be able to do that for 3rd party code I want to integrate with later.
Am I missing some essential piece from documentation? Tutorials are all alike and claim to work where my example fails. Please help!
Go version 1.16.4
There are two things going on here:
go build compiles *.c in addition to *.go1
#include <print.c> is exactly equivalent to overwriting the include statement with the contents of print.c
As a result, you are compiling the contents of print.c twice: once when print.c is compiled by CC, and once when print.go is compiled by CGo. Thus, the object files for print.c and print.go each contain all of the exported symbols defined in print.c. So you get two copies of do_print. Making do_print static works because a function declared as static will not be exported by CC.
Including a .c file (e.g. #include <file.c>) is pretty much always a bad idea. If you have a legitimate reason to #include function bodies, the convention is to use a header (.h) file. This is what C++ templating libraries do (like Boost, IIRC). Because C files normally are not included, and H files normally are included, including a C file subverts expectations and thus is likely to cause confusion and problems.
1 IIRC, Go will compile C files in a package if any of the Go files in the package import "C". That is, if no Go files in a package use CGo, Go will ignore C files in that package.

Makefile variable shared between custom packages for openwrt

While adding custom packages, package A is using Makefile variable that needs to be updated (or not) by other package B, however package B is not mandatory for compiling package A. In other words, if configuration selects package B in that case shared Makefile variable be populated and values can be seen by other packages.
Package A
$(variable1)
package B
variable1 := "-Dxyz -Dabc"
I tried with setting empty "variable1" in rules.mk which is not working. Appreciate if anyone can point me existing examples for same.

How to export data from a test package and use them in another test package in go

I'm creating some vars in B_test.go and I want to use those same vars in A_test.go. Can this be done in Go? I think the question boils down to whether I can export functions from B_test.go during go test only.
Example:
In package A_test.go
package A
var from_B = B.ExportedVars()
In package B_test.go
package B
ExportedVars() []int {
return []int{0, 1)
}
Running go test gives
undefined B.ExportedVars
Putting ExportedVars() in B.go instead of B_test.go solves the problem but this is not what I want. I want it to live in the test file.
Packages cannot see exported symbols from other package's test files, because the go tools do not build those files into the installed package. One option you do have is to use build constraints.
Create a file or files that contain everything you want to export for whatever reason, without using the _test.go suffix. Then mark them for build only when using your chosen tag.
// +build export
package normal
var ExportedName = somethingInternal
When testing a package dependent on this ExportedName in package normal, you will need to add the -tags export flag to the test run.
This is also useful for various other reasons, like having a -tags debug build, which can add extra functionality such as importing net/http/pprof or expvar

Importing local library and files in an application

I'm new to Go (but not at programming), I love the language but I have a bit of trouble fully understanding the way I'm supposed to make internal libraries in an application through packages. For reference, getting external packages and then importing/using them is fine.
Let's say I'm making an application A.
/home/me/A/a.go (package main)
Then, I realize a.go start to be rather big, so I cut it into two parts
/home/me/A/a.go (package main)
/home/me/A/b.go (package main)
How am I supposed to import/include b.go from a.go to make its function available ?
As a continuation of the question, in the A I'm manipulation lots of objects O, so I figure it would be a lot better if I just give them their own package and encapsulate the functionalities in a public/exported api. How do I do that ?
I've tried creating ./lib/o.go (package o) and import lib/o but I keep getting error like
./a.go:6: imported and not used: "o"
./a.go:43: undefined: o
I have no GOPATH in my env but I tried export GOPATH=$GOPATH:/home/me/A and it didn't change the result.
I've tried to read the article on "go layout" but it felt a bit too overwhelming at once and I would really love a simpler explanation of that one "small" step I am trying to make
Thanks !
GOPATH/src/me/a/a.go:
package main
func main() {
test()
}
GOPATH/src/me/a/test.go:
package main
import "fmt"
func test() {
fmt.Println("test func !")
}
Exec:
$ go run a.go
# command-line-arguments
./a.go:4: undefined: test
EDIT: got my answer here: https://groups.google.com/forum/?fromgroups#!topic/golang-nuts/qysy2bM_o1I
Either list all files in go run (go run a.go test.go) or use go build and run the resulting executable.
You're trying to use the Go build system while not following the necessaary required directory layouts. You will benefit a lot from reading this document.
In short, these are, wrt the go tool, the show stoppers:
You must have a valid, exported GOPATH
Package files with import path "example/foo" must be located in the $GOPATH/src/example/foo directory.
For more details please see the above linked article.

Resources