When I try to build a Go program that has multiple libraries that import the same C libraries via CGO, I get duplicate symbol errors.
Here is some sample code
package main
import (
_ "github.com/karalabe/usb"
_ "github.com/trezor/trezord-go/usb"
)
func main() {
}
When I try to build this, I get errors like:
/Users/tlainevool/.asdf/installs/golang/1.14.2/go/pkg/tool/darwin_amd64/link: running clang failed: exit status 1
duplicate symbol _hid_read_timeout in:
/var/folders/5f/r8k9s56d5wbfpx5sj9d739pwnv1297/T/go-link-572074318/000002.o
/var/folders/5f/r8k9s56d5wbfpx5sj9d739pwnv1297/T/go-link-572074318/000016.o
duplicate symbol _hid_get_feature_report in:
/var/folders/5f/r8k9s56d5wbfpx5sj9d739pwnv1297/T/go-link-572074318/000002.o
/var/folders/5f/r8k9s56d5wbfpx5sj9d739pwnv1297/T/go-link-572074318/000016.o
This is just a toy example, my actual code uses an internal library and another external one that both reference the same third-party C libraries, but I believe the solution would be the same.
Is there any way to get clang to recognize the symbols from the two different libraries as the same ones? Or have them be in different namespaces or something?
Related
I'm creating a grammar to run a console application, using VSCode or IntelliJ extensions everything seems fine with my .g4 grammar file. But when I execute the command to get the antlr/parser folder:
antlr4 -Dlanguage=Go -o parser items.g4
I get the following errors from the items_lexer.go and items_parser.go files:
Unresolved reference 'DeserializeFromUInt16'
Unresolved reference 'DecisionToState'
Unresolved reference 'DecisionToState'
The first one appears in the following line:
var lexerAtn = lexerDeserializer.DeserializeFromUInt16(serializedLexerAtn)
Which uses the var: lexerDeserializer declared in the previous line:
var lexerDeserializer = antlr.NewATNDeserializer(nil)
And here is the origin of the problem, directly from antlr lib for some reason the DeserializeFromUInt16 func is not founded. I already tried using another grammar. Im using ANTRL 4.7.2, and I made sure to had a jdk installed, the go get github.com/antlr/antlr4/runtime/Go/antlr, and the .jar folder in my project.
I would appreciate any kind of suggestions about how can I free my files from errors.
I want to make a simple example of calling C code from Go with CGO. But for some reason I can't achieve desired. Compilation fails with the following error:
go build main.go
# awesomeProject1/print
duplicate symbol '_do_print' in:
$WORK/b002/_x002.o
$WORK/b002/_x003.o
ld: 1 duplicate symbol for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
The code:
// print/print.c
#include <stdio.h>
void do_print(char * x){
printf("%s", x);
}
// print/print.go
package print
// #include <print.c>
import "C"
func DoPrint() {
C.do_print(C.CString("Hello!"))
}
// main.go
package main
import "awesomeProject1/print"
func main() {
print.DoPrint()
}
If I make do_print function static it compiles but I wouldn't be able to do that for 3rd party code I want to integrate with later.
Am I missing some essential piece from documentation? Tutorials are all alike and claim to work where my example fails. Please help!
Go version 1.16.4
There are two things going on here:
go build compiles *.c in addition to *.go1
#include <print.c> is exactly equivalent to overwriting the include statement with the contents of print.c
As a result, you are compiling the contents of print.c twice: once when print.c is compiled by CC, and once when print.go is compiled by CGo. Thus, the object files for print.c and print.go each contain all of the exported symbols defined in print.c. So you get two copies of do_print. Making do_print static works because a function declared as static will not be exported by CC.
Including a .c file (e.g. #include <file.c>) is pretty much always a bad idea. If you have a legitimate reason to #include function bodies, the convention is to use a header (.h) file. This is what C++ templating libraries do (like Boost, IIRC). Because C files normally are not included, and H files normally are included, including a C file subverts expectations and thus is likely to cause confusion and problems.
1 IIRC, Go will compile C files in a package if any of the Go files in the package import "C". That is, if no Go files in a package use CGo, Go will ignore C files in that package.
I have 2 project, the first, name as A, there is a submodule a imported sqlite3(github.com/mattn/go-sqlite3). Another B project import A's submodule a, and in another submodule b, it also import the same sqlite3.
Both A and B put there imports under vendor dir(managed by govendor). My Golang version is go version go1.12 linux/amd64.
While build B (go build main.go), throwing following errors(too many, part of them):
/usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/tmp/go-link-281256755/000029.o: In function `callbackTrampoline':
/tmp/go-build/_cgo_export.c:25: multiple definition of `callbackTrampoline'
/tmp/go-link-281256755/000005.o:/tmp/go-build/_cgo_export.c:25: first defined here
/tmp/go-link-281256755/000029.o: In function `stepTrampoline':
...
/home/xxx/go/src/gitlab.xxxxxxxxx.com/xxxxxxxxx-tools/A/vendor/github.com/mattn/go-sqlite3/sqlite3.go:129: multiple definition of `_sqlite3_result_text'
/tmp/go-link-281256755/000009.o:/home/xxx/go/src/gitlab.xxxxxxxxx.com/xxxxxxxxx-tools/A/vendor/github.com/mattn/go-sqlite3/sqlite3.go:129: first defined here
/tmp/go-link-281256755/000033.o: In function `_sqlite3_result_blob':
...
But building A works well. To testing the error, I started following demo, also with vendor inited by govendor, and build ok.
package main
import (
"database/sql"
"fmt"
"gitlab.xxxxxxxxx.com/xxxxxxxxxxxxxxx/A/a"
_ "github.com/mattn/go-sqlite3"
)
func main() {
fmt.Println(a.ModuleVariable) // use submodule `a` just like B is doing
_, _ = sql.Open(`sqlite3`, `test.db`) // use sqlite too
}
I think the compiler first compile A's sqlite3, objects are created under /tmp/go-link-281256755/000005.o (but no this dir after building), then compile B's importing of sqlite3 and also create a object contains the same-name function, then the compiler find 2 same-name symbols, the linking failed.
How to fix these situation? Is there any golang env settings to avoid these?
After I remove the sqlite3 package under vendor both of A and B, they both use the sqlite3 under ~/go/src/github.com/mattn/go-sqlite3/, they all build ok. But I can't do these, due to project A's deploy platform, I must put all dependencies under vendor, is there any another option to use multiple import with the same package?
For the cgo's issue of linking error "multiple definition of ...", the (work-around) solution is dependent on the nature of the linked C codes:
If two Go packages link to the same C codes (libraries), you should pass option --allow-multiple-definition to the linker (see ld man page), either via command options
go build --ldflags '-extldflags "-Wl,--allow-multiple-definition"',
or via #cgo directive in Go source of the packages linking to C codes:
//#cgo LDFLAGS: -Wl,--allow-multiple-definition
import "C"
If two Go packages link to different C codes containing some functions & variables with the same names, you should refactor those C codes:
Make sure to put keyword static to all declarations whose usage is within that C object only (not intended to be linked to Go nor to other C objects).
Find some way to do name mangling or put those duplicated identifiers into different namespaces (like in C++). It would be better if cgo supported some mechanism to do automatic namespacing using Go package name, but until now (2020) you must do it yourself. The C preprocessor's "token pasting" operator ## may help in this namespacing task. Eg.
//File: my_package1.h
#define NS(id) my_package1_ ## id
void NS(my_function1)(int);
void NS(my_function2)(float);
char NS(my_shared_var);
If you have any C function definition in the Go source, like in this question, you must move those definitions into a separate C source file under the same package folder, leaving only declarations in the Go source.
I have a CMake setup to link together 4 static libraries and 1 shared one into a top level shared library (Let's call it Top.dll). This will work fine except for one thing. I have a module definition file that expresses which symbols should be public. Top.dll builds and so far so good.
Now when I try to link an executable with Top.dll via CMake I get linker errors for every public symbol claiming it is defined in two places (Top.dll and the static library in which it was actually defined) even though Top.dll contains no original definitions of its own. If I remove the static library then as expected I get unresolved symbol errors. If I remove the module definition file, I get the same. It seems like it is either there zero times or twice. Is there some setting I am missing here? I don't think I'm using CMake in a non-basic way...
UPDATE An explanation via CMake
# setup the lib
add_subdirectory(vendor/A) #shared library
add_subdirectory(vendor/B) #static library
add_subdirectory(vendor/C) #static library
add_library(Top SHARED ${ALL_SRC_FILES})
target_link_libraries(Top A B C)
set_target_properties(Top PROPERTIES LINK_FLAGS
"/def:${PROJECT_SOURCE_DIR}/definitions.def") #contains symbols from B
add_subdirectory(C/Tests)
# CMakeLists.txt from C/Tests
add_executable(Tests ${SRC_FILES})
target_link_libraries(Tests Top)
The above is simplified, but I will get errors like the following at the point that the C/tests project is compiled:
B.lib(xxx.obj) : error LNK2005: _ABC already defined in Top.lib(Top.dll)
If I remove B.lib from the target_link_libraries call, then as expected I get unresolved symbols. If I remove the /def line, same result.
I've been able to get around this by setting the target_link_libraries of B and C to private. This may or may not be the correct solution and I will wait for other answers. If I don't do this it appears that the dependency is carried up to the final executable (So it links to both Top.dll and B.lib, etc).
I'm doing low-level IO (for library bindings) in Haskell and am experiencing a segfault. I would like to use GHCi's :break to figure out what's going on, but here's what happens:
> import SDL
> :break SDL.setPaletteColors
cannot set breakpoint on setPaletteColors: module SDL.Video.Renderer is not interpreted
Since the offending code is not inside my own modules, but rather inside a module in an external package, it's loaded as compiled code and apparently I can't use :break on compiled modules.
GHCi manual confirms this and provides a hint:
There is one major restriction: breakpoints and single-stepping are only available in interpreted modules; compiled code is invisible to the debugger[5].
[5] Note that packages only contain compiled code, so debugging a package requires finding its source and loading that directly.
Let's try it directly:
> :load some_path/sdl2/src/SDL/Video/Renderer.hs
some_path/sdl2/src/SDL/Video/Renderer.hs:101:8:
Could not find module ‘Control.Monad.IO.Class’
It is a member of the hidden package ‘transformers-0.3.0.0’.
Perhaps you need to add ‘transformers’ to the build-depends in your .cabal file.
Use -v to see a list of the files searched for.
I can add the dependencies to my .cabal file, but this already feels wrong. Once I've done that:
> :load some_path/sdl2/src/SDL/Video/Renderer.hs
some_path/sdl2/src/SDL/Video/Renderer.hs:119:8:
Could not find module ‘SDL.Internal.Numbered’
it is a hidden module in the package ‘sdl2-2.0.0’
Use -v to see a list of the files searched for.
I could make those modules public (probably? by modifying the package .cabal?), but at this point it seems a really awkward way to do things and I didn't pursue it further.
EDIT:
I actually tried that and got the baffling result:
> :load some_path/sdl2/src/SDL/Video/Renderer.hs
[1 of 1] Compiling SDL.Video.Renderer ( some_path/sdl2/src/SDL/Video/Renderer.hs, interpreted )
Ok, modules loaded: SDL.Video.Renderer.
> :break SDL.setPaletteColors
cannot set breakpoint on SDL.setPaletteColors: module SDL.Video.Renderer is not interpreted
My (uneducated) guess: it's because the external module is still linked to my code as a binary, and loading it dynamically in interpreted mode doesn't change that.
So, to sum up the question: what is a good way to debug IO in an external package?
Additional notes:
I do have the source to the package I need to debug; in fact, it's been added to the project with cabal sandbox add-source
An alternative option to using GHCi would be to add traces to the package source, but this is an unfortunate option, since it involves recompilation of the package on each modification (whenever I need more information about the execution and modify the traces), and that takes a really long time. Interactive debugging with GHCi seems a better tool for this job, if only I knew how to use it.
Stack has some support for this. Running stack ghci --load-local-deps $TARGET will load your project and any dependencies that are in the packages field of stack.yaml, including if they're marked as extra-deps. Breakpoints will work then. You can debug a dependency in GHCi by running stack unpack $PACKAGE and adding it to packages in stack.yaml.
This is not a panacea however. If the packages have conflicting package-global language extensions (or other dynamic flags) or module name clashes it won't work. For example, if your top-level package has default-extensions: NoImplicitPrelude and your dependencies don't, they won't have a prelude imported and will almost certainly not load. See this GHC bug.