How to inspect the content of Go package/object files (.a, .o)
The only thing I found is showing the full disassembly with go tool objdump.
But how to show the file structure, like imported and exported symbols, code, data and other sections, other metadata etc.?
Compiled Go packages are stored in Unix ar archive files, so you can extract the files within them using ar or go tool pack x pkg.a.
Package files contain compiled code (_go_.o), used by the linker to build binaries, and export data (__.PKGDEF), used by the compiler when building dependent packages. If the package contained assembly files or cgo code, there may be additional .o files, too.
You can use golang.org/x/tools/go/gcexportdata to read export data into a format that can be understood by go/types.
It sounds like you've already found go tool objdump for disassembly. You might also find go tool nm useful for listing symbols.
Related
I'm trying to understand how the source code for third-party dependencies is or is not compiled into my Go binary. I'm building in a Docker container, so I can see precisely what's fetched for my build without interference from other builds.
After my go build completes I see source code files for several dependencies under go/pkg/mod/$module#$version directories. The Module cache documentation tells me that these directories contain "extracted contents of a module .zip file. This serves as a module root directory for a downloaded module." My best guess is that the presence of extracted source code for these dependencies indicates that "yes, these dependencies are definitely compiled into your binary."
I also see many more dependencies pulled into go/pkg/mod/cache/download/$module directories. The Module cache documentation tells me that this directory contains "files downloaded from module proxies and files derived from version control systems," which I don't fully understand. As far as I can see, these files do not include any extracted source code, though there are several .zip files that I assume contain the source. For the most part these files seem to be .mod files that just contain text representing some sort of dependency graph.
My question is: if a third-party dependency has module files under go/pkg/mod/cache/download but no source code under go/pkg/mod/$module#$version, does that mean that dependency's code was NOT compiled into my Go binary?
I don't understand why the Go build pulls in all these module files but only has extracted source code for some of the third-party modules. Perhaps Go preemptively parses and pulls module information for the full transitive set of modules referenced from the modules my first-party code imports, but perhaps many of those modules don't end up being needed for my binary's compile + build process and therefore don't get extracted. If that's not true and the answer to my question is no, then I don't understand how or why my binary can link in those dependencies without go build fetching their source code.
As mentioned in "Compile and install packages and dependencies"
Compiled packages are cached automatically.
GOPATH and Module includes:
When using modules, GOPATH is no longer used for resolving imports.
However, it is still used to store downloaded source code (in GOPATH/pkg/mod) and compiled commands (in GOPATH/bin).
So if you see sources in pkg/mod which are not in pkg/mod/cache, try a go mod tidy
add missing and remove unused modules
From there, you should have the same modules between sources (pkg/mod) and compiled modules (pkg/mod/cache)
Based on the OP's comment
I need to know exactly what's included in the binary for compliance reasons.
I would recommend a completely different approach: dumping the list of symbols contained in the binary and then correlating it with whatever information is needed.
The command
go tool nm -type /path/to/the/executable/image/file
would dump the symbols — names of the functions — whose code was taken from both the standard library packages, 3rd-party and/or vendored packages and internal packages, compiled and linked into the binary, and print to its standard output stream a sequence of lines
address type name
which you can then process programmatically.
Another approach you might employ is to use go list with various flags to query the program's source code about the packages and/or modules which will be used when building: whatever that command outputs describing the full dependency graph of the source code is whatever go build will use when building — provided the source code is not changed between these calls.
Yet another possibility is to build the program using go build -x, save the debug trace it produces on its standard error stream and parsing it for exact module names the command reported as used during building.
I am working on a go library which I intend to distribute as a binary artifact. Now I am aware that there are different -buildmode options and archive should pretty much do that for me and go 1.7+'s //go:binary-only-package makes the trick official.
However, when I build my library the resulting archive (*.a) contains my library only, not any dependencies. I am in fact having a dependency which itself is binary only and comes as a statically linked archive itself (it's a C library which I am integrating with).
Using proper native code archives I am actually able to assemble such a fat archive using ar or libtool trickery - but these tools do not really work with my go artifacts.
Is there a way I can distribute a single *.a file or do I have to resort to packaging multiple archives into, say, a zip file which resembles the directory structure in $GOPATH and tell the my client's developers to just unzip that into their $GOPATH and be done with it?
I wrote a function (not! main) and prompted go install. This command generated a path and a package in my pkg-directory. I tested the function by using it in a main function, generated the .exe and everything worked just fine.
After that, I wanted to see if I understood the concept of packages in go correctly and deleted the source file of the function in the src-directory and deleted the main .exe. I did not remove the package file in my pkg-directory. Then I tried to go install the main .exe again, but it didn't work: "package can not be found". I obviously missunderstood the whole concept because I thought I could use the packages in pkg without the source files in src. If my conclusion was correct, why do I need the "pkg" directory at all?
For more explanation take a look at this picture please:
In /bin is the binary code of the main function "hello". This main function also contains the function "reverse" of the "stringutil" package.
By generating the "hello.exe", Go also generates the package "stringutil" into pkg.
My question is: Should I not be able to delete "reverse.go" in src and still be able to use the same function because it was already put into pkg?
Is it just the way the AST works now that they've rewritten the compiler in go? It checks for GOPATH/src/**/.go when it parses the "imports", then when the linker goes to build the final binary, it'll go check pkg. So the compiler errors out first when trying to feed the ast to the assembler because of the incomplete source tree.
Thank you very much!!!
It is true that the pkg dir is usually used as a cache directory, but also It is possible to use packages without having the source code available, with a feature named binary only packages AFAIK It was implemented since Go 1.7.
However, there's a caveat for this approach: Versions of the compiler used to build the package and the compiler to 'use' the package to generate a new library/executable must match. Also, the files must match the pair of os/architecture to build against. If you want cross-compiling, you'll need to distribute your package to every pair of os/architecture you'll want to build.
This project has a demo for the aforementioned feature.
I hope my explanation was detailed enough :)
The pkg dir is just a local cache for compiling, it's not something you can depend upon and it doesn't replace .go files, it's not a static lib dir but a temp build products dir. It speeds up compilation, so you don't need the pkg dir, but compiles might be slower if it is empty, and it is for the compiler's use, not yours.
As you've discovered, you do need the src dir.
To link a static library to your project, you could copy out that .a file and use ldflags to link but then you lose all the nice things like cross-compiling and having the entire source for your app, so unlike c for example people don't typically do that.
I'm messing around with compilers, .obj files, assembly, etc. The .obj file contains info that eventually ends up in the PDB, but I can't find any reference to the format that's used within the debug sections of the .obj file. (I have, however, found a reference to the COFF file format -- so I already know about that).
So: What's the format of the .debug$S and .debug$T sections when the source C file is compiled with the /Zi flag?
This information isn't published (the format used for native PDBs). If you can link the object file into an executable using "link" there are windows "debugging apis" you can use to "interrogate" symbols in an image. However, the format used for object files is not made publicly available.
You could try and reverse engineer it. If you find any info, please share it.
So I used the VisualMagick tool to set up a Static Library project and compiled all of the ImageMagick Source to static lib files. Then I created a new solution and moved all those libs and the needed .h files to my lib folder in my new solution. My cpp file that I want to use the lib files compiles fine, even with #include Magick++.h in the header, until I add any references to things in imageMagick. Like if I say Magick::Image image; It will give me LNK1120. I have added the lib folder to my projects Additional Library Locations (or something like that) in the solution properties. I am new to the whole Linking language thing, coming mostly from a Python/Java background. Any suggestions? I have tried a brute google search and tried a lot of the suggestions I have seen.
I'll put the foregoing comments interchange in the form of a real answer:
To convince VS20xx to link your app with some non-standard library, including maybe a new library that you just built:
Under the VS main-menu "Project" tab, take "<your-project-name> Properties ..." and then
First, tell the linker where to look for a lib (like make -L):
Linker --> General
In the "Additional Library Directories" edit box, give the paths -- just the directories -- where lib files live.
Second, tell the linker what library files you want to lilnk with (like make -l):
Linker --> Input and then
In the "Additional Dependencies" edit box, add the space-separated unadorned lib-file name(s), no quotes needed, like:
Additional Dependencies mysqlclient.lib libcurl.lib mynewlib.lib
That should be it. (yeah, suuuuuuuuure :-)