I'm working on a private build system that produces debian packages using a toplevel Makefile. By including /usr/share/dpkg/default.mk, I get the package name and version from debian/control and debian/changelog. There is also $(DEB_TARGET_ARCH), which would be useful, except it isn't smart enought to notice if the package is "all" architecture.
Is there an official way to get the package arch at build time?
As it stands I need 2 versions. One for arch (e.g. "amd64") packages and one for "all"
Here is the top of my Makefile
include /usr/share/dpkg/default.mk
DEB_PKG := ../$(DEB_SOURCE)_$(DEB_VERSION)_all.deb
my_stuff := blabity blah and so forth
all:
deb: $(DEB_PKG)
$(DEB_PKG): debian $(my_stuff)
gbp buildpackage --git-ignore-new
I don't think you can determine in the general case whether a Debian package build will produce only architecture-independent packages. You have to run debian/rules binary and see what that produces. Even packages listed in debian/control might not actually be built in the end.
Related
Say I have the following basic CMakeLists.txt file.
target_include_directories( addnum
PUBLIC
"${CMAKE_CURRENT_SOURCE_DIR}/include"
)
add_executable(addnumapp src/main.cpp)
target_link_libraries(addnumapp addnum)
SET(CPACK_GENERATOR "DEB")
set(CPACK_DEBIAN_PACKAGE_DEPENDS "g++ (>= 7), libffi-dev, libncurses5-dev, libsqlite3-dev, mcpp, zlib1g-dev")
SET(CPACK_DEBIAN_PACKAGE_MAINTAINER "Some person")
INCLUDE(CPack)
To create a "deb" package, I create a build directory and run the following command to build my project:
cmake -S . -B ./build
I then run the cpack command from within the build directory and a ".deb" package is generated. This is great to generate a single ".deb" package; however, I would like to generate multiple packages using cmake and cpack and I'm unsure what the best way to go about this is.
For example, how do I generate ".deb" packages for all the supported Debian architectures?
From my research, I'm aware of the following command, which allows you to specify an architecture:
set(CPACK_DEBIAN_PACKAGE_ARCHITECTURE, "i386")
However, this only works if you're generating a single Debian package through the method I've outlined above. How does one generate multiple Debian packages for all the different architectures?
My second question is about generating multiple packages for different Operating Systems. For example, the above CMakeLists.txt file just generates a Debian package; however, I would also like to generate packages for MacOS and Windows.
From my research, I'm aware that the following edits to my CMakeLists.txt file should theoretically be the minimal changes required for me to generate a ".dmg" package for MacOS in addition to the Debian package I also want generated.
...
SET(CPACK_GENERATOR "DEB;DragNDrop") # modification here
set(CPACK_DEBIAN_PACKAGE_DEPENDS "g++ (>= 7), libffi-dev, libncurses5-dev, libsqlite3-dev, mcpp, zlib1g-dev")
SET(CPACK_DEBIAN_PACKAGE_ARCHITECTURE i386)
SET(CPACK_DEBIAN_PACKAGE_MAINTAINER "Some person") #required
INCLUDE(CPack)
INCLUDE(CPackDMG) # addition here
However, when I run cmake -S . -B ./build on this modified CMakeLists.txt file (from an Ubuntu OS), I get the following error:
CMake Error at CMakeLists.txt:28 (INCLUDE): INCLUDE could not find
requested file:
CPackDMG
-- Configuring incomplete, errors occurred!
Is this error because I'm trying to generate a MacOS "dmg" package from Ubuntu or is it because I'm missing some cpack dependency?
Further, is this the best way to generate different packages for different platforms?
To summarise, my two questions are (1) how to generate multiple packages for different architectures and (2) how to generate multiple packages for different platforms? (Please bear in mind that I'm very new to C++, CMake and CPack.)
(1) how to generate multiple packages for different architectures
Just use different build environments (hardware, docker containers, virtual machines, ...) or use cross-compilation and toolchain files.
(2) how to generate multiple packages for different platforms?
CPackDMG is an internal CPack module. You don't need to include it. Your CMakeLists.txt should use only CPack and set desired variables before. Set CPACK_GENERATOR to the list of package types you want and the generator-specific variables. However, some generators use external tools -- e.g., to build RPM package your environment requires working rmbuild tool... meaning that it'll be hard to build RPM package within a Debian distro %) In other words, you most likely fail to build DMG when building your package in Linux/Windows :)
I've installed Go 1.13.4 in Debian Linux 10 using brew, package golang (previously I've removed golang-1.11 installed via apt). It follows brew info output.
$ brew info golang
go: stable 1.13.4 (bottled), HEAD
Open source programming language to build simple/reliable/efficient software
https://golang.org
/home/linuxbrew/.linuxbrew/Cellar/go/1.13.4 (9,271 files, 408.1MB) *
Poured from bottle on 2019-12-07 at 14:31:52
From: https://github.com/Homebrew/linuxbrew-core/blob/master/Formula/go.rb
==> Requirements
Required: macOS is required ✔
==> Options
--HEAD
Install HEAD version
==> Analytics
install: 1,571 (30 days), 8,628 (90 days), 31,650 (365 days)
install-on-request: 784 (30 days), 4,096 (90 days), 13,267 (365 days)
build-error: 0 (30 days)
When I try to execute go build on a package with code calls native OS functions, GO compiler tells that it's unable to find gcc-5 command as presented below.
$ go build
# _/home/giacomo/src/goproc/process
exec: "gcc-5": executable file not found in $PATH
So I've installed latest GCC (9.2.1) from testing (deb http://ftp.us.debian.org/debian testing main contrib non-free) repository and created a symbolic link to /usr/bin/x86_64-linux-gnu-gcc-9.
This solved the problem. But the question is: is this the correct way to fix the problem? Or is there a place to properly configure the GCC used by golang?
Any clarification very appreciated!
Since it looks like you have installed go with linuxbrew, you can also install gcc-5:
brew install gcc#5
This solves the problem for me cleanly.
got this issue in linuxbrew. to fix run:
go env -w CC=gcc CXX="g++"
This uses the system gcc/g++ instead of the ones from linuxbrew
Another option to try is setting the CC environment variable to gcc.
(Not really an answer as what you asked is hardly a real question—see below.)
There's multiple points which seem wrong with your situation; let's consider them all.
The first thing to consider is that "Go" means two things: a language which has certain syntax and semantics, and one of implementations of it.
Go-the-language has at least two mature implementations: the "reference" one—available from the Go's main site—and another one—a part of the GCC. Stock contemporary Debian distribution (Debian 10, "Buster") ships both of them: golang-go is the former and gccgo is the latter.
As you can see, it's not clear which one you've installed, in the first place.
The second thing to consider is that the reference implementation (dubbed gc by its original developers for the reasons I forgot) is completely free-standing (and even self-bootstrapping) and does not use any C compiler (from GCC or other) to build Go code. Conversely, gccgo naturally uses other parts of the GCC toolchain to build the Go code.
Still, Go code features a special subsystem called cgo which can be use to interface Go code with code written in C (and with compiled libraries adhering to the C API. When building a program which uses cgo, both Go suites do rely on at least a C compiler (and may be linker—I don't know this for sure), and by default gc expects to be able to use a GCC-compatible compiler, and naturally, gccgo uses the C compiler of GCC, too.
Let's now recap a bit.
As you should supposedly see by now, you at least ought to sort several things for you:
What implementation of Go you're talking about (and/or want installed)?
Is what you have installed by brew is really what you intended to get?
Does the code you're trying to build use cgo?
The fourth thing to consider is why on Earth have you decided to use a kludge invented to help Mac users compensate for the lack of package management system on their platform to deal with packages in Debian.
Debian already ships Go; if you're not satisfied with the its packaged version (1.11), it looks simpler to merely grab the latest-and-greatest binary package from https://golang.org/dl, unack it and use.
Another thing to consider is that since version 1.5 Go is self-bootstrapping because it's written in Go, so if you want to have the latest upstream version, you can just apt install golang-go — to have some version of Go installed — and then use it to build the latest one, like this:
Get the Go source code:
$ cd ~
$ mkdir golang
$ git clone https://github.com/golang/go golang
Pick the version you want to build:
$ cd golang
$ git checkout go1.13.5
$ cd src
$ ./make.bash
Then make sure you have /home/user/golang/bin listed in your $PATH.
Note that building Go is lightning-fast: on a laptop with SSD it builds from cold start in under a minute.
An answer based on nemo's answer.
If you have Go installed via the system's package manager
and you installed linuxbrew on that system
and somehow, in my case I installed buf via brew,
linuxbrew's go package was pulled in,
you system's Go package was replaced by brew's Go package.
So in order to use your system's Go package again you have to uninstall brew's Go package.
brew uninstall go
or if you want to keep brew's Go package, do what nemo answered:
go env -w CC=gcc CXX="g++"
I have a repository with a group of nested go packages organized as follows:
$GOPATH/src/
- mypackage/common/utils.go
- mypackage/app1/main.go
- mypackage/app2/main.go
...
It compiles to a handful of binaries. For releasing, I'm cross-compiling for a multitude of platforms/archs (I deploy repeatedly with a different GOOS and GOARCH). I'm trying to write the compilation results to a directory of my choice, but I'm fighting with the toolchain.
I can:
Combine GOBIN and go install when compiling for my own architecture
(i.e. not cross compiling):
# build + output all binaries in /somedir/bin
# this works great when compiling for my local architecture.
GOBIN=/somedir/bin go install mypackage/app1 mypackage/app2 mypackage/app3
But unfortunately, GOBIN conflicts with cross-compilation e.g:
# Throws: "cannot install cross-compiled binaries when GOBIN is set"
GOBIN=/somedir/bin GOARCH=amdm64 GOOS=darwin go install mypackage/app1 mypackage/app2
Use go build with GOOS=X and GOARCH=Y for each subpackage
# cross compile one of the binaries
cd /output/darwin-386-bin/ && \
GOOS=darwin GOARCH=386 go build mypackage/app1
# but if given multiple binaries, there is no output. (as documented)
GOOS=darwin GOARCH=386 go build mypackage/app1 mypackage/app2
When go build is given multiple packages, it no longer emits binaries -- it just checks that the code compiles. To emit all the binaries, it seems I have to run go build once for each subpackage, so it takes longer, esp when building with -a.
Another possible issue with using go build like this, is that it's potentially mixing binaries and intermediary results across multiple architectures in the same workspace. But maybe that's just a matter of taste. Hopefully the toolchain keeps cached results separate for different architectures.
How is this handled in practice? Should I treat each subpackage as an individual package, despite their shared common code (would it then be safe to build them all in parallel)?
Related articles:
go build vs go install: https://groups.google.com/forum/#!topic/golang-nuts/s8Csz3-7EXA (Little info on cross-compilation, or multipackage setups)
cross compile with go 1.5.: https://dave.cheney.net/2015/08/22/cross-compilation-with-go-1-5 (does not specifically address multipackage / common code compilation. Also, the statement about having to recompile the standard library into /usr/local/go/pkg seems to no longer hold in 1.9.1, as far as I can tell)
Go build multiple nested packages: Go build multiple/nested packages? . ( The accepted answer, go build ./... behaves the same as passing multiple packages in a single build command, and therefore outputs no binaries. The thread also does not cover cross compilation)
What I'm about to suggest feels like a hack at best, but it might work for you if you're cross-compiling stuff in a container or an isolated build environment.
You can drop the GOBIN from the install command:
# I'm not on this platform, so this is a cross compile
GOOS=darwin GOARCH=amd64 go install mypackage/app1 mypackage/app2
Assuming your package is in $GOPATH/src/mypackage the above command will install the two binaries to:
$GOPATH/bin/darwin_amd64/{app1, app2}
and the .a compile dependency files in:
$GOPATH/pkg/darwin_amd64/mypackage/{app1,app2,common}
If you're running this in a for loop for all the platforms you plan on supporting, one nuisance with this process is that when you pass GOOS=x GOARCH=y matching your local architecture (i.e. not cross compiling), then the executables will be placed directly in $GOPATH/bin/ and not $GOPATH/bin/x_y/.
(packages on the other hand always end up in $GOPATH/pkg/x_y/, cross compilation or not).
To determine your local architecture, you can follow steps in this answer: https://stackoverflow.com/a/35669816/5556676 .
Simulating GOBIN=/foo/bin GOOS=x GOARCH=y go install mypackage/app{1,2,3}
Changing $GOPATH has little effect on where go install writes output. With some go commands, like go get, you tweak where packages are installed by adding a new component at the front of $GOPATH. But in go 1.9, go install will always copy binaries in the bin/ folder that's a sibling of the src/ which contains the packages to install.
Instead of using GOBIN=/foo/bin, you can pretend your source files are in /foo/src/mypackage (you may use a symlink), and then do GOPATH=/foo GOOS=x GOARCH=y go install mypackage/app{1,2,3}. This will put binaries in /foo/bin, because of the behavior I describe in the previous paragraph.
It's probably simpler to just grab the binaries from where you expect them to be though, than mucking with copying source trees around.
If there a relatively simple way to make go + libxml2 + gokogiri work on windows?
I mean that I may be can install it (but at the moment I can not, stuck with Package libxml-2.0 was not found in the pkg-config search path), but then I need to provide my utilite to other people, who will never be able (or would wish ) to install lall libxml2 dependencies, modify PATH etc on windows...
It work flawless on Ubuntu...
I found this https://github.com/moovweb/gokogiri/issues/49 thats funny with installation of Gimp 2 (what?!), but I still cannot make it run with such error, I guess might be issue with PATH, but all PATH are set
$ go get github.com/moovweb/gokogiri
# github.com/moovweb/gokogiri/help
Documents\go\src\github.com\moovweb\gokogiri\help\help.go:6:25: fatal error: lib
xml/tree.h: No such file or directory
#include <libxml/tree.h>
^
compilation terminated.
# github.com/moovweb/gokogiri/xpath
Documents\go\src\github.com\moovweb\gokogiri\xpath\expression.go:4:26: fatal err
or: libxml/xpath.h: No such file or directory
#include <libxml/xpath.h>
^
compilation terminated.
You are struggling because it is hard to combine packages that were built by different people for different purposes and get your environment set up correctly. I think it is best to use MSYS2, an environment for Windows that provides a consistent set of packages for things like gcc, go, libxml2, and iconv. MSYS2 has a package manager (pacman) that helps you easily install them and keep them updated.
I don't do much programming with Go, but I am familiar with MSYS2 and it seems like I was able to get gokogiri installed using MSYS2. You should open MSYS2's "MinGW-w64 Win64 Shell" from the Start menu (mingw64_shell.bat), and try running these commands:
pacman -S mingw-w64-x86_64-{gcc,go,libxml2,iconv}
export GOROOT=/mingw64/
export GOPATH=/c/Users/David/Documents/goproj/
mkdir -p $GOPATH
go get github.com/moovweb/gokogiri
I think GOPATH should be set to the directory of your project. If you run into an error, it might be because some pacman package is required that I didn't list here.
The string mingw-w64-x86_64-{gcc,go,libxml2,iconv} gets expanded by Bash into the following list of packages:
mingw-w64-x86_64-gcc
mingw-w64-x86_64-go
mingw-w64-x86_64-libxml2
mingw-w64-x86_64-iconv
If you are actually using 32-bit Windows, replace x86_64 with i686 in the instructions above.
If you are curious, the scripts for building those packages are here: https://github.com/Alexpux/MINGW-packages
As a disclaimer, I haven't actually compiled any go programs in MSYS2, so there could be big problems I am unaware of.
Also, one of the main developers of MSYS2 (alexpux) said this in the #msys2 IRC chat on 2015-06-21:
We not build go for a long time.
This package in very WIP state
Also see
https://github.com/Alexpux/MINGW-packages/issues/421
So you might need to fix some issues with the MSYS2 Go package and recompile it yourself to really make this work. But you have the PKGBUILD script that was used to build it, so maybe that will be less hard than what you are trying to do right now, which involves compiling/collecting every dependency of gokogiri.
MSYS2 would make your other installation of go, libxml2, and iconv obsolete. You can delete those things once you get your MSYS2 environment working.
If you are using visual studio and want to add dependency to your project then just install it using NuGet Package Manager it's easiest method.
Install command: Install-Package libxml2
How can I put my Go binary into a Debian package? Since Go is statically linked, I just have a single executable--I don't need a lot of complicated project metadata information. Is there a simple way to package the executable and resource files without going through the trauma of debuild?
I've looked all over for existing questions; however, all of my research turns up questions/answers about a .deb file containing the golang development environment (i.e., what you would get if you do sudo apt-get install golang-go).
Well. I think the only "trauma" of debuild is that it runs lintian after building the package, and it's lintian who tries to spot problems with your package.
So there are two ways to combat the situation:
Do not use debuild: this tool merely calls dpkg-buildpackage which really does the necessary powerlifting. The usual call to build a binary package is dpkg-buildpackage -us -uc -b. You still might call debuild for other purposes, like debuild clean for instance.
Add the so-called "lintian override" which can be used to make lintian turn a blind eye to selected problems with your package which, you insist, are not problems.
Both approaches imply that you do not attempt to build your application by the packaging tools but rather treat it as a blob which is just wrapped to a package. This would require slightly abstraining from the normal way debian/rules work (to not attempt to build anything).
Another solution which might be possible (and is really way more Debian-ish) is to try to use gcc-go (plus gold for linking): since it's a GCC front-end, this tool produces a dynamically-linked application (which links against libgo or something like this). I, personally, have no experience with it yet, and would only consider using it if you intend to try to push your package into the Debian proper.
Regarding the general question of packaging Go programs for Debian, you might find the following resources useful:
This thread started on go-nuts by one of Go for Debian packagers.
In particular, the first post in that thread links to this discussion on debian-devel.
The second thread on debian-devel regarding that same problem (it's a logical continuation of the former thread).
Update on 2015-10-15.
(Since this post appears to still be searched and found and studied by people I've decided to update it to better reflec the current state of affairs.)
Since then the situation with packaging Go apps and packages got improved dramatically, and it's possible to build a Debian package using "classic" Go (the so-called gc suite originating from Google) rather than gcc-go.
And there exist a good infrastructure for packages as well.
The key tool to use when debianizing a Go program now is dh-golang described here.
I've just been looking into this myself, and I'm basically there.
Synopsis
By 'borrowing' from the 'package' branch from one of Canonical's existing Go projects, you can build your package with dpkg-buildpackage.
install dependencies and grab a 'package' branch from another repo.
# I think this list of packages is enough. May need dpkg-dev aswell.
sudo apt-get install bzr debhelper build-essential golang-go
bzr branch lp:~niemeyer/cobzr/package mypackage-build
cd mypackage-build
Edit the metadata.
edit debian/control file (name, version, source). You may need to change the golang-stable dependency to golang-go.
The debian/control file is the manifest. Note the 'build dependencies' (Build-Depends: debhelper (>= 7.0.50~), golang-stable) and the 3 architectures. Using Ubuntu (without the gophers ppa), I had to change golang-stable to golang-go.
edit debian/rules file (put your package name in place of cobzr).
The debian/rules file is basically a 'make' file, and it shows how the package is built. In this case they are relying heavily on debhelper. Here they set up GOPATH, and invoke 'go install'.
Here's the magic 'go install' line:
cd $(GOPATH)/src && find * -name '*.go' -exec dirname {} \; | xargs -n1 go install
Also update the copyright file, readme, licence, etc.
Put your source inside the src folder. e.g.
git clone https://github.com/yourgithubusername/yourpackagename src/github.com/yourgithubusername/yourpackagename
or e.g.2
cp .../yourpackage/ src/
build the package
# -us -uc skips package signing.
dpkg-buildpackage -us -uc
This should produce a binary .deb file for your architecture, plus the 'source deb' (.tgz) and the source deb description file (.dsc).
More details
So, I realised that Canonical (the Ubuntu people) are using Go, and building .deb packages for some of their Go projects. Ubuntu is based on Debian, so for the most part the same approach should apply to both distributions (dependency names may vary slightly).
You'll find a few Go-based packages in Ubuntu's Launchpad repositories. So far I've found cobzr (git-style branching for bzr) and juju-core (a devops project, being ported from Python).
Both of these projects have both a 'trunk' and a 'package' branch, and you can see the debian/ folder inside the package branch. The 2 most important files here are debian/control and debian/rules - I have linked to 'browse source'.
Finally
Something I haven't covered is cross-compiling your package (to the other 2 architectures of the 3, 386/arm/amd64). Cross-compiling isn't too tricky in go (you need to build the toolchain for each target platform, and then set some ENV vars during 'go build'), and I've been working on a cross-compiler utility myself. Eventually I'll hopefully add .deb support into my utility, but first I need to crystallize this task.
Good luck. If you make any progress then please update my answer or add a comment. Thanks
Building deb or rpm packages from Go Applications is also very easy with fpm.
Grab it from rubygems:
gem install fpm
After building you binary, e.g. foobar, you can package it like this:
fpm -s dir -t deb -n foobar -v 0.0.1 foobar=/usr/bin/
fpm supports all sorts of advanced packaging options.
There is an official Debian policy document describing the packaging procedure for Go: https://go-team.pages.debian.net/packaging.html
For libraries: Use dh-make-golang to create a package skeleton. Name your package with a name derived from import path, with a -dev suffix, e.g. golang-github-lib-pq-dev. Specify the dependencies ont Depends: line. (These are source dependencies for building, not binary dependencies for running, since Go statically links all source.)
Installing the library package will install its source code to /usr/share/golang/src (possibly, the compiled libraries could go into .../pkg). Building depending Go packages will use the artifacts from those system-wide locations.
For executables: Use dh-golang to create the package. Specify dependencies in Build-Depends: line (see above regarding packaging the dependencies).
I recently discovered https://packager.io/ - I'm quite happy with what they're doing. Maybe open up one of the packages to see what they're doing?