How to get CMake/CPack to generate multiple packages for different OSs and different architectures within each OS? - makefile

Say I have the following basic CMakeLists.txt file.
target_include_directories( addnum
PUBLIC
"${CMAKE_CURRENT_SOURCE_DIR}/include"
)
add_executable(addnumapp src/main.cpp)
target_link_libraries(addnumapp addnum)
SET(CPACK_GENERATOR "DEB")
set(CPACK_DEBIAN_PACKAGE_DEPENDS "g++ (>= 7), libffi-dev, libncurses5-dev, libsqlite3-dev, mcpp, zlib1g-dev")
SET(CPACK_DEBIAN_PACKAGE_MAINTAINER "Some person")
INCLUDE(CPack)
To create a "deb" package, I create a build directory and run the following command to build my project:
cmake -S . -B ./build
I then run the cpack command from within the build directory and a ".deb" package is generated. This is great to generate a single ".deb" package; however, I would like to generate multiple packages using cmake and cpack and I'm unsure what the best way to go about this is.
For example, how do I generate ".deb" packages for all the supported Debian architectures?
From my research, I'm aware of the following command, which allows you to specify an architecture:
set(CPACK_DEBIAN_PACKAGE_ARCHITECTURE, "i386")
However, this only works if you're generating a single Debian package through the method I've outlined above. How does one generate multiple Debian packages for all the different architectures?
My second question is about generating multiple packages for different Operating Systems. For example, the above CMakeLists.txt file just generates a Debian package; however, I would also like to generate packages for MacOS and Windows.
From my research, I'm aware that the following edits to my CMakeLists.txt file should theoretically be the minimal changes required for me to generate a ".dmg" package for MacOS in addition to the Debian package I also want generated.
...
SET(CPACK_GENERATOR "DEB;DragNDrop") # modification here
set(CPACK_DEBIAN_PACKAGE_DEPENDS "g++ (>= 7), libffi-dev, libncurses5-dev, libsqlite3-dev, mcpp, zlib1g-dev")
SET(CPACK_DEBIAN_PACKAGE_ARCHITECTURE i386)
SET(CPACK_DEBIAN_PACKAGE_MAINTAINER "Some person") #required
INCLUDE(CPack)
INCLUDE(CPackDMG) # addition here
However, when I run cmake -S . -B ./build on this modified CMakeLists.txt file (from an Ubuntu OS), I get the following error:
CMake Error at CMakeLists.txt:28 (INCLUDE): INCLUDE could not find
requested file:
CPackDMG
-- Configuring incomplete, errors occurred!
Is this error because I'm trying to generate a MacOS "dmg" package from Ubuntu or is it because I'm missing some cpack dependency?
Further, is this the best way to generate different packages for different platforms?
To summarise, my two questions are (1) how to generate multiple packages for different architectures and (2) how to generate multiple packages for different platforms? (Please bear in mind that I'm very new to C++, CMake and CPack.)

(1) how to generate multiple packages for different architectures
Just use different build environments (hardware, docker containers, virtual machines, ...) or use cross-compilation and toolchain files.
(2) how to generate multiple packages for different platforms?
CPackDMG is an internal CPack module. You don't need to include it. Your CMakeLists.txt should use only CPack and set desired variables before. Set CPACK_GENERATOR to the list of package types you want and the generator-specific variables. However, some generators use external tools -- e.g., to build RPM package your environment requires working rmbuild tool... meaning that it'll be hard to build RPM package within a Debian distro %) In other words, you most likely fail to build DMG when building your package in Linux/Windows :)

Related

Conan boost dlls not installed on Windows -> runtime exception

I have a conan recipe of a package, named Package, that requires boost as a shared library:
def requirements(self):
self.requires("boost/1.79.0#")
self.options["boost"].shared = True
self.options["boost"].bzip2 = False
self.options["boost"].without_stacktrace = True
The used generators are CMakeDeps and CMakeToolchain.
def generate(self):
tc = CMakeToolchain(self)
tc.variables['BUILD_SHARED_LIBS'] = "ON" if self.options.shared == True else "OFF"
tc.variables['CMAKE_FIND_ROOT_PATH_MODE_PACKAGE'] = 'NEVER'
tc.variables['CMAKE_POSITION_INDEPENDENT_CODE'] = 'ON'
tc.generate()
The unit tests of this conan package use a CMakeLists.txt which defines a PackageTests target that links against Package and boost::boost.
Building the Package and PackageTests works fine for both Ubuntu and Windows but only on Ubuntu the tests run without issues. On Windows I get exceptions for all the tests because the boost dlls are not found. Using ldd PackageTests and readelf -d PackageTests on Ubuntu shows that boost so files are used from the conan cache.
Using conans VirtualRunEnv generator and then activating the generated environement helps to also run the PackageTests.exe on Windows but I would like to know if there is another way using for example pure CMake to install/copy the required boost dlls to the bin/PackageTests.exe folder? Or is there a way to extend the conan recipe to install the dlls on Windows?
Why are boost shared libraries found correctly in the conan cache but not on Windows? Is there some extra manual work needed or shouldn't this be handeled by conan as well?
Edit:
Trying to use the following to copy the dlls results in a cmake command usage error because the TARGET_RUNTIME_DLLS generator expression is empty.
add_custom_command(TARGET PackageTest POST_BUILD
COMMAND ${CMAKE_COMMAND} -E copy_if_different $<TARGET_RUNTIME_DLLS:PackageTest > $<TARGET_FILE_DIR:PackageTest>
COMMAND_EXPAND_LISTS
)
Also IMPORTED_LOCATION property for the following targets are *-NOTFOUND:
get_target_property(VAR-boost boost::boost IMPORTED_LOCATION)
message(${VAR-boost})
> VAR-boost-NOTFOUND
get_target_property(VAR-Package Package IMPORTED_LOCATION)
message(${VAR-Package})
> VAR-Package-NOTFOUND
get_target_property(VAR-Package PackageTest IMPORTED_LOCATION)
message(${VAR-Package})
> VAR-PackageTest-NOTFOUND
From the boost conan recipe package_info() I can see that the CMakeDeps generator will only create the BoostConfig.cmake and BoostTargets.cmake scripts, which is also the case for Package. There is no FindBoost.cmake generated. By default CMakeDeps only creates config scripts, except the recipes define cmake_find_mode to be both. Not sure though if adding both to the recipe could help. Even if it would help, this is no immediate solution as this is not directly in my control (hosted on conan-center-index repo). I am still not able to see the reason why everything works fine on Ubuntu but on Windows the dlls are not found/copied at all by conan.

How to correctly build cabal project using hmatrix under Windows 10?

Using Windows 10 64-bit, Cabal-3.4.0.0, ghc-8.10.7.
I installed OpenBLAS in MSYS2 environment with command
pacman -S mingw-w64-x86_64-openblas.
Than, I successfully installed hmatrix-0.20.2 with command
cabal install --lib hmatrix --flags=openblas --extra-include-dirs="C:\\ghcup\\msys64\\mingw64\\include\\OpenBLAS" --extra-lib-dirs="C:\\ghcup\\msys64\\mingw64\\bin" --extra-lib-dirs="C:\\ghcup\\msys64\\mingw64\\lib"
I am trying to build simple test project using cabal build cabalhmatrix with Main
module Main where
import Numeric.LinearAlgebra
main :: IO ()
main = do
putStrLn $ show $ vector [1,2,3] * vector [3,0,-2]
But now I am getting output
Resolving dependencies...
Build profile: -w ghc-8.10.7 -O1
In order, the following will be built (use -v for more details):
- hmatrix-0.20.2 (lib) (requires build)
- cabalhmatrix-0.1.0.0 (exe:cabalhmatrix) (first run)
Starting hmatrix-0.20.2 (lib)
Failed to build hmatrix-0.20.2. The failure occurred during the configure
step.
Build log (
C:\cabal\logs\ghc-8.10.7\hmatrix-0.20.2-6dd2e8f2795550e4dd624770ac98c326dacc0cac.log
):
Warning: hmatrix.cabal:21:28: Packages with 'cabal-version: 1.12' or later
should specify a specific version of the Cabal spec of the form
'cabal-version: x.y'. Use 'cabal-version: 1.18'.
Configuring library for hmatrix-0.20.2..
cabal-3.4.0.0.exe: Missing dependencies on foreign libraries:
* Missing (or bad) C libraries: blas, lapack
This problem can usually be solved by installing the system packages that
provide these libraries (you may need the "-dev" versions). If the libraries
are already installed but in a non-standard location then you can use the
flags --extra-include-dirs= and --extra-lib-dirs= to specify where they are.If
the library files do exist, it may contain errors that are caught by the C
compiler at the preprocessing stage. In this case you can re-run configure
with the verbosity flag -v3 to see the error messages.
cabal-3.4.0.0.exe: Failed to build hmatrix-0.20.2 (which is required by
exe:cabalhmatrix from cabalhmatrix-0.1.0.0). See the build log above for
details.
What should I do to correctly build that package?
I guess I need to somehow pass arguments --flags=openblas --extra-include-dirs="C:\\ghcup\\msys64\\mingw64\\include\\OpenBLAS" --extra-lib-dirs="C:\\ghcup\\msys64\\mingw64\\bin" --extra-lib-dirs="C:\\ghcup\\msys64\\mingw64\\lib" to hmatrix during compilation, but don't know how to do that. To be honest, I don't understand for what program exactly are those arguments (cabal, ghc, ghc-pkg or something else) and why cabal is trying to install hmatrix again. I see hmatrix in directory "C:\cabal\store\ghc-8.10.7\hmatrix-0.20.2-e917eca0fc7690010007a19f4f2a3602d86df0f0".
Created cabal.project file:
packages: .
package hmatrix
flags: +openblas
extra-include-dirs: C:\\ghcup\\msys64\\mingw64\\include\\OpenBLAS
extra-lib-dirs: C:\\ghcup\\msys64\\mingw64\\bin, C:\\ghcup\\msys64\\mingw64\\libenter code here
After adding libopenblas.dll location to PATH variable cabal project is working.
Even though there is the --lib flag, it's generally best to work under the assumption that Cabal doesn't do library installs. Never install a library, instead just depend on it – and have Cabal install, update etc. it whenever necessary.
But then how can you pass the necessary flags? With a cabal.project file.
packages: .
package hmatrix
flags: openblas
extra-include-dirs: C:\\ghcup\\msys64\\mingw64\\include\\OpenBLAS
...
Put this file in the working directory of your own project, together with cabalhmatrix.cabal. Then running cabal build in that directory will use a hmatrix install with the suitable library etc. flags.

How to build/install cross-compiled nested packages quickly?

I have a repository with a group of nested go packages organized as follows:
$GOPATH/src/
- mypackage/common/utils.go
- mypackage/app1/main.go
- mypackage/app2/main.go
...
It compiles to a handful of binaries. For releasing, I'm cross-compiling for a multitude of platforms/archs (I deploy repeatedly with a different GOOS and GOARCH). I'm trying to write the compilation results to a directory of my choice, but I'm fighting with the toolchain.
I can:
Combine GOBIN and go install when compiling for my own architecture
(i.e. not cross compiling):
# build + output all binaries in /somedir/bin
# this works great when compiling for my local architecture.
GOBIN=/somedir/bin go install mypackage/app1 mypackage/app2 mypackage/app3
But unfortunately, GOBIN conflicts with cross-compilation e.g:
# Throws: "cannot install cross-compiled binaries when GOBIN is set"
GOBIN=/somedir/bin GOARCH=amdm64 GOOS=darwin go install mypackage/app1 mypackage/app2
Use go build with GOOS=X and GOARCH=Y for each subpackage
# cross compile one of the binaries
cd /output/darwin-386-bin/ && \
GOOS=darwin GOARCH=386 go build mypackage/app1
# but if given multiple binaries, there is no output. (as documented)
GOOS=darwin GOARCH=386 go build mypackage/app1 mypackage/app2
When go build is given multiple packages, it no longer emits binaries -- it just checks that the code compiles. To emit all the binaries, it seems I have to run go build once for each subpackage, so it takes longer, esp when building with -a.
Another possible issue with using go build like this, is that it's potentially mixing binaries and intermediary results across multiple architectures in the same workspace. But maybe that's just a matter of taste. Hopefully the toolchain keeps cached results separate for different architectures.
How is this handled in practice? Should I treat each subpackage as an individual package, despite their shared common code (would it then be safe to build them all in parallel)?
Related articles:
go build vs go install: https://groups.google.com/forum/#!topic/golang-nuts/s8Csz3-7EXA (Little info on cross-compilation, or multipackage setups)
cross compile with go 1.5.: https://dave.cheney.net/2015/08/22/cross-compilation-with-go-1-5 (does not specifically address multipackage / common code compilation. Also, the statement about having to recompile the standard library into /usr/local/go/pkg seems to no longer hold in 1.9.1, as far as I can tell)
Go build multiple nested packages: Go build multiple/nested packages? . ( The accepted answer, go build ./... behaves the same as passing multiple packages in a single build command, and therefore outputs no binaries. The thread also does not cover cross compilation)
What I'm about to suggest feels like a hack at best, but it might work for you if you're cross-compiling stuff in a container or an isolated build environment.
You can drop the GOBIN from the install command:
# I'm not on this platform, so this is a cross compile
GOOS=darwin GOARCH=amd64 go install mypackage/app1 mypackage/app2
Assuming your package is in $GOPATH/src/mypackage the above command will install the two binaries to:
$GOPATH/bin/darwin_amd64/{app1, app2}
and the .a compile dependency files in:
$GOPATH/pkg/darwin_amd64/mypackage/{app1,app2,common}
If you're running this in a for loop for all the platforms you plan on supporting, one nuisance with this process is that when you pass GOOS=x GOARCH=y matching your local architecture (i.e. not cross compiling), then the executables will be placed directly in $GOPATH/bin/ and not $GOPATH/bin/x_y/.
(packages on the other hand always end up in $GOPATH/pkg/x_y/, cross compilation or not).
To determine your local architecture, you can follow steps in this answer: https://stackoverflow.com/a/35669816/5556676 .
Simulating GOBIN=/foo/bin GOOS=x GOARCH=y go install mypackage/app{1,2,3}
Changing $GOPATH has little effect on where go install writes output. With some go commands, like go get, you tweak where packages are installed by adding a new component at the front of $GOPATH. But in go 1.9, go install will always copy binaries in the bin/ folder that's a sibling of the src/ which contains the packages to install.
Instead of using GOBIN=/foo/bin, you can pretend your source files are in /foo/src/mypackage (you may use a symlink), and then do GOPATH=/foo GOOS=x GOARCH=y go install mypackage/app{1,2,3}. This will put binaries in /foo/bin, because of the behavior I describe in the previous paragraph.
It's probably simpler to just grab the binaries from where you expect them to be though, than mucking with copying source trees around.

Get travisCI build environment working for netcdf-dependent Fortran built in R package

The title should have two subtitles:
What is the pathname to any installed libraries in the Travis CI environment?
or
How do I get my Makevars file portable for NetCDF libraries??
Background:
I am developing and R package that is supposed to work with a shared library I have written in Fortran. I want to check my builds with TravisCI, so my package is currently on GitHub.
So upon package installation, the Fortran source code should be compiled. I can do this locally, but TravisCI errors with the following message:
gfortran -fdefault-real-8 -c HANDLE_ERR.f90
HANDLE_ERR.f90:4: Error: Can't open included file 'netcdf.inc'
make: *** [HANDLE_ERR.o] Error 1
Which I understand as the compiler not finding the NetCDF library, which I made sure I installed by adding this to my .travis.yml:
before_install:
- sudo apt-get install libnetcdf-dev -y
Example
I have created a minimal working example, which is failing in TravisCI, with the error message (above) I am getting on my big project.
See here for the travis build https://travis-ci.org/teatree1212/nctest
you can access my minimal working example repository from there, but here is the link as well: https://github.com/teatree1212/nctest/tree/master
The compilation works when I do it locally, as I can specify the NetCDF library directories. I don't know where these are installed in the Travis build environment, so I think this is where my problem lies at the moment.
However, I would like to make this package portable and not only make it work in the travis container. Therefore these two questions:
What is the pathname to any installed libraries in the Travis CI environment?
and more importantly:
How do I make my Makevars file portable for compiling with NetCDF libraries?

Packaging Go application for Debian

How can I put my Go binary into a Debian package? Since Go is statically linked, I just have a single executable--I don't need a lot of complicated project metadata information. Is there a simple way to package the executable and resource files without going through the trauma of debuild?
I've looked all over for existing questions; however, all of my research turns up questions/answers about a .deb file containing the golang development environment (i.e., what you would get if you do sudo apt-get install golang-go).
Well. I think the only "trauma" of debuild is that it runs lintian after building the package, and it's lintian who tries to spot problems with your package.
So there are two ways to combat the situation:
Do not use debuild: this tool merely calls dpkg-buildpackage which really does the necessary powerlifting. The usual call to build a binary package is dpkg-buildpackage -us -uc -b. You still might call debuild for other purposes, like debuild clean for instance.
Add the so-called "lintian override" which can be used to make lintian turn a blind eye to selected problems with your package which, you insist, are not problems.
Both approaches imply that you do not attempt to build your application by the packaging tools but rather treat it as a blob which is just wrapped to a package. This would require slightly abstraining from the normal way debian/rules work (to not attempt to build anything).
Another solution which might be possible (and is really way more Debian-ish) is to try to use gcc-go (plus gold for linking): since it's a GCC front-end, this tool produces a dynamically-linked application (which links against libgo or something like this). I, personally, have no experience with it yet, and would only consider using it if you intend to try to push your package into the Debian proper.
Regarding the general question of packaging Go programs for Debian, you might find the following resources useful:
This thread started on go-nuts by one of Go for Debian packagers.
In particular, the first post in that thread links to this discussion on debian-devel.
The second thread on debian-devel regarding that same problem (it's a logical continuation of the former thread).
Update on 2015-10-15.
(Since this post appears to still be searched and found and studied by people I've decided to update it to better reflec the current state of affairs.)
Since then the situation with packaging Go apps and packages got improved dramatically, and it's possible to build a Debian package using "classic" Go (the so-called gc suite originating from Google) rather than gcc-go.
And there exist a good infrastructure for packages as well.
The key tool to use when debianizing a Go program now is dh-golang described here.
I've just been looking into this myself, and I'm basically there.
Synopsis
By 'borrowing' from the 'package' branch from one of Canonical's existing Go projects, you can build your package with dpkg-buildpackage.
install dependencies and grab a 'package' branch from another repo.
# I think this list of packages is enough. May need dpkg-dev aswell.
sudo apt-get install bzr debhelper build-essential golang-go
bzr branch lp:~niemeyer/cobzr/package mypackage-build
cd mypackage-build
Edit the metadata.
edit debian/control file (name, version, source). You may need to change the golang-stable dependency to golang-go.
The debian/control file is the manifest. Note the 'build dependencies' (Build-Depends: debhelper (>= 7.0.50~), golang-stable) and the 3 architectures. Using Ubuntu (without the gophers ppa), I had to change golang-stable to golang-go.
edit debian/rules file (put your package name in place of cobzr).
The debian/rules file is basically a 'make' file, and it shows how the package is built. In this case they are relying heavily on debhelper. Here they set up GOPATH, and invoke 'go install'.
Here's the magic 'go install' line:
cd $(GOPATH)/src && find * -name '*.go' -exec dirname {} \; | xargs -n1 go install
Also update the copyright file, readme, licence, etc.
Put your source inside the src folder. e.g.
git clone https://github.com/yourgithubusername/yourpackagename src/github.com/yourgithubusername/yourpackagename
or e.g.2
cp .../yourpackage/ src/
build the package
# -us -uc skips package signing.
dpkg-buildpackage -us -uc
This should produce a binary .deb file for your architecture, plus the 'source deb' (.tgz) and the source deb description file (.dsc).
More details
So, I realised that Canonical (the Ubuntu people) are using Go, and building .deb packages for some of their Go projects. Ubuntu is based on Debian, so for the most part the same approach should apply to both distributions (dependency names may vary slightly).
You'll find a few Go-based packages in Ubuntu's Launchpad repositories. So far I've found cobzr (git-style branching for bzr) and juju-core (a devops project, being ported from Python).
Both of these projects have both a 'trunk' and a 'package' branch, and you can see the debian/ folder inside the package branch. The 2 most important files here are debian/control and debian/rules - I have linked to 'browse source'.
Finally
Something I haven't covered is cross-compiling your package (to the other 2 architectures of the 3, 386/arm/amd64). Cross-compiling isn't too tricky in go (you need to build the toolchain for each target platform, and then set some ENV vars during 'go build'), and I've been working on a cross-compiler utility myself. Eventually I'll hopefully add .deb support into my utility, but first I need to crystallize this task.
Good luck. If you make any progress then please update my answer or add a comment. Thanks
Building deb or rpm packages from Go Applications is also very easy with fpm.
Grab it from rubygems:
gem install fpm
After building you binary, e.g. foobar, you can package it like this:
fpm -s dir -t deb -n foobar -v 0.0.1 foobar=/usr/bin/
fpm supports all sorts of advanced packaging options.
There is an official Debian policy document describing the packaging procedure for Go: https://go-team.pages.debian.net/packaging.html
For libraries: Use dh-make-golang to create a package skeleton. Name your package with a name derived from import path, with a -dev suffix, e.g. golang-github-lib-pq-dev. Specify the dependencies ont Depends: line. (These are source dependencies for building, not binary dependencies for running, since Go statically links all source.)
Installing the library package will install its source code to /usr/share/golang/src (possibly, the compiled libraries could go into .../pkg). Building depending Go packages will use the artifacts from those system-wide locations.
For executables: Use dh-golang to create the package. Specify dependencies in Build-Depends: line (see above regarding packaging the dependencies).
I recently discovered https://packager.io/ - I'm quite happy with what they're doing. Maybe open up one of the packages to see what they're doing?

Resources