I'm trying to use stack install to build and install some executables. My package also has an internal library that the test suite and benchmarks depend on, but none of the executables do. But when I do stack install, it attempts to build the internal library. I haven't been able to find a way to stop that, trying various combinations of targets and --skip. Is it possible?
My root package.yaml is laid out like
name: my-lib
library:
...
internal-libraries:
test-lib:
...
dependencies:
- my-lib
tests:
test:
...
dependencies:
- test-lib
...
benchmarks:
bench:
...
dependencies:
- test-lib
...
And the package.yamls for the executables are just
name: some-exe
executables:
some-exe:
...
dependencies:
- my-lib
...
Nothing lists test-lib in its dependencies. test-lib only exists so that the benchmark can use some utilities defined in the test suite; until recently it was just part of the test suite, not a separate library, and stack install wasn't trying to build it.
An option I've found is to create a separate package for just the internal library. I've created a directory test-pkg and moved the everything from the internal-libraries line down into the package.yaml in there. I also created symlinks to the source directories for my test suite and benchmarks, and assorted other relevant things found in the root of my original package.
Then I can do
stack install `stack ide targets 2>&1 | grep :exe:`
To install all the binaries.
I don't like this very much, but it seems to work.
Related
Using Windows 10 64-bit, Cabal-3.4.0.0, ghc-8.10.7.
I installed OpenBLAS in MSYS2 environment with command
pacman -S mingw-w64-x86_64-openblas.
Than, I successfully installed hmatrix-0.20.2 with command
cabal install --lib hmatrix --flags=openblas --extra-include-dirs="C:\\ghcup\\msys64\\mingw64\\include\\OpenBLAS" --extra-lib-dirs="C:\\ghcup\\msys64\\mingw64\\bin" --extra-lib-dirs="C:\\ghcup\\msys64\\mingw64\\lib"
I am trying to build simple test project using cabal build cabalhmatrix with Main
module Main where
import Numeric.LinearAlgebra
main :: IO ()
main = do
putStrLn $ show $ vector [1,2,3] * vector [3,0,-2]
But now I am getting output
Resolving dependencies...
Build profile: -w ghc-8.10.7 -O1
In order, the following will be built (use -v for more details):
- hmatrix-0.20.2 (lib) (requires build)
- cabalhmatrix-0.1.0.0 (exe:cabalhmatrix) (first run)
Starting hmatrix-0.20.2 (lib)
Failed to build hmatrix-0.20.2. The failure occurred during the configure
step.
Build log (
C:\cabal\logs\ghc-8.10.7\hmatrix-0.20.2-6dd2e8f2795550e4dd624770ac98c326dacc0cac.log
):
Warning: hmatrix.cabal:21:28: Packages with 'cabal-version: 1.12' or later
should specify a specific version of the Cabal spec of the form
'cabal-version: x.y'. Use 'cabal-version: 1.18'.
Configuring library for hmatrix-0.20.2..
cabal-3.4.0.0.exe: Missing dependencies on foreign libraries:
* Missing (or bad) C libraries: blas, lapack
This problem can usually be solved by installing the system packages that
provide these libraries (you may need the "-dev" versions). If the libraries
are already installed but in a non-standard location then you can use the
flags --extra-include-dirs= and --extra-lib-dirs= to specify where they are.If
the library files do exist, it may contain errors that are caught by the C
compiler at the preprocessing stage. In this case you can re-run configure
with the verbosity flag -v3 to see the error messages.
cabal-3.4.0.0.exe: Failed to build hmatrix-0.20.2 (which is required by
exe:cabalhmatrix from cabalhmatrix-0.1.0.0). See the build log above for
details.
What should I do to correctly build that package?
I guess I need to somehow pass arguments --flags=openblas --extra-include-dirs="C:\\ghcup\\msys64\\mingw64\\include\\OpenBLAS" --extra-lib-dirs="C:\\ghcup\\msys64\\mingw64\\bin" --extra-lib-dirs="C:\\ghcup\\msys64\\mingw64\\lib" to hmatrix during compilation, but don't know how to do that. To be honest, I don't understand for what program exactly are those arguments (cabal, ghc, ghc-pkg or something else) and why cabal is trying to install hmatrix again. I see hmatrix in directory "C:\cabal\store\ghc-8.10.7\hmatrix-0.20.2-e917eca0fc7690010007a19f4f2a3602d86df0f0".
Created cabal.project file:
packages: .
package hmatrix
flags: +openblas
extra-include-dirs: C:\\ghcup\\msys64\\mingw64\\include\\OpenBLAS
extra-lib-dirs: C:\\ghcup\\msys64\\mingw64\\bin, C:\\ghcup\\msys64\\mingw64\\libenter code here
After adding libopenblas.dll location to PATH variable cabal project is working.
Even though there is the --lib flag, it's generally best to work under the assumption that Cabal doesn't do library installs. Never install a library, instead just depend on it – and have Cabal install, update etc. it whenever necessary.
But then how can you pass the necessary flags? With a cabal.project file.
packages: .
package hmatrix
flags: openblas
extra-include-dirs: C:\\ghcup\\msys64\\mingw64\\include\\OpenBLAS
...
Put this file in the working directory of your own project, together with cabalhmatrix.cabal. Then running cabal build in that directory will use a hmatrix install with the suitable library etc. flags.
I have a repository with a group of nested go packages organized as follows:
$GOPATH/src/
- mypackage/common/utils.go
- mypackage/app1/main.go
- mypackage/app2/main.go
...
It compiles to a handful of binaries. For releasing, I'm cross-compiling for a multitude of platforms/archs (I deploy repeatedly with a different GOOS and GOARCH). I'm trying to write the compilation results to a directory of my choice, but I'm fighting with the toolchain.
I can:
Combine GOBIN and go install when compiling for my own architecture
(i.e. not cross compiling):
# build + output all binaries in /somedir/bin
# this works great when compiling for my local architecture.
GOBIN=/somedir/bin go install mypackage/app1 mypackage/app2 mypackage/app3
But unfortunately, GOBIN conflicts with cross-compilation e.g:
# Throws: "cannot install cross-compiled binaries when GOBIN is set"
GOBIN=/somedir/bin GOARCH=amdm64 GOOS=darwin go install mypackage/app1 mypackage/app2
Use go build with GOOS=X and GOARCH=Y for each subpackage
# cross compile one of the binaries
cd /output/darwin-386-bin/ && \
GOOS=darwin GOARCH=386 go build mypackage/app1
# but if given multiple binaries, there is no output. (as documented)
GOOS=darwin GOARCH=386 go build mypackage/app1 mypackage/app2
When go build is given multiple packages, it no longer emits binaries -- it just checks that the code compiles. To emit all the binaries, it seems I have to run go build once for each subpackage, so it takes longer, esp when building with -a.
Another possible issue with using go build like this, is that it's potentially mixing binaries and intermediary results across multiple architectures in the same workspace. But maybe that's just a matter of taste. Hopefully the toolchain keeps cached results separate for different architectures.
How is this handled in practice? Should I treat each subpackage as an individual package, despite their shared common code (would it then be safe to build them all in parallel)?
Related articles:
go build vs go install: https://groups.google.com/forum/#!topic/golang-nuts/s8Csz3-7EXA (Little info on cross-compilation, or multipackage setups)
cross compile with go 1.5.: https://dave.cheney.net/2015/08/22/cross-compilation-with-go-1-5 (does not specifically address multipackage / common code compilation. Also, the statement about having to recompile the standard library into /usr/local/go/pkg seems to no longer hold in 1.9.1, as far as I can tell)
Go build multiple nested packages: Go build multiple/nested packages? . ( The accepted answer, go build ./... behaves the same as passing multiple packages in a single build command, and therefore outputs no binaries. The thread also does not cover cross compilation)
What I'm about to suggest feels like a hack at best, but it might work for you if you're cross-compiling stuff in a container or an isolated build environment.
You can drop the GOBIN from the install command:
# I'm not on this platform, so this is a cross compile
GOOS=darwin GOARCH=amd64 go install mypackage/app1 mypackage/app2
Assuming your package is in $GOPATH/src/mypackage the above command will install the two binaries to:
$GOPATH/bin/darwin_amd64/{app1, app2}
and the .a compile dependency files in:
$GOPATH/pkg/darwin_amd64/mypackage/{app1,app2,common}
If you're running this in a for loop for all the platforms you plan on supporting, one nuisance with this process is that when you pass GOOS=x GOARCH=y matching your local architecture (i.e. not cross compiling), then the executables will be placed directly in $GOPATH/bin/ and not $GOPATH/bin/x_y/.
(packages on the other hand always end up in $GOPATH/pkg/x_y/, cross compilation or not).
To determine your local architecture, you can follow steps in this answer: https://stackoverflow.com/a/35669816/5556676 .
Simulating GOBIN=/foo/bin GOOS=x GOARCH=y go install mypackage/app{1,2,3}
Changing $GOPATH has little effect on where go install writes output. With some go commands, like go get, you tweak where packages are installed by adding a new component at the front of $GOPATH. But in go 1.9, go install will always copy binaries in the bin/ folder that's a sibling of the src/ which contains the packages to install.
Instead of using GOBIN=/foo/bin, you can pretend your source files are in /foo/src/mypackage (you may use a symlink), and then do GOPATH=/foo GOOS=x GOARCH=y go install mypackage/app{1,2,3}. This will put binaries in /foo/bin, because of the behavior I describe in the previous paragraph.
It's probably simpler to just grab the binaries from where you expect them to be though, than mucking with copying source trees around.
After being unable to run stack haddock on my project because one package causes it to hang (see How to show progress of `stack haddock`? ) I tried to skip that package with
stack haddock --skip haskell-src-exts
But it still tries to haddock haskell-src-exts anyway (and hangs). What is --skip supposed to do?
--skip was introduced in stack 1.6.1 to avoid building a specific component of a project. A component corresponded to a test suite, an executable or a benchmark suite. As backpack support is implemented, this will have to include libraries, too.
From stack build --help
--skip ARG Skip given component, can be specified multiple times
The docs go into more detail.
--skip, to skip building components of a local package. It allows
you to skip test suites and benchmark without specifying other components
(e.g. stack test --skip long-test-suite will run the tests without the
long-test-suite test suite). Be aware that skipping executables won't work
the first time the package is built due to
an issue in cabal.
This option can be specified multiple times to skip multiple components.
For example, a package my-package might have a library, an executable my-executable, and two test suites (unit-tests and integration-tests). To build and run tests, you may run stack test. Adding --skip integration-tests will cause the integration-tests component not to be built (nor run).
Unfortunately, the current version of stack (1.9 branch) does not support skipping haddock for individual dependencies, though there is something like this due in the near future. I encountered this exact same problem (with haskell-src-exts). For now, I suggest skipping all haddock dependencies (--no-haddock-deps).
You may wish to invoke haddock with stack exec -- haddock if you wish to pass arguments to haddock which do not work with the stack flag --haddock-arguments.
The title should have two subtitles:
What is the pathname to any installed libraries in the Travis CI environment?
or
How do I get my Makevars file portable for NetCDF libraries??
Background:
I am developing and R package that is supposed to work with a shared library I have written in Fortran. I want to check my builds with TravisCI, so my package is currently on GitHub.
So upon package installation, the Fortran source code should be compiled. I can do this locally, but TravisCI errors with the following message:
gfortran -fdefault-real-8 -c HANDLE_ERR.f90
HANDLE_ERR.f90:4: Error: Can't open included file 'netcdf.inc'
make: *** [HANDLE_ERR.o] Error 1
Which I understand as the compiler not finding the NetCDF library, which I made sure I installed by adding this to my .travis.yml:
before_install:
- sudo apt-get install libnetcdf-dev -y
Example
I have created a minimal working example, which is failing in TravisCI, with the error message (above) I am getting on my big project.
See here for the travis build https://travis-ci.org/teatree1212/nctest
you can access my minimal working example repository from there, but here is the link as well: https://github.com/teatree1212/nctest/tree/master
The compilation works when I do it locally, as I can specify the NetCDF library directories. I don't know where these are installed in the Travis build environment, so I think this is where my problem lies at the moment.
However, I would like to make this package portable and not only make it work in the travis container. Therefore these two questions:
What is the pathname to any installed libraries in the Travis CI environment?
and more importantly:
How do I make my Makevars file portable for compiling with NetCDF libraries?
How can I put my Go binary into a Debian package? Since Go is statically linked, I just have a single executable--I don't need a lot of complicated project metadata information. Is there a simple way to package the executable and resource files without going through the trauma of debuild?
I've looked all over for existing questions; however, all of my research turns up questions/answers about a .deb file containing the golang development environment (i.e., what you would get if you do sudo apt-get install golang-go).
Well. I think the only "trauma" of debuild is that it runs lintian after building the package, and it's lintian who tries to spot problems with your package.
So there are two ways to combat the situation:
Do not use debuild: this tool merely calls dpkg-buildpackage which really does the necessary powerlifting. The usual call to build a binary package is dpkg-buildpackage -us -uc -b. You still might call debuild for other purposes, like debuild clean for instance.
Add the so-called "lintian override" which can be used to make lintian turn a blind eye to selected problems with your package which, you insist, are not problems.
Both approaches imply that you do not attempt to build your application by the packaging tools but rather treat it as a blob which is just wrapped to a package. This would require slightly abstraining from the normal way debian/rules work (to not attempt to build anything).
Another solution which might be possible (and is really way more Debian-ish) is to try to use gcc-go (plus gold for linking): since it's a GCC front-end, this tool produces a dynamically-linked application (which links against libgo or something like this). I, personally, have no experience with it yet, and would only consider using it if you intend to try to push your package into the Debian proper.
Regarding the general question of packaging Go programs for Debian, you might find the following resources useful:
This thread started on go-nuts by one of Go for Debian packagers.
In particular, the first post in that thread links to this discussion on debian-devel.
The second thread on debian-devel regarding that same problem (it's a logical continuation of the former thread).
Update on 2015-10-15.
(Since this post appears to still be searched and found and studied by people I've decided to update it to better reflec the current state of affairs.)
Since then the situation with packaging Go apps and packages got improved dramatically, and it's possible to build a Debian package using "classic" Go (the so-called gc suite originating from Google) rather than gcc-go.
And there exist a good infrastructure for packages as well.
The key tool to use when debianizing a Go program now is dh-golang described here.
I've just been looking into this myself, and I'm basically there.
Synopsis
By 'borrowing' from the 'package' branch from one of Canonical's existing Go projects, you can build your package with dpkg-buildpackage.
install dependencies and grab a 'package' branch from another repo.
# I think this list of packages is enough. May need dpkg-dev aswell.
sudo apt-get install bzr debhelper build-essential golang-go
bzr branch lp:~niemeyer/cobzr/package mypackage-build
cd mypackage-build
Edit the metadata.
edit debian/control file (name, version, source). You may need to change the golang-stable dependency to golang-go.
The debian/control file is the manifest. Note the 'build dependencies' (Build-Depends: debhelper (>= 7.0.50~), golang-stable) and the 3 architectures. Using Ubuntu (without the gophers ppa), I had to change golang-stable to golang-go.
edit debian/rules file (put your package name in place of cobzr).
The debian/rules file is basically a 'make' file, and it shows how the package is built. In this case they are relying heavily on debhelper. Here they set up GOPATH, and invoke 'go install'.
Here's the magic 'go install' line:
cd $(GOPATH)/src && find * -name '*.go' -exec dirname {} \; | xargs -n1 go install
Also update the copyright file, readme, licence, etc.
Put your source inside the src folder. e.g.
git clone https://github.com/yourgithubusername/yourpackagename src/github.com/yourgithubusername/yourpackagename
or e.g.2
cp .../yourpackage/ src/
build the package
# -us -uc skips package signing.
dpkg-buildpackage -us -uc
This should produce a binary .deb file for your architecture, plus the 'source deb' (.tgz) and the source deb description file (.dsc).
More details
So, I realised that Canonical (the Ubuntu people) are using Go, and building .deb packages for some of their Go projects. Ubuntu is based on Debian, so for the most part the same approach should apply to both distributions (dependency names may vary slightly).
You'll find a few Go-based packages in Ubuntu's Launchpad repositories. So far I've found cobzr (git-style branching for bzr) and juju-core (a devops project, being ported from Python).
Both of these projects have both a 'trunk' and a 'package' branch, and you can see the debian/ folder inside the package branch. The 2 most important files here are debian/control and debian/rules - I have linked to 'browse source'.
Finally
Something I haven't covered is cross-compiling your package (to the other 2 architectures of the 3, 386/arm/amd64). Cross-compiling isn't too tricky in go (you need to build the toolchain for each target platform, and then set some ENV vars during 'go build'), and I've been working on a cross-compiler utility myself. Eventually I'll hopefully add .deb support into my utility, but first I need to crystallize this task.
Good luck. If you make any progress then please update my answer or add a comment. Thanks
Building deb or rpm packages from Go Applications is also very easy with fpm.
Grab it from rubygems:
gem install fpm
After building you binary, e.g. foobar, you can package it like this:
fpm -s dir -t deb -n foobar -v 0.0.1 foobar=/usr/bin/
fpm supports all sorts of advanced packaging options.
There is an official Debian policy document describing the packaging procedure for Go: https://go-team.pages.debian.net/packaging.html
For libraries: Use dh-make-golang to create a package skeleton. Name your package with a name derived from import path, with a -dev suffix, e.g. golang-github-lib-pq-dev. Specify the dependencies ont Depends: line. (These are source dependencies for building, not binary dependencies for running, since Go statically links all source.)
Installing the library package will install its source code to /usr/share/golang/src (possibly, the compiled libraries could go into .../pkg). Building depending Go packages will use the artifacts from those system-wide locations.
For executables: Use dh-golang to create the package. Specify dependencies in Build-Depends: line (see above regarding packaging the dependencies).
I recently discovered https://packager.io/ - I'm quite happy with what they're doing. Maybe open up one of the packages to see what they're doing?