bjam - build subfolder project (change directory to subfolder project before build) - gcc

Question

I am trying to build https://github.com/wallix/redemption, which uses bjam. I don't really know bjam (and I think, given that I already don't really know make and cmake, I don't really have that much space left in in my head for yet another build system language).
More specifically, I want to build projects/qtclient in there, as part of the main project; so that when I run tools/packager.py (which repeats the bjam build process from scratch), i also get the qtclient executable in the generated .deb.
Note that, when I'm the root directory of the repository, for the main build, I can do:
redemption.git$ bjam exe libs
warning: No toolsets are configured.
warning: Configuring default toolset "gcc".
warning: If the default is wrong, your build may not work correctly.
warning: Use the "toolset=xxxxx" option to override our guess.
warning: For more configuration options, please consult
warning: http://boost.org/boost-build2/doc/html/bbv2/advanced/configuration.html
...patience...
...found 2502 targets...
...updating 267 targets...
gcc.compile.c++ bin/gcc-9/debug/log.o
...
... and it works. If I try to build the projects/qtclient from the same directory location, it fails:
redemption.git$ bjam projects/qtclient/
warning: No toolsets are configured.
warning: Configuring default toolset "gcc".
warning: If the default is wrong, your build may not work correctly.
warning: Use the "toolset=xxxxx" option to override our guess.
warning: For more configuration options, please consult
warning: http://boost.org/boost-build2/doc/html/bbv2/advanced/configuration.html
redemption-src: /home/USER
/home/USER/jam/redemption-config.jam: No such file or directory
/home/USER/jam/defines.jam: No such file or directory
Assume Qt5. (bjam -s qt5)
projects/qtclient/Jamroot:48: in modules.load
ERROR: rule "setvar" unknown in module "Jamfile</home/USER/src/redemption_git/projects/qtclient>".
/usr/share/boost-build/src/build/project.jam:372: in load-jamfile
/usr/share/boost-build/src/build/project.jam:64: in load
/usr/share/boost-build/src/build/project.jam:142: in project.find
/usr/share/boost-build/src/build/targets.jam:453: in find-really
/usr/share/boost-build/src/build/targets.jam:475: in class#project-target.find
/usr/share/boost-build/src/build-system.jam:724: in load
/usr/share/boost-build/src/kernel/modules.jam:295: in import
/usr/share/boost-build/src/kernel/bootstrap.jam:139: in boost-build
/usr/share/boost-build/boost-build.jam:8: in module scope
... but if I change directory to projects/qtclient first, and call bjam qtclient (or just bjam) there, it works:
demption.git/projects/qtclient$ bjam qtclient
redemption-src: /home/USER/src/redemption_git
Assume Qt5. (bjam -s qt5)
warning: No toolsets are configured.
warning: Configuring default toolset "gcc".
warning: If the default is wrong, your build may not work correctly.
warning: Use the "toolset=xxxxx" option to override our guess.
warning: For more configuration options, please consult
warning: http://boost.org/boost-build2/doc/html/bbv2/advanced/configuration.html
warning: non-free usage requirements <threading>multi ignored
warning: in main-target QtCore at Jamroot:146
...patience...
...found 2454 targets...
...updating 107 targets...
qt5.moc bin/gcc-9/release/src/qt_input_output_api/moc_qt_input_output_clipboard.cpp
...
gcc.link bin/gcc-9/release/qt5client
...updated 107 targets...
Right - so this shows me, that the error 'ERROR: rule "setvar" unknown in module "Jamfile</home/USER/src/redemption_git/projects/qtclient>".' is due to bjam running in the "project root" directory, and not in the projects/qtclient subdirectory.
So, now I try to integrate the build of projects/qtclient inside the main Jamroot; I've tried adding qtclient at end of alias exe line (before the semicolon):
alias exe : rdpproxy rdpclient rdpinichecker qtclient;
Result: build does not even start:
redemption.git$ bjam libs exe
error: Unable to find file or target named
error: 'qtclient'
error: referred to from project at
error: '.'
And I've tried adding projects/qtclient at end of alias exe line :
alias exe : rdpproxy rdpclient rdpinichecker projects/qtclient;
Result: build starts, but fails as bjam does not run in the project subdirectory (same error: 'ERROR: rule "setvar" unknown in module'):
redemption_git$ bjam libs exe
warning: No toolsets are configured.
...
/home/USER/jam/redemption-config.jam: No such file or directory
/home/USER/jam/defines.jam: No such file or directory
Assume Qt5. (bjam -s qt5)
projects/qtclient/Jamroot:48: in modules.load
ERROR: rule "setvar" unknown in module "Jamfile</home/USER/src/redemption_git/projects/qtclient>".
/usr/share/boost-build/src/build/project.jam:372: in load-jamfile
/usr/share/boost-build/src/build/project.jam:64: in load
...
As per Boost-build - dependency on subproject target I also tried modifying the code so it has this, while restoring the original alias exe line:
import feature ;
feature.feature qtclient : : dependency free ;
project redemption
: requirements
$(REDEMPTION_CXXFLAGS)
$(REDEMPTION_FFMPEG_FLAGS)
$(REDEMPTION_BOOST_STACKTRACE_FLAGS)
$(GCOV)
<cxxflags>-fno-rtti
<toolset>gcc:<cxxflags>-pipe
# <cxx-lto-default>on
# <cxx-stack-protector-default>on # strong, all
# <cxxflags>-fpie
<qtclient>projects/qtclient//qtclient
: default-build release
;
...
alias exe : rdpproxy rdpclient rdpinichecker ;
... and again, build starts, but fails as bjam does not run in the project subdirectory (same error: 'ERROR: rule "setvar" unknown in module'):
$ bjam libs exe
warning: No toolsets are configured.
...
redemption-src: /home/USER
/home/USER/jam/redemption-config.jam: No such file or directory
/home/USER/jam/defines.jam: No such file or directory
Assume Qt5. (bjam -s qt5)
/home/USER/src/redemption_git/projects/qtclient/Jamroot:48: in modules.load
ERROR: rule "setvar" unknown in module "Jamfile</home/USER/src/redemption_git/projects/qtclient>".
/usr/share/boost-build/src/build/project.jam:372: in load-jamfile
...
Also, as per How to build multiple targets with Boost and Jamroot? I've tried adding reference to projects/qtclient/Jamroot at end of alias exe line:
alias exe : rdpproxy rdpclient rdpinichecker projects/qtclient/Jamroot;
This actually runs and build completes without errors - however there is no mention of "qtclient" anywhere in the build log, and the corresponding executable does not get generated.
The same happens for build-project that I got from bjam - how to always execute shell script before building a target?:
build-project projects/qtclient//qtclient ;
... or for
build-project projects/qtclient ;
I also tried to cheat and just call from the shell:
Echo [ SHELL "cd projects/qtclient; bjam release qtclient" ] ;
... this indeed builds qtclient - but then erases the build folder, before starting to build the main project :(
So - how do I have bjam compile an extra project in a subfolder, as part of the main project, correctly (meaning that bjam would change the current working directory to the subfolder, before attempting to build the extra project)?

Answer 1

EDIT: the procedure described below, while it adds the subproject executable so it runs correctly after installing, it also messes up paths for executables from the main project. A correct procedure has been provided by developers of the original project in this comment:
But if you want, you can add cd projects/qtclient; bjam qtclient / cd projects/qtclient; bjam install in packaging/template/debian/rules, add the dependencies in the packaging/targets/ files and the new executable in packaging/template/debian/redemption.install (the last step may be useless thanks to %PREFIX%/bin/*).
Right, so this was a more complicated thing, and I came to a nasty workaround - but, better than nothing. There were several problems here - first, let's look at the Jamroot of the subfolder project; I'll post this as a diff:
diff --git a/projects/qtclient/Jamroot b/projects/qtclient/Jamroot
index 2379580db..d72e3edb5 100644
--- a/projects/qtclient/Jamroot
+++ b/projects/qtclient/Jamroot
## -1,6 +1,13 ##
-REDEMPTION_PUBLIC_PATH ?= [ SHELL "readlink -n -f ../.." ] ;
+#REDEMPTION_PUBLIC_PATH ?= [ SHELL "readlink -n -f ../.." ] ;
-ECHO "redemption-src: $(REDEMPTION_PUBLIC_PATH)" ;
+path-constant MYPATH : . ;
+REDEMPTION_PUBLIC_PATH ?= [ SHELL "readlink -n -f $(MYPATH)/../.." ] ;
+
+ECHO "qtclient Jamroot: MYPATH $(MYPATH)" ;
+ECHO "qtclient Jamroot: REDEMPTION_ROOT_PATH $(REDEMPTION_ROOT_PATH:E=not_set)" ;
+ECHO "qtclient Jamroot: REDEMPTION_PUBLIC_PATH $(REDEMPTION_PUBLIC_PATH)" ;
+ECHO "qtclient Jamroot: .all-features $(.all-features:E=not_set)" ;
+#ECHO [ SHELL "pstree -s -p $$ && echo PID $$ PWD $PWD" ] ;
JAM_INCLUDE_PATH ?= $(REDEMPTION_PUBLIC_PATH)/jam ;
REDEMPTION_INCLUDE_PATH ?= $(REDEMPTION_PUBLIC_PATH)/include ;
## -124,7 +131,7 ## add_obj graphics.o : src/qt_input_output_api/graphics.cpp ;
local requirement_list = ;
# generated by `bjam targets.jam` from redemption project
-include redemption_deps.jam ;
+include $(MYPATH)/redemption_deps.jam ;
constant EXE_DEPENDENCIES :
Originally, it tried to find REDEMPTION_PUBLIC_PATH via [ SHELL "readlink -n -f ../.." ] ; - and if a shell is being called with readlink -f ../.., it has no other choice but to refer to its CWD; it does not "know" whether it is in a subfolder of a project. Luckily, it turns out there is a bjam syntax that lets a jam file find its own path: the path-constant rule (boost-build/bjam constant for path to Jamroot), and that is what the above change uses, to provide a correct reference to _INCLUDE_PATHs and included files - which gets rid of the "ERROR: rule "setvar" unknown in module".
Once this is done, the next problem is, that if the build of the subfolder project is called from the main Jamfile (e.g. via build-project), we'll get a new error:
/usr/share/boost-build/src/build/feature.jam:140: in feature from module feature
error: feature already defined:
error: in feature declaration:
error: feature "<cxx-color>" : "_" "default" "auto" "never" "always" : "propagated"
The problem is that both the main Jamroot and the subproject Jamroot do include $(JAM_INCLUDE_PATH)/cxxflags.jam ; etc; I tried removing those manually from the subproject Jamroot, for the case when build is started from the main folder - but then eventually we get to the subproject jam complaining about unknows stuff:
/usr/share/boost-build/src/kernel/modules.jam:107: in modules.call-in
ERROR: rule "Jamfile</home/USER/src/redemption.git/projects/qtclient>.flags" unknown in module "Jamfile</home/USER/src/redemption.git/projects/qtclient>".
... so it still needs those files, but they are already in conflict ... so this way won't work.
So again we have to go back to "changing directory", and apparently we must do that from the shell ... So that part is in the main Jamroot, again shown as a diff:
diff --git a/Jamroot b/Jamroot
index eb4188fdd..e4a756002 100644
--- a/Jamroot
+++ b/Jamroot
## -46,6 +46,27 ## JAM_INCLUDE_PATH ?= jam ;
#
#############################
+path-constant REDEMPTION_ROOT_PATH : . ;
+ECHO "main: REDEMPTION_ROOT_PATH $(REDEMPTION_ROOT_PATH)" ;
+import modules ;
+tbuildcmd = "" ;
+local args = [ modules.peek : ARGV ] ;
+if clean in $(args)
+{
+ tbuildcmd = clean ;
+}
+else if release in $(args)
+{
+ tbuildcmd = release ;
+}
+else if debug in $(args)
+{
+ tbuildcmd = debug ;
+}
+echo "tbuildcmd $(tbuildcmd)" ;
+echo "ARGV: $(args)" ;
+echo "INSTALLDIR/BINPREFIX: $(INSTALLDIR:E=not_set)$(BIN_PREFIX:E=not_set)" ;
+
include $(JAM_INCLUDE_PATH)/redemption-config.jam ;
include $(JAM_INCLUDE_PATH)/cxxflags.jam ;
include $(JAM_INCLUDE_PATH)/defines.jam ;
## -154,10 +175,22 ## alias install :
install-etc-ppocr
install-share
install-gettext
+ install-qtclient
;
alias exe : rdpproxy rdpclient rdpinichecker ;
alias libs : libredrec ;
+import notfile ;
+notfile post_qtclient : #my-post-command : libs exe ;
+actions my-post-command
+{
+ echo my-post-command $(tbuildcmd) runs.
+ cd projects/qtclient ;
+ bjam $(tbuildcmd) ;
+ cd ../.. ;
+ rsync -aP projects/qtclient/bin/gcc-9/$(tbuildcmd)/. bin/gcc-9/$(tbuildcmd)/
+}
+
alias ocr_tools : display_learning extract_text ppocr_extract_text ;
alias install-etc-ppocr : install-etc-ppocr-latin install-etc-ppocr-latin-cyrillic ;
## -199,6 +232,39 ## install install-lib
: <location>$(INSTALLDIR)$(LIB_PREFIX)
;
+import path : basename ;
+
+#epath = "$(REDEMPTION_ROOT_PATH)/projects/qtclient/bin/gcc-9/release/qt5client" ;
+#ename = basename ( $(epath) ) ; # nope, causes target to not be found :(
+# adds newline at end
+#ename = [ SHELL "basename $(epath)" ] ;
+#ename = SHELL "basename $(epath)" ; # does not work
+actions make_qtclient_install
+{
+ echo "make_qtclient_install: REDEMPTION_ROOT_PATH $(REDEMPTION_ROOT_PATH) PWD $(PWD:E=not_set)" ;
+ echo [ SHELL "pstree -s -p $$ && echo PID $$ PWD $PWD" ] ;
+ # unfortunately, when we get here, bjam is not called with debug or release, so tbuildcmd is empty
+ # so assuming we delete our directories manually:
+ # rm -rf bin/gcc-9/* projects/qtclient/bin/gcc-9/* debian/* ./bin/project-cache.jam ./projects/qtclient/bin/project-cache.jam
+ # we can count on only one qt5client being built in qtclient bin dir, so target it with wildcards
+ # also, debian/buildtmp/usr/local/bin/ $(INSTALLDIR)$(BIN_PREFIX)/ might not exist at this time
+ # so use install instead of cp
+ # don't use local here, crashes "not in a function"
+ # cannot get the stupid glob to work, write manually
+ #epath = [ glob "$(REDEMPTION_ROOT_PATH)/projects/qtclient/bin/gcc-9/*/qt*client" ] ;
+ #epath="$(REDEMPTION_ROOT_PATH)/projects/qtclient/bin/gcc-9/release/qt5client" ;
+ #ename = [ SHELL "basename $(epath)" ] ;
+ #ename=`basename $(epath)`
+ # don't bother with variables, nothing works here https://stackoverflow.com/questions/75342602/bjam-cannot-assign-a-literal-to-a-variable
+ #echo "epath $(epath) ename $(ename)" ;
+ #cp -av $(REDEMPTION_ROOT_PATH)/projects/qtclient/bin/gcc-9/*/qt*client $(INSTALLDIR)$(BIN_PREFIX)/ ;
+ # just type everything verbatim:
+ echo install -m 775 -D $(REDEMPTION_ROOT_PATH)/projects/qtclient/bin/gcc-9/release/qt5client $(INSTALLDIR)$(BIN_PREFIX)/qt5client ;
+ install -m 775 -D $(REDEMPTION_ROOT_PATH)/projects/qtclient/bin/gcc-9/release/qt5client $(INSTALLDIR)$(BIN_PREFIX)/qt5client ;
+}
+explicit install-qtclient ;
+make install-qtclient : : #make_qtclient_install ;
+
exe redrec
:
First, we'd like to know whether we're doing debug or release build; apparently in bjam this is a "variant" which is a type of "feature", and I couldn't figure out how to read it ( Getting the build type/variant inside a bjam Jamfile / echo a feature? ). So I made a workaround where command line arguments are used - of course, this is only good when you use a bjam command that explicitly includes debug or release on the command line.
Then, a notfile is used to define something that functions like a new, custom/manual target - in the sense that we can call post_qtclient and it will trigger the build of libs and exe as dependencies; and then the my-post-command action is ran. The commands in this action run in the shell, so we can change directory to the subfolder project, run bjam in debug or release (via variable), and once done, we copy all files form the subfolder project bin directory to the main bin directory.
This is good enough for a build via bjam -a release post_qtclient - but not good enough if you intend to run debhelper scripts to build a .deb package (done by a tool in this repository, ./tools/packager.py --build-package --force-build).
To do that, we must copy the resulting executable file of the subfolder into the right install location; that is done by the install-qtclient "make" action (or whatever that is). It is very tricky to handle string there (left comments to document that) - so in the end, this part does a copy of the hardcoded subfolder project executable into the "bin" install location, as for other executables - however, using install, since at the time this piece runs, $(INSTALLDIR)$(BIN_PREFIX) might not yet exist!
This generally takes care of the install part; and to finally make sure that the subproject executable does end up in the .deb file, change /packaging/template/debian/rules like so:
diff --git a/packaging/template/debian/rules b/packaging/template/debian/rules
index 014c470e4..dab9a70ea 100755
--- a/packaging/template/debian/rules
+++ b/packaging/template/debian/rules
## -32,7 +32,8 ## build:
# dh_testdir - test directory before building Debian package. It makes sure
# that debian/control exists also check Jamroot for redemption project.
dh_testdir Jamroot
- bjam -q $(BJAM_EXTRA_INSTALL) exe libs
+ #bjam -q $(BJAM_EXTRA_INSTALL) exe libs
+ bjam -q $(BJAM_EXTRA_INSTALL) post_qtclient
... and this should finally result with a build of a .deb file, that has the subproject executable in the same bin folder as the other executables that are products of this process.
Note that rebuilding the project from scratch might be tricky because of caching, so here I've done the below command to "clean everything":
rm -rf bin/gcc-9/* projects/qtclient/bin/gcc-9/* debian/* ./bin/project-cache.jam ./projects/qtclient/bin/project-cache.jam
Well, hope that was it for this problem ...