I believe that the auto-scheduler adopted in lesson_21_auto_scheduler_generate.cpp is a classical Mullapudi2016 algorithm.
How do I switch this auto-scheduler to Adam2019 or Li2018 for this lesson ?
Is Adam2019 still in experimental stage ?
According to the paper, the result is very much better than that of Mullapudi2016.
Perhaps, Halide developers should add a lesson for using Adam2019 to the tutorial.
Is Adam2019 still in experimental stage?
It is not! It's ready to use.
How do I switch this auto-scheduler to Adam2019 or Li2018 for this lesson?
This is done when running the generator. Run your generator without arguments to see the help text, which will show you how to load the auto_schedule.so plugin. If you're on Windows or if you're otherwise using the new CMake build, it's quite easy. After building and installing Halide to a local directory, just:
cmake_minimum_required(VERSION 3.16)
project(Lesson21)
set(CMAKE_CXX_STANDARD 11)
set(CMAKE_CXX_STANDARD_REQUIRED YES)
set(CMAKE_CXX_EXTENSIONS NO)
find_package(Halide REQUIRED)
# Add the generator
add_executable(lesson_21_auto_scheduler_generate lesson_21_auto_scheduler_generate.cpp)
target_link_libraries(lesson_21_auto_scheduler_generate PRIVATE Halide::Generator)
# Use the generator to create the two variants that the runner expects
add_halide_library(auto_schedule_false FROM lesson_21_auto_scheduler_generate
GENERATOR auto_schedule_gen
PARAMS auto_schedule=false)
add_halide_library(auto_schedule_true FROM lesson_21_auto_scheduler_generate
GENERATOR auto_schedule_gen
PARAMS auto_schedule=true
AUTOSCHEDULER Halide::Adams2019)
# Link them to the runner executable
add_executable(lesson_21_auto_scheduler_run lesson_21_auto_scheduler_run.cpp)
target_link_libraries(lesson_21_auto_scheduler_run
PRIVATE
auto_schedule_false
auto_schedule_true
Halide::Tools)
The add_halide_library function manages running generators to produce static libraries. The AUTOSCHEDULER argument lets you select between Halide::Adams2019 and Halide::Li2018. At time of writing, you omit it completely to select the classic (default) autoscheduler. It will be renamed in the future to Halide::Mullapudi2016. You do still need to pass auto_schedule=true in the PARAMS list.
Then build with:
$ cd build
$ cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_PREFIX_PATH=/path/to/Halide-install ..
$ cmake --build .
$ ./lesson_21_auto_scheduler_run
Manual schedule: 9.31708ms
Auto schedule: 1.64732ms
Perhaps, Halide developers should add a lesson for using Adam2019 to the tutorial.
I have been meaning to add a tutorial for using Halide from CMake for a while, now. No promises on the timeline, but hopefully I can get to it soon.
Related
I have an autotools project which successfully builds and tests an app (https://github.com/goglecm/AutoBrightnessCam). The app is installed in the bin directory (preceded by any prefix the user specifies). That's pretty straightforward. I now need to make a systemd service to start it at boot time. I've created the service file and ran it manually and it works fine.
The last bit is to tell configure.ac and Makefile.am to patch a *.service.in file with the correct path for the app (just like config.h is created from config.h.in).
Will using AC_CONFIG_HEADERS be appropriate to patch *.service.in into *.service? Is there another macro used for "non-headers" perhaps?
Also, how do I specify that the service file should land (i.e. installed) in /etc/systemd/system?
Is there perhaps a better way of starting this app at boot time without systemd?
How do I specify that the service file should land (i.e. installed) in /etc/systemd/system?
According to Systemd's daemon man page:
<BEGINQUOTE>
Installing systemd Service Files
At the build installation time (e.g. make install during package build), packages are recommended to install their systemd unit files in the directory returned by pkg-config systemd --variable=systemdsystemunitdir (for system services) or pkg-config systemd --variable=systemduserunitdir (for user services). This will make the services available in the system on explicit request but not activate them automatically during boot. Optionally, during package installation (e.g. rpm -i by the administrator), symlinks should be created in the systemd configuration directories via the enable command of the systemctl(1) tool to activate them automatically on boot.
Packages using autoconf(1) are recommended to use a configure script excerpt like the following to determine the unit installation path during source configuration:
PKG_PROG_PKG_CONFIG
AC_ARG_WITH([systemdsystemunitdir],
[AS_HELP_STRING([--with-systemdsystemunitdir=DIR], [Directory for systemd service files])],,
[with_systemdsystemunitdir=auto])
AS_IF([test "x$with_systemdsystemunitdir" = "xyes" -o "x$with_systemdsystemunitdir" = "xauto"], [
def_systemdsystemunitdir=$($PKG_CONFIG --variable=systemdsystemunitdir systemd)
AS_IF([test "x$def_systemdsystemunitdir" = "x"],
[AS_IF([test "x$with_systemdsystemunitdir" = "xyes"],
[AC_MSG_ERROR([systemd support requested but pkg-config unable to query systemd package])])
with_systemdsystemunitdir=no],
[with_systemdsystemunitdir="$def_systemdsystemunitdir"])])
AS_IF([test "x$with_systemdsystemunitdir" != "xno"],
[AC_SUBST([systemdsystemunitdir], [$with_systemdsystemunitdir])])
AM_CONDITIONAL([HAVE_SYSTEMD], [test "x$with_systemdsystemunitdir" != "xno"])
This snippet allows automatic installation of the unit files on systemd machines, and optionally allows their installation even on machines lacking systemd. (Modification of this snippet for the user unit directory is left as an exercise for the reader.)
Additionally, to ensure that make distcheck continues to work, it is recommended to add the following to the top-level Makefile.am file in automake(1)-based projects:
AM_DISTCHECK_CONFIGURE_FLAGS = \
--with-systemdsystemunitdir=$$dc_install_base/$(systemdsystemunitdir)
Finally, unit files should be installed in the system with an automake excerpt like the following:
if HAVE_SYSTEMD
systemdsystemunit_DATA = \
foobar.socket \
foobar.service
endif
...
</ENDQUOTE>
So it appears you should use systemdsystemunitdir and systemduserunitdir. How well Autotools supports it, well...
A quick grep on Fedora 31 using grep systemdsystemunitdir /bin/autoconf and grep -IR systemdsystemunitdir /usr/share shows no Autotools support yet. 7 years and counting...
Is there perhaps a better way of starting this app at boot time without systemd?
Systemd should be OK to start your app. Simply use systemctl(1) to enable and start them as you normally would.
Based on your GitHub and autobrightnesscam.service.in, I would not dick around with Autotools for this. You can waste copious amounts of time working around Autotols short comings (speaking from experience).
My configure.ac script (which is just a shell script) would copy autobrightnesscam.service.in to autobrightnesscam.service, and then use sed to copy-in the correct directories and files. Then, I would copy the updated autobrightnesscam.service to its proper location in AC_CONFIG_COMMANDS_POST. Maybe something like:
SERVICE_FILE=autobrightnesscam.service
SYSTEMD_DIR=`pkg-config systemd --variable=systemdsystemunitdir`
# Use default if SYSTEMD_DIR is empty
if test x"$SYSTEMD_DIR" = "x"; then
SYSTEMD_DIR=/etc/systemd/system
fi
AC_CONFIG_COMMANDS_POST([cp "$SERVICE_FILE" "$SYSTEMD_DIR"])
AC_CONFIG_COMMANDS_POST([systemctl enable "$SYSTEMD_DIR/$SERVICE_FILE"])
AC_CONFIG_COMMANDS_POST([systemctl start "$SERVICE_FILE"])
Will using AC_CONFIG_HEADERS be appropriate to patch *.service.in into *.service? Is there another macro used for "non-headers" perhaps?
No. AC_CONFIG_HEADERS is for setting up configuration headers to support your build. It is rarely used for anything other than building a config.h recording the results of certain tests that Autoconf performs, and it is not as flexible as other options in this area.
If you have additional files that you want Autoconf to build from templates then you should tell Autoconf about them via AC_CONFIG_FILES. Example:
AC_CONFIG_FILES([Makefile AutoBrightnessCam.service])
But if some of the data with which you are filling that template are installation directories then Autoconf is probably not the right place to do this at all, because it makes provision for the installation prefix to be changed by arguments to make. You would at least need to work around that, but the best thing to do is to roll with it instead, and build the .service file under make's control. It's not that hard, and there are several technical advantages, some applying even if there aren't any installation directory substitutions to worry about.
You can do it the same way that configure does, by running the very same template you're already using through sed, with an appropriate script. Something like this would appear in your Makefile.am:
SERVICE_SUBS = \
s,[#]VARIABLE_NAME[#],$(VARIABLE_NAME),g; \
s,[#]OTHER_VARIABLE[#],$(OTHER_VARIABLE),g
AutoBrightnessCam.service: AutoBrightnessCam.service.in
$(SED) -e '$(SERVICE_SUBS)' < $< > $#
Also, how do I specify that the service file should land (i.e.
installed) in /etc/systemd/system?
You use Automake's standard mechanism for specifying custom installation locations. Maybe something like this:
sytemdsysdir = $(sysconfdir)/systemd/system
systemdsys_DATA = AutoBrightnessCam.service
Is there perhaps a better way of
starting this app at boot time without systemd?
On a systemd-based machine, systemd is in control of what starts at boot. If you want the machine to start your application automatically at boot, then I think your options are limited to
Configuring systemd to start it
Configuring something in a chain of programs ultimately started by systemd to start it
Hacking the bootloader or kernel to start it
There is room for diverging opinions here, but I think the first of those is cleanest and most future-proof, and I cannot recommend the last.
CLion is a powerful multi-platform IDE that allows to run and debug C++ applications.
I tried to use it with an AzerothCore core project.
It is smart enough to detect all the processes:
Mostly I'm interested in running the worldserver.
However when I try to run or debug it, it correctly compile and runs but it will look for the worldserver.conf.dist configuration file in the directory /usr/local/etc/, giving error because that file is not there.
I would like to manually specify the path of such configuration file, as well as passing other CMake parameters.
I made it work by opening File -> Settings and looking for "CMake" under "Build, Execution, Deployment".
From that window I could pass my CMake options, which in my case where:
-G "Unix Makefiles" -DTOOLS=0 -DSCRIPTS=static -DCMAKE_C_FLAGS="-Werror" -DCMAKE_CXX_FLAGS="-Werror" -DCMAKE_INSTALL_PREFIX=/path/to/the/main/server/dir
The /path/to/the/main/server/dir is where my etc, data, etc... folders are and the worldserver.config.dist is inside this etc folder. So doing this everything worked fine.
I also changed the "Build options" to better use my processor, passing -j 10.
For macOS users, you'll probably need to add these CMake options as well:
-DMYSQL_ADD_INCLUDE_PATH=/usr/local/include
-DMYSQL_LIBRARY=/usr/local/lib/libmysqlclient.dylib
-DREADLINE_INCLUDE_DIR=/usr/local/opt/readline/include
-DREADLINE_LIBRARY=/usr/local/opt/readline/lib/libreadline.dylib
-DOPENSSL_INCLUDE_DIR=/usr/local/opt/openssl/include
-DOPENSSL_SSL_LIBRARIES=/usr/local/opt/openssl/lib/libssl.dylib
-DOPENSSL_CRYPTO_LIBRARIES=/usr/local/opt/openssl/lib/libcrypto.dylib
I apologize in advance for my lack of experience with m4sh.
I have a configure.ac file containing the following lines:
AC_INIT([libhelloworld], [2.5])
...
AC_OUTPUT([
Makefile
src/helloworld-${PACKAGE_VERSION}.pc
src/Makefile
])
The reason behind the argument of AC_OUTPUT() is that I would like to avoid to copy and paste the new version of my program in multiple places on every update. Therefore I decided to exploit the PACKAGE_VERSION macro, automatically defined when invoking AC_INIT() at the beginning of configure.ac.
The line src/helloworld-${PACKAGE_VERSION}.pc then correctly expands into src/helloworld-2.5.pc and everything seems to be working fine. However I have a couple of questions.
I use ${PACKAGE_VERSION} as a shell variable, but PACKAGE_VERSION itself is a m4 macro. Can I trust that this will always work? Will it always be defined as such when AC_OUTPUT() is invoked?
Are there other ways of obtaining the value of PACKAGE_VERSION within configure.ac? For example, if instead of configure.ac I were inside Makefile.am I would not use the curly brackets, but the command evaluation syntax instead, as in $(PACKAGE_VERSION). What is the proper correct way of doing what I want within configure.ac?
The documentation for AC_INIT states that PACKAGE_VERSION is an "output variable", meaning when you call AC_INIT, something like this gets executed:
AC_SUBST([PACKAGE_VERSION], [2.5])
This allows the configuration of input files such as Makefile.in (generated from Makefile.am) to rely on #PACKAGE_VERSION# inside those files being replaced by 2.5.
There's nothing wrong with your approach if it works, but you might consider using AS_VAR_SET([hello_version], [AC_PACKAGE_VERSION]) to set the hello_version shell variable and src/helloworld-${hello_version}.pc in Autoconf input. This way, even if Autoconf no longer exposes a PACKAGE_VERSION shell variable in some future release, your code won't break because you'll be relying upon your own hello_version variable.
As an aside, it's a bit irregular to use helloworld-2.5.pc when the helloworld version is 1.0 or greater (i.e. the API is stable). It's common to see helloworld.pc, except then there's the problem of what happens when you release 3.0 and replace the installed 2.x version of helloworld.pc with the 3.0 version: assuming you're using semantic versioning, 3.0 is incompatible with 2.x, and any code relying on something like pkg-config --libs helloworld will break.
You might then consider using helloworld-2.pc instead and when you release 3.0, you'd instead have helloworld-3.pc to avoid users of your library linking the incorrect/incompatible library (and also allowing users the option of moving to the new version at their own pace); one can also apply this idea in Automake for a version-specific header directory:
## SOURCE PATH => INSTALL PATH
## include/hello.h => $(includedir)/helloworld-2/hello.h
helloincludedir = #includedir#/helloworld-#hello_major#
helloinclude_HEADERS = include/hello.h
Autoconf also allows you to specify an output file's inputs, so an output file src/helloworld-${hello_major}.pc in the build directory could be generated from src/helloworld.pc.in in the source directory without you needing to update the src/helloworld.pc.in filename when moving from 2.x to 3.0; this could also be used with AC_INIT if you're OK with macros, allowing you to control the version info in one central location:
m4_define([hello_version_major], [2]) dnl
m4_define([hello_version_minor], [5]) dnl
m4_define([hello_version], [hello_version_major[.]hello_version_minor]) dnl
AC_PREREQ([2.69])
AC_INIT([libhelloworld], [hello_version])
AS_VAR_SET([hello_major], [hello_version_major])
AS_VAR_SET([hello_minor], [hello_version_minor])
# For automake and configuration of pkg-config file
AC_SUBST([hello_major])
AC_SUBST([hello_minor])
AC_SUBST([hello_version])
...
AC_CONFIG_FILES([
Makefile
src/Makefile
src/helloworld-]hello_version_major[.pc:src/helloworld.pc.in
])
AC_OUTPUT
I realize it looks surprisingly more complicated than one might expect, but that's Autoconf for you. Note that I had to use some odd quoting in AC_CONFIG_FILES to make use of the macro. Using
src/helloworld-${hello_major}.pc:src/helloworld.pc.in
instead of the macro resulted in a crippled config.status file being generated in Autoconf 2.69 (try config.status with no arguments, then config.status src/helloworld-2.pc to see the issue); I haven't tested any other versions. I've reported the bug, but the macro works until the next release.
While setting up a deploy pipeline for optimised builds of a server application, I ran into some trouble getting the GHC options right with stack-1.6.5.
In particular, from the docs it doesn't get clear to me how the various ways to specify GHC options work together and when and how they are applied.
As far as I can tell, there are X ways of specifying GHC options:
globally as ghc-options: in ~/.stack/config.yaml and/or /etc/stack/config.yaml, per package or with "$locals", "$targets" or "$everything"
in the project stack.yaml file, per package or with "$locals", "$targets" or "$everything"
in the project package.yaml/.cabal file, globally or per target
in a dependency stack.yaml/package.yaml/.cabal files
on the stack command line via --ghc-options
and there is the apply-ghc-options: setting locals/targets/everything in stack.yaml and ~/.stack/config.yaml and/or /etc/stack/config.yaml
I'd like to know which options are applied in the different build phases snapshots/locals/targets in which order and in which cases they are additive or override options given elsewhere.
good question, this is not adequately documented. These tend to be additive. Most of the logic for this is here: https://github.com/commercialhaskell/stack/blob/657937b0ac5dbef29114b43e9c69e2b57198af85/src/Stack/Build/Source.hs#L131 . Here's the order, where later items in the list come later in the options provided to ghc:
Options specified in the package.yaml / cabal file.
$everything in ghc-options in stack.yaml
$locals in ghc-options in stack.yaml
$targets in ghc-options in stack.yaml
Special options like -fhpc (--coverage) / fprof-auto -fprof-cafs (--profile) / -g (--no-strip).
Options specified via --ghc-options on the CLI
There is currently an issue where $everything / $locals / $targets ghc-options specified in .stack/config.yaml are not additive. Instead they are currently shadowed by the project stack.yaml. There is a PR fixing this, it will probably get merged at some point: https://github.com/commercialhaskell/stack/pull/3781
Introduction
I have a do_install task in a BitBake recipe which I've written for a driver where I execute a custom install script. The task fails because the installation script cannot find kernel source header files within <the image rootfs>/usr/src/kernel. This script runs fine on the generated OS.
What's Happening
Here's the relevant part of my recipe:
SRC_URI += "file://${TOPDIR}/example"
DEPENDS += " virtual/kernel linux-libc-headers "
do_install () {
( cd ${TOPDIR}/example/Install ; ./install )
}
Here's a relevant portion of the install script:
if [ ! -d "/usr/src/kernel/include" ]; then
echo ERROR: Linux kernel source include directory not found.
exit 1
fi
cd /usr/src/kernel
make scripts
...
./install_drv pci ${DRV_ARGS}
I checked changing to if [ ! -d "/usr/src/kernel" ], which also failed. install passes different options to install_drv, which I have a relevant portion of below:
cd ${DRV_PATH}/pci
make NO_SYSFS=${ARG_NO_SYSFS} NO_INSTALL=${ARG_NO_INSTALL} ${ARGS_HWINT}
if [ ${ARG_NO_INSTALL} == 0 ]; then
if [ `/sbin/lsmod | grep -ci "uceipci"` -eq 1 ]; then
./unload_pci
fi
./load_pci DEBUG=${ARG_DEBUG}
fi
The make target build: within ${DRV_PATH}/pci is essentially this:
make -C /usr/src/kernel SUBDIRS=${PWD} modules
My Research
I found these comments within linux-libc-headers.inc relevant:
# You're probably looking here thinking you need to create some new copy
# of linux-libc-headers since you have your own custom kernel. To put
# this simply, you DO NOT.
#
# Why? These headers are used to build the libc. If you customise the
# headers you are customising the libc and the libc becomes machine
# specific. Most people do not add custom libc extensions to the kernel
# and have a machine specific libc.
#
# But you have some kernel headers you need for some driver? That is fine
# but get them from STAGING_KERNEL_DIR where the kernel installs itself.
# This will make the package using them machine specific but this is much
# better than having a machine specific C library. This does mean your
# recipe needs a DEPENDS += "virtual/kernel" but again, that is fine and
# makes total sense.
#
# There can also be a case where your kernel extremely old and you want
# an older libc ABI for that old kernel. The headers installed by this
# recipe should still be a standard mainline kernel, not your own custom
# one.
I'm a bit unclear if I can 'get' the headers from the STAGING_KERNEL_DIR properly since I'm not using make.
Within kernel.bbclass provided in the meta/classes directory, there is this variable assigment:
# Define where the kernel headers are installed on the target as well as where
# they are staged.
KERNEL_SRC_PATH = "/usr/src/kernel"
This path is then packaged later within that .bbclass file here:
PACKAGES = "kernel kernel-base kernel-vmlinux kernel-image kernel-dev kernel-modules"
...
FILES_kernel-dev = "/boot/System.map* /boot/Module.symvers* /boot/config* ${KERNEL_SRC_PATH} /lib/modules/${KERNEL_VERSION}/build"
Update (1/21):
A suggestion on the yocto IRC channel was to use the following line:
do_configure[depends] += "virtual/kernel:do_shared_workdir"
which is corroborated by the Yocto Project Reference Manual, which states that in version 1.8, there was the following change:
The kernel build process was changed to place the source in a common shared work area and to place build artifacts separately in the source code tree. In theory, migration paths have been provided for most common usages in kernel recipes but this might not work in all cases. In particular, users need to ensure that ${S} (source files) and ${B} (build artifacts) are used correctly in functions such as do_configure and do_install. For kernel recipes that do not inherit from kernel-yocto or include linux-yocto.inc, you might wish to refer to the linux.inc file in the meta-oe layer for the kinds of changes you need to make. For reference, here is the commit where the linux.inc file in meta-oewas updated.
Recipes that rely on the kernel source code and do not inherit the module classes might need to add explicit dependencies on the do_shared_workdir kernel task, for example:
do_configure[depends] += "virtual/kernel:do_shared_workdir"
But I'm having difficulties applying this to my recipe. From what I understand, I should be able to change the above line to:
do_install[depends] += "virtual/kernel:do_shared_workdir"
Which would mean that the do_install task now must be run after do_shared_workdir task of the virtual/kernel recipe, which means that I should be able to work with the shared workdir (see Question 3 below), but I still have the same missing kernel header issue.
My Questions
I'm using a custom linux kernel (v3.14) from git.kernel.org. which inherits the kernel class. Here are some of my questions:
Shouldn't the package kernel-dev be a part of any recipe which inherits the kernel class? (this section of the variables glossary)
If I add the virtual/kernel to the DEPENDS variable, wouldn't that mean that the kernel-dev would be brought in?
If kernel-dev is part of the dependencies of my recipe, wouldn't I be able to point to the /usr/src/kernel directory from my recipe? According to this reply on the Yocto mailing list, I think I should.
How can I properly reference the kernel source header files, preferably without changing the installation script?
Consider your Environment
Remember that there are different environments within the the build time environment, consisting of:
sysroots
in the case of kernels, a shared work directory
target packages
kernel-dev is a target package, which you'd install into the rootfs of the target system for certain things like kernel symbol maps which are needed by profiling tools like perf/oprofile. It is not present at build time although some of its contents are available in the sysroots or shared workdir.
Point to the Correct Directories
Your do_install runs at build time so this is within the build directory structures of the build system, not the target one. In particular, /usr/src/ won't be correct, it would need to be some path within your build directory. The virtual/kernel do_shared_workdir task populates ${STAGING_DIR_KERNEL} so you would want to change to that directory in your script.
Adding a Task Dependency
The:
do_install[depends] += "virtual/kernel:do_shared_workdir
dependency like looks correct for your use case, assuming nothing in do_configure or do_compile accesses the data there.
Reconsider the module BitBake class
The other answers are correct in the recommendation to look at module.bbclass, since this illustrates how common kernel modules can be built. If you want to use custom functions or make commands, this is fine, you can just override them. If you really don't want to use that class, I would suggest taking inspiration from it though.
Task Dependencies
Adding virtual/kernel to DEPENDS means virtual/kernel:do_populate_sysroot must run before our do_configure task. Since you need a dependency for do_shared_workdir here, a DEPENDS on virtual/kernel is not enough.
Answer to Question 3
The kernel-dev package would be built, however it would then need to be installed into your target image and used at runtime on a real target. You need this at build time so kernel-dev is not appropriate.
Other Suggestions
You'd likely want the kernel-devsrc package for what you're doing, not the kernel-dev package.
I don't think anyone can properly answer that last question here. You are using a non-standard install method: we can't know how to interact with it...
That said, take a look at what meta/classes/module.bbclass does. It sets several related variables for make: KERNEL_SRC=${STAGING_KERNEL_DIR}, KERNEL_PATH=${STAGING_KERNEL_DIR}, O=${STAGING_KERNEL_BUILDDIR}. Maybe your installer supports some of these environment variables and you could set them in your recipe?