Is there a Buildroot setting for making kernel source available to compile against? - embedded-linux

I've been given a .zip file containing source for a proprietary kernel module. Once unzip'd, there is an install script that needs to be run. The install script untar's the actual source and builds the kernel module. It requires kernel headers to compile against.
Here is my Buildroot .mk file:
FOOCO_VERSION = 1.0
FOOCO_SOURCE = cust_kernel_drvr.zip
FOOCO_SITE = /mnt/third-party/fooco
FOOCO_SITE_METHOD = local
define FOOCO_CONFIGURE_CMDS
unzip $(#D)/$(FOOCO_SOURCE) -d $(#D)
endef
define FOOCO_BUILD_CMDS
chmod +x $(#D)/TOOLS/Linux_x64/DRIVER/install
cd $(#D)/TOOLS/Linux_x64/DRIVER; $(SHELL) ./install
rm -rf $(#D)
endef
$(eval $(generic-package))
This results in the following log output and error:
(Note: I enabled debugging that shows the start and end of each step.)
DEBUG: start | rsync | fooco
>>> fooco 1.0 Syncing from source dir /mnt/third-party/fooco
rsync -au --chmod=u=rwX,go=rX --exclude .svn --exclude .git --exclude .hg --exclude .bzr --exclude CVS /mnt/third-party/fooco/ /root/buildroot-2022.02.1/output/build/fooco-1.0
DEBUG: end | rsync | fooco
DEBUG: start | configure | fooco
>>> fooco 1.0 Configuring
unzip /root/buildroot-2022.02.1/output/build/fooco-1.0/cust_kernel_drvr.zip -d /root/buildroot-2022.02.1/output/build/fooco-1.0
Archive: /root/buildroot-2022.02.1/output/build/foofo-1.0/cust_kernel_drvr.zip
[snip]
creating: /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/
creating: /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/
inflating: /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/install
inflating: /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/cust_kernel_drvr-1.2.0.15-0.noarch.rpm
inflating: /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/cust_kernel_drvr.tar.gz
inflating: /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/license_gpl.txt
[snip]
DEBUG: end | configure | fooco
DEBUG: start | build | fooco
>>> fooco 1.0 Building
chmod +x /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/install
cd /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER; /bin/bash ./install
Extracting archive..OK!
Compiling the driver...Error: make[1]: Entering directory '/root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/fooco_cust/src/linux/driver'
common.mk:82: *** Kernel header files not in any of the expected locations.
common.mk:83: *** Install the appropriate kernel development package, e.g.
common.mk:84: *** kernel-devel, for building kernel modules and try again. Stop.
make[1]: Leaving directory '/root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/fooco_cust/src/linux/driver'
Error: unable to find driver file (fooco_cust.ko) in /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/fooco_cust/src/linux/driver
rm -rf /root/buildroot-2022.02.1/output/build/fooco-1.0
DEBUG: end | build | fooco
touch: cannot touch '/root/buildroot-2022.02.1/output/build/fooco-1.0/.stamp_built': No such file or directory
make: *** [/root/buildroot-2022.02.1/output/build/fooco-1.0/.stamp_built] Error 1
package/pkg-generic.mk:289: recipe for target
'/root/buildroot-2022.02.1/output/build/fooco-1.0/.stamp_built' failed
I found that the make files that came with the kernel module are looking in several places for the kernel headers:
/lib/modules/${BUILD_KERNEL}/source \
/lib/modules/${BUILD_KERNEL}/build \
/usr/src/linux-${BUILD_KERNEL} \
/usr/src/linux-$(${BUILD_KERNEL} | sed 's/-.*//') \
/usr/src/kernel-headers-${BUILD_KERNEL} \
/usr/src/kernel-source-${BUILD_KERNEL} \
/usr/src/linux-$(${BUILD_KERNEL} | sed 's/\([0-9]*\.[0-9]*\)\..*/\1/') \
/usr/src/linux \
/usr/src/kernels/${BUILD_KERNEL} \
/usr/src/kernels
Why is the kernel source not visible to this build? I thought that, since Buildroot is building the kernel as part of the overall process, the header files would be available for subsequent kernel module compiles. Am I missing a setting? I feel that I'm not understanding the Buildroot process in a basic way, even after referring to the manual many times.
I'm using Buildroot 2022.02.1 and kernel 5.15.33.

Your download/extract logic is very convoluted. You should really use something like this:
FOO_SITE = /mnt/third-party/fooco
FOO_SOURCE = cust_kernel_drvr.zip
FOO_SITE_METHOD = file
define FOO_EXTRACT_CMDS
unzip $(FOO_DLDIR)/$(FOOCO_SOURCE) -d $(#D)
endef
Regarding the build issue: it is impossible to help without studying the specific build system of this kernel module. Very likely you will need to pass some environment variables to tell the build system where your kernel source code is located, and possibly other things. But without looking at the specific details, it's impossible to help you.
You can have a look at how standard out of tree kernel modules are handled by looking at the package/pkg-kernel-module.mk code. However, that will not be directly useful to a package like yours that uses a custom installation script.

The magic was the LINUX_DIR variable which, according to Buildroot user manual:
contains the path to where the Linux kernel has been extracted and built.
I was able to patch the install script to send this variable to the make file that was looking for the kernel.

Related

Unable to `make clean`: "No rule to make target ...Plo"

Question Summary
I'm trying to make clean and I'm getting the following output:
Makefile:835: /absolute/path/to/server/server/example/.deps/libfoo-bar.Plo: No such file or directory
Makefile:836: /absolute/path/to/server/server/example/.deps/libfoo-baz.Plo: No such file or directory
Makefile:837: /absolute/path/to/server/server/example/.deps/libfoo-qux.Plo: No such file or directory
Makefile:838: /absolute/path/to/server/server/example/.deps/libfoo-quux.Plo: No such file or directory
Makefile:839: /absolute/path/to/server/server/example/.deps/libfoo-quuz.Plo: No such file or directory
Makefile:840: /absolute/path/to/server/server/example/.deps/libfoo-corge.Plo: No such file or directory
Makefile:841: /absolute/path/to/server/server/example/.deps/libfoo-grault.Plo: No such file or directory
Makefile:842: /absolute/path/to/server/server/example/.deps/libfoo-garply.Plo: No such file or directory
Makefile:843: /absolute/path/to/server/server/example/.deps/libfoo-waldo.Plo: No such file or directory
Makefile:844: /absolute/path/to/server/server/example/.deps/libfoo-fred.Plo: No such file or directory
Makefile:845: /absolute/path/to/server/server/example/.deps/libfoo-plugh.Plo: No such file or directory
Makefile:846: /absolute/path/to/server/server/example/.deps/libfoo-xyzzy.Plo: No such file or directory
Makefile:847: /absolute/path/to/server/server/example/.deps/libfoo-babble.Plo: No such file or directory
Makefile:848: /absolute/path/to/server/server/example/.deps/libfoo-thud.Plo: No such file or directory
Makefile:848: /absolute/path/to/server/server/example/.deps/libfoo-flarp.Plo: No such file or directory
make: *** No rule to make target '/absolute/path/to/server/server/example/.deps/libfoo-flarp.Plo'. Stop.
How can I successfully do a make clean? I don't understand what these .Plo files are, and they are usually something I don't mess with.
Investigation
My Makefile.am
The Makefile.am which is used to generate the Makefile from which I'm trying to run the clean contains this preamble:
include $(top_srcdir)/server/include.am
include $(top_srcdir)/server/tests/include.am
...
The first $(top_srcdir)/server/include.am includes the following:
...
server_libdir = $(exec_prefix)/lib
serverdir = $(top_srcdir)/server
...
and in the specified $(top_srcdir)/server/tests/include.am, I have a lot of things including:
...
server_lib_LTLIBRARIES = libfoo.la
libfoo_la_SOURCES = \
/absolute/path/to/server/server/example/bar.c \
/absolute/path/to/server/server/example/baz.c \
/absolute/path/to/server/server/example/qux.c \
/absolute/path/to/server/server/example/quux.c \
/absolute/path/to/server/server/example/quuz.c \
/absolute/path/to/server/server/example/corge.c \
/absolute/path/to/server/server/example/grault.c \
/absolute/path/to/server/server/example/garply.c \
/absolute/path/to/server/server/example/waldo.c \
/absolute/path/to/server/server/example/fred.c \
/absolute/path/to/server/server/example/plugh.c \
/absolute/path/to/server/server/example/xyzzy.c \
...
Investigating the Makefile Contents
If I look at the Makefile directly and find the lines generating these errors, I see the following:
include /absolute/path/to/server/server/example/$(DEPDIR)/libfoo-bar.Plo
include /absolute/path/to/server/server/example/$(DEPDIR)/libfoo-baz.Plo
include /absolute/path/to/server/server/example/$(DEPDIR)/libfoo-qux.Plo
...
Where are the .Plo files generated?
Frustratingly, there seems to be the exact .Plo files that I need generated all over the place, just not in the right place; for example these files exist:
/absolute/path/to/server/tests/absolute/path/to/server/server/example/$(DEPDIR)/libfoo-bar.Plo
/absolute/path/to/server/tests/absolute/path/to/server/server/example/$(DEPDIR)/libfoo-baz.Plo
/absolute/path/to/server/tests/absolute/path/to/server/server/example/$(DEPDIR)/libfoo-qux.Plo
...
and there are 7 other directories where these .Plo files are created.
Unsuccessful Attempts
Running config.status directly
I found this message on the GNU mailing list:
Actually, these .P files are created by config.status for each
directory it creates an Automake Makefile in. (BTW, make sure
./config.status --help' lists the relevant Makfiles in the Configuration files' section).
You can request the creation of the .P files for a given directory
by running
% rm -Rf kernel/framework/.deps
% ./config.status kernel/framework/Makefile depfiles
And ran ./config.status as described but the output of make clean hasn't changed
According to this other Stack Overflow question, they were able to generate .Plo files, but I don't know what is different for them.
re-run configure with the --disable-dependency-tracking option

Where's the location of the app/binary defined in a yocto recipe?

I have a following recipe which runs a said service which in turns runs an app on boot-up, but I am trying to understand where the location of the app defined which ends up in sysfs image.
Currently, the appSource binary (defined in Makefile) gets stored in /usr/bin but I'm not sure where the destination location (/usr/bin) is defined.
The following command results in
$ bitbake -e appSource | grep ^FILES_${PN}
FILES_appSource="/usr/bin/* /usr/sbin/* /usr/libexec/* /usr/lib/lib*.so.* /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev /usr/lib/udev /lib/udev /usr/lib/udev
Here's the recipe
inherit autotools-brokensep pkgconfig
DESCRIPTION = "A sample recipe"
LICENSE = "CLOSED"
DEPENDS = "glib-2.0"
FILESPATH =+ "${THISDIR}:"
SRC_URI = "file://appSource"
S = "${WORKDIR}/appSource""
FILES_${PN} += "${systemd_unitdir}/*"
INIT_MANAGER = "systemd"
do_install_append() {
if ${#bb.utils.contains('DISTRO_FEATURES', 'systemd', 'true', 'false', d)}; then
install -d ${D}/etc/initscripts
install -d ${D}${systemd_unitdir}/system
install -m 0644 ${WORKDIR}/appService/appService.service ${D}${systemd_unitdir}/system/appService.service
install -d ${D}${systemd_unitdir}/system/multi-user.target.wants/
ln -sf ${systemd_unitdir}/system/appService.service ${D}${systemd_unitdir}/system/multi-user.target.wants/appService.service
fi
}
Here is what I find out:
You are inheriting autotools-brokensep which has the following content:
# Autotools class for recipes where separate build dir doesn't work
# Ideally we should fix software so it does work. Standard autotools supports
# this.
inherit autotools
B = "${S}"
So, it inherit autotools which has do_install with the content of:
autotools_do_install() {
oe_runmake 'DESTDIR=${D}' install
# Info dir listing isn't interesting at this point so remove it if it exists.
if [ -e "${D}${infodir}/dir" ]; then
rm -f ${D}${infodir}/dir
fi
}
So, it runs the install target of your Makefile into ${D} which is ${WORKDIR}/image.
So, I assume that your Makefile has an install target that copies the binary into /usr/bin.
For the FILES variable content, this is defined in bitbake.conf:
FILES_${PN} = "${bindir}/* ${sbindir}/* ${libexecdir}/* ${libdir}/lib*${SOLIBS} \ ...
Provide your Makefile to confirm my assumption, or I do a further research on the topic.

Using a portable Makefile for AWS Lambda

I'm using a Makefile for building a lambda extension. It works fine when I run a sam build command from Azure DevOps using Ubuntu. However, sometimes I want to build from my PC and the Makefile doesn't work because commands are different. I suppose I could setup command aliases for dos but that seems treacherous. I'd prefer to use something like OS detecting makefile but when I tried that it didn't work out.
Here's the pertinent snippet from my template:
TokenCacheLayer:
Type: AWS::Serverless::LayerVersion
Metadata:
BuildMethod: makefile
Properties:
LayerName: !Ref TokenCacheLayerName
Description: AWS Lambda extension to cache tokens aquired from an external API.
ContentUri: ./token-cache
LicenseInfo: MIT-0
CompatibleRuntimes:
- nodejs12.x
RetentionPolicy: Retain
Here's my Makefile:
build-TokenCacheLayer:
cp -R extensions "$(ARTIFACTS_DIR)"
cp -R token-cache-extension "$(ARTIFACTS_DIR)"
Here's the 2 ways I've tried altering to make it OS independent:
build-TokenCacheLayer:
ifeq ($(OS),Windows_NT)
xcopy extensions "$(ARTIFACTS_DIR)\extensions"
xcopy token-cache-extension "$(ARTIFACTS_DIR)"
else
cp -R extensions "$(ARTIFACTS_DIR)"
cp -R token-cache-extension "$(ARTIFACTS_DIR)"
endif
and
ifeq ($(OS),Windows_NT)
build-TokenCacheLayer:
xcopy extensions "$(ARTIFACTS_DIR)\extensions"
xcopy token-cache-extension "$(ARTIFACTS_DIR)"
else
build-TokenCacheLayer:
cp -R extensions "$(ARTIFACTS_DIR)"
cp -R token-cache-extension "$(ARTIFACTS_DIR)"
endif
The first way I got the error:
Error: CustomMakeBuilder:MakeBuild - Make Failed: process_begin: CreateProcess(NULL, cp -R extensions D:\Development\AWS.Integration\buildpro\layers\sam-build\TokenCacheLayer, ...) failed.
make (e=2): The system cannot find the file specified.
make: *** [D:\Development\AWS.Integration\buildpro\layers\token-cache\Makefile:6: build-TokenCacheLayer] Error 2
The second gave me this:
Error: CustomMakeBuilder:MakeBuild - Make Failed: D:\Development\AWS.Integration\buildpro\layers\token-cache\Makefile:2: *** recipe commences before first target. Stop.
Update
I went back and tried the first way again and while I'm not getting an error it doesn't copy the files either.

Make: only build something if the source's md5 checksum was changed

Is it possible to tell make to only build a target if it's sources md5 checksum has changed (instead of the edit time)?
I'm using make to compile my .tex files and I need to prevent it from building everything twice all the time.
I've tried using Scons, but I feel this isn't adaptable to other needs I have.
No, this is not supported by Make — as you've found out, support for this feature is one of the reasons why tools like Scons exist.
I found a manual recipe for GNU make, though. Maybe you can use that as a work around.
I found 3 ways: From simply
Using temporary md5-files over an
elegant but unreadable hack to
patching GNU Make yourself
(from patchy but simple to clean but hard to realize)
Maybe my scons latex and org-mode recipe helps you:
## I need a couple of special builders for my projects
# the $SOURCE replacement only uses the first source file. $SOURCES gives all.
# specifying all source files makes it possible to rerun the build if a single source file changed.
orgexportpdf = 'emacs --batch --visit "$SOURCE" --funcall org-export-as-pdf'
pyxplot = 'pyxplot $SOURCE'
# pdflatex is quite dirty. I directly clean up after it with rm.
pdflatex = 'pdflatex $SOURCE -o $TARGET; rm -f *_flymake* flymake* *.log *.out *.toc *.aux *.snm *.nav *.vrb'
# build the PhD thesis from emacs org-mode.
Command("doktorarbeit.pdf", "doktorarbeit.org",
orgexportpdf)
# create plots
Command(["images/comp-t3-s07-tem-boas.png",
"images/comp-t3-s07-tem-bona.png"],
["nee-comp.pyx",
"nee-comp.txt"],
pyxplot)
# build my sink.pdf
Command("sink.pdf",
["sink.tex",
"images/comp-t3-s07-tem-boas.png",
"images/comp-t3-s07-tem-bona.png",
"images/bona-marble.png",
"images/boas-marble.png"],
pdflatex)
# My editors leave tempfiles around. I want them gone after a build clean. This is not yet supported!
tempfiles = Glob('*~') + Glob('#*#') + Glob('*.bak')
# using this here would run the cleaning on every run.
#Command("clean", [], Delete(tempfiles))
It is the counterpart for my Makefile:
all: doktorarbeit.pdf sink.pdf
sink.pdf : sink.tex images/comp-t3-s07-tem-boas.png images/comp-t3-s07-tem-bona.png images/bona-marble.png images/boas-marble.png
pdflatex sink.tex
rm -f *_flymake* flymake* *.log *.out *.toc *.aux *.snm *.nav *.vrb # kill litter
comp-t3-s07-tem-boas.png comp-t3-s07-tem-bona.png : nee-comp.pyx nee-comp.txt
pyxplot nee-comp.pyx
# http://www.reddit.com/r/emacs/comments/dy9yt/anyone_know_of_a_script_that_i_can_feed_an/
# http://blog.nguyenvq.com/2010/10/30/bash-batch-script-to-convert-org-mode-file-to-html/comment-page-1/#comment-27013
doktorarbeit.pdf : doktorarbeit.org
emacs --batch --visit "doktorarbeit.org" --funcall org-export-as-pdf
# this is not what clean is intended to do, but I do it anyway.
clean :
rm -f \#* *~ *.bak # kill editor backups
# alternatively run scons. Should I switch to SCons, just put this in all :)
sconsrun : scons
python scons/bootstrap.py -Q
scons :
hg clone https://bitbucket.org/ArneBab/scons

Install data directory tree with massive number of files using automake

I have a data directory which I would like automake to generate install and uninstall targets for. Essentially, I just want to copy this directory verbatim to the DATA directory, Normally, I might list all the files individually, like
dist_whatever_DATA=dir/subdir/filea ...
But the problem arises when my directory structure looks like this
*root
*subdir
*~10 files
*subdir
*~10 files
*subdir
*~700 files
*subdir
...
~20 subdirs
I just cannot list all 1000+ files included as part of my Makefile.am. That would be ridiculous.
I need to preserve the directory structure as well. I should note that this data is not generated at all by the build process, and is actually largely short audio recordings. So it's not like I would want automake to "check" that every file I want to install has actually been created, as they're either there or not, and whatever file is there, I know I want it to be installed, and whatever file is not, should not be installed. I know that this is the justification used in other places to not do wildcard instsalls, but all the possible reasons don't apply here.
I would use a script to generate a Makefile fragment that lists all the files:
echo 'subdir_files =' > subfiles.mk
find subdir -type f -print | sed 's/^/ /;$q;s/$/ \\/' >> subfiles.mk
and then include this subfiles.mk from your main Makefile.am:
include $(srcdir)/subfiles.mk
nobase_dist_pkgdata_DATA = $(subdir_files)
A second option is to EXTRA_DIST = subdir, and then to write custom install-data-local and uninstall-local rules.
The problem here is that EXTRA_DIST = subdir will distributes all files in subdir/, including backup files, configuration files (e.g. from your VCS), and other things you would not want to distribute.
Using a script as above let you filter the files you really want to distribute.
I've found that installing hundreds of files separately makes for a tormentingly long invocation of make install. I had a similar case where I wanted to install hundreds of files, preserving the directory structure, and I did not want to change my Makefile.am every time a file was added to or removed from the collection.
I included a LZMA archive of the files in my distribution, and made automake rules like so:
GIANTARCHIVE = My_big_archive.tar.lz
dist_pkgdata_DATA = $(GIANTARCHIVE)
install-data-hook:
cd $(DESTDIR)$(pkgdatadir); \
cat $(GIANTARCHIVE) | unlzma | $(TAR) --list > uninstall_manifest.txt; \
cat $(GIANTARCHIVE) | unlzma | $(TAR) --no-same-owner --extract; \
rm --force $(GIANTARCHIVE); \
cat uninstall_manifest.txt | sed --expression='s/^\|$$/"/g' | xargs chmod a=rX,u+w
uninstall-local:
cd $(DESTDIR)$(pkgdatadir); \
cat uninstall_manifest.txt | sed --expression='s/ /\\ /g' | xargs rm --force; \
rm --force uninstall_manifest.txt
This way, automake installs My_big_archive.tar.lz in the $(pkgdata) directory, and extracts it there, making a list of all the files that were in it, so it can uninstall them later. This also runs much faster than listing each file as an install target, even if you were to autogenerate that list.
I would write a script (either as a separate shell script, or in the Makefile.am), that is run as part of the install-data-hook target.

Resources