Using a portable Makefile for AWS Lambda - makefile

I'm using a Makefile for building a lambda extension. It works fine when I run a sam build command from Azure DevOps using Ubuntu. However, sometimes I want to build from my PC and the Makefile doesn't work because commands are different. I suppose I could setup command aliases for dos but that seems treacherous. I'd prefer to use something like OS detecting makefile but when I tried that it didn't work out.
Here's the pertinent snippet from my template:
TokenCacheLayer:
Type: AWS::Serverless::LayerVersion
Metadata:
BuildMethod: makefile
Properties:
LayerName: !Ref TokenCacheLayerName
Description: AWS Lambda extension to cache tokens aquired from an external API.
ContentUri: ./token-cache
LicenseInfo: MIT-0
CompatibleRuntimes:
- nodejs12.x
RetentionPolicy: Retain
Here's my Makefile:
build-TokenCacheLayer:
cp -R extensions "$(ARTIFACTS_DIR)"
cp -R token-cache-extension "$(ARTIFACTS_DIR)"
Here's the 2 ways I've tried altering to make it OS independent:
build-TokenCacheLayer:
ifeq ($(OS),Windows_NT)
xcopy extensions "$(ARTIFACTS_DIR)\extensions"
xcopy token-cache-extension "$(ARTIFACTS_DIR)"
else
cp -R extensions "$(ARTIFACTS_DIR)"
cp -R token-cache-extension "$(ARTIFACTS_DIR)"
endif
and
ifeq ($(OS),Windows_NT)
build-TokenCacheLayer:
xcopy extensions "$(ARTIFACTS_DIR)\extensions"
xcopy token-cache-extension "$(ARTIFACTS_DIR)"
else
build-TokenCacheLayer:
cp -R extensions "$(ARTIFACTS_DIR)"
cp -R token-cache-extension "$(ARTIFACTS_DIR)"
endif
The first way I got the error:
Error: CustomMakeBuilder:MakeBuild - Make Failed: process_begin: CreateProcess(NULL, cp -R extensions D:\Development\AWS.Integration\buildpro\layers\sam-build\TokenCacheLayer, ...) failed.
make (e=2): The system cannot find the file specified.
make: *** [D:\Development\AWS.Integration\buildpro\layers\token-cache\Makefile:6: build-TokenCacheLayer] Error 2
The second gave me this:
Error: CustomMakeBuilder:MakeBuild - Make Failed: D:\Development\AWS.Integration\buildpro\layers\token-cache\Makefile:2: *** recipe commences before first target. Stop.
Update
I went back and tried the first way again and while I'm not getting an error it doesn't copy the files either.

Related

Is there a Buildroot setting for making kernel source available to compile against?

I've been given a .zip file containing source for a proprietary kernel module. Once unzip'd, there is an install script that needs to be run. The install script untar's the actual source and builds the kernel module. It requires kernel headers to compile against.
Here is my Buildroot .mk file:
FOOCO_VERSION = 1.0
FOOCO_SOURCE = cust_kernel_drvr.zip
FOOCO_SITE = /mnt/third-party/fooco
FOOCO_SITE_METHOD = local
define FOOCO_CONFIGURE_CMDS
unzip $(#D)/$(FOOCO_SOURCE) -d $(#D)
endef
define FOOCO_BUILD_CMDS
chmod +x $(#D)/TOOLS/Linux_x64/DRIVER/install
cd $(#D)/TOOLS/Linux_x64/DRIVER; $(SHELL) ./install
rm -rf $(#D)
endef
$(eval $(generic-package))
This results in the following log output and error:
(Note: I enabled debugging that shows the start and end of each step.)
DEBUG: start | rsync | fooco
>>> fooco 1.0 Syncing from source dir /mnt/third-party/fooco
rsync -au --chmod=u=rwX,go=rX --exclude .svn --exclude .git --exclude .hg --exclude .bzr --exclude CVS /mnt/third-party/fooco/ /root/buildroot-2022.02.1/output/build/fooco-1.0
DEBUG: end | rsync | fooco
DEBUG: start | configure | fooco
>>> fooco 1.0 Configuring
unzip /root/buildroot-2022.02.1/output/build/fooco-1.0/cust_kernel_drvr.zip -d /root/buildroot-2022.02.1/output/build/fooco-1.0
Archive: /root/buildroot-2022.02.1/output/build/foofo-1.0/cust_kernel_drvr.zip
[snip]
creating: /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/
creating: /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/
inflating: /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/install
inflating: /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/cust_kernel_drvr-1.2.0.15-0.noarch.rpm
inflating: /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/cust_kernel_drvr.tar.gz
inflating: /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/license_gpl.txt
[snip]
DEBUG: end | configure | fooco
DEBUG: start | build | fooco
>>> fooco 1.0 Building
chmod +x /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/install
cd /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER; /bin/bash ./install
Extracting archive..OK!
Compiling the driver...Error: make[1]: Entering directory '/root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/fooco_cust/src/linux/driver'
common.mk:82: *** Kernel header files not in any of the expected locations.
common.mk:83: *** Install the appropriate kernel development package, e.g.
common.mk:84: *** kernel-devel, for building kernel modules and try again. Stop.
make[1]: Leaving directory '/root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/fooco_cust/src/linux/driver'
Error: unable to find driver file (fooco_cust.ko) in /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/fooco_cust/src/linux/driver
rm -rf /root/buildroot-2022.02.1/output/build/fooco-1.0
DEBUG: end | build | fooco
touch: cannot touch '/root/buildroot-2022.02.1/output/build/fooco-1.0/.stamp_built': No such file or directory
make: *** [/root/buildroot-2022.02.1/output/build/fooco-1.0/.stamp_built] Error 1
package/pkg-generic.mk:289: recipe for target
'/root/buildroot-2022.02.1/output/build/fooco-1.0/.stamp_built' failed
I found that the make files that came with the kernel module are looking in several places for the kernel headers:
/lib/modules/${BUILD_KERNEL}/source \
/lib/modules/${BUILD_KERNEL}/build \
/usr/src/linux-${BUILD_KERNEL} \
/usr/src/linux-$(${BUILD_KERNEL} | sed 's/-.*//') \
/usr/src/kernel-headers-${BUILD_KERNEL} \
/usr/src/kernel-source-${BUILD_KERNEL} \
/usr/src/linux-$(${BUILD_KERNEL} | sed 's/\([0-9]*\.[0-9]*\)\..*/\1/') \
/usr/src/linux \
/usr/src/kernels/${BUILD_KERNEL} \
/usr/src/kernels
Why is the kernel source not visible to this build? I thought that, since Buildroot is building the kernel as part of the overall process, the header files would be available for subsequent kernel module compiles. Am I missing a setting? I feel that I'm not understanding the Buildroot process in a basic way, even after referring to the manual many times.
I'm using Buildroot 2022.02.1 and kernel 5.15.33.
Your download/extract logic is very convoluted. You should really use something like this:
FOO_SITE = /mnt/third-party/fooco
FOO_SOURCE = cust_kernel_drvr.zip
FOO_SITE_METHOD = file
define FOO_EXTRACT_CMDS
unzip $(FOO_DLDIR)/$(FOOCO_SOURCE) -d $(#D)
endef
Regarding the build issue: it is impossible to help without studying the specific build system of this kernel module. Very likely you will need to pass some environment variables to tell the build system where your kernel source code is located, and possibly other things. But without looking at the specific details, it's impossible to help you.
You can have a look at how standard out of tree kernel modules are handled by looking at the package/pkg-kernel-module.mk code. However, that will not be directly useful to a package like yours that uses a custom installation script.
The magic was the LINUX_DIR variable which, according to Buildroot user manual:
contains the path to where the Linux kernel has been extracted and built.
I was able to patch the install script to send this variable to the make file that was looking for the kernel.

Makefile: No rules to make target files in servers

I want to create a makefile that syncs files from a server to my local PC. With this makefile I want to only copy from my server to my local computer those files that were modified. To do that I did a makefile based in boths answers from here and here. This is my code:
#!/usr/bin/make -f
FILES = *.csv
SCP = scp id#server:~/Results
LAST_SYNC = .last_sync
.PHONY: sync
sync : $(LAST_SYNC)
$(LAST_SYNC) : $(FILES)
$(SCP) $?
touch $(LAST_SYNC)
If I run make it appears the following error:
make: *** No rule to make target '*.csv', needed by '.last_sync'. Stop.
Are there any way to solve this problem?
EDIT:
I've just changed the problem with the FILES variable according to G.M.'s suggestion:
FILES := $(wildcard *.csv)
Now the following error appears:
scp id#server:~/Results results1.csv results2.csv results3.csv
results3.csv: Not a directory
make: *** [Makefile:13: .last_sync] Error 1

Unable to `make clean`: "No rule to make target ...Plo"

Question Summary
I'm trying to make clean and I'm getting the following output:
Makefile:835: /absolute/path/to/server/server/example/.deps/libfoo-bar.Plo: No such file or directory
Makefile:836: /absolute/path/to/server/server/example/.deps/libfoo-baz.Plo: No such file or directory
Makefile:837: /absolute/path/to/server/server/example/.deps/libfoo-qux.Plo: No such file or directory
Makefile:838: /absolute/path/to/server/server/example/.deps/libfoo-quux.Plo: No such file or directory
Makefile:839: /absolute/path/to/server/server/example/.deps/libfoo-quuz.Plo: No such file or directory
Makefile:840: /absolute/path/to/server/server/example/.deps/libfoo-corge.Plo: No such file or directory
Makefile:841: /absolute/path/to/server/server/example/.deps/libfoo-grault.Plo: No such file or directory
Makefile:842: /absolute/path/to/server/server/example/.deps/libfoo-garply.Plo: No such file or directory
Makefile:843: /absolute/path/to/server/server/example/.deps/libfoo-waldo.Plo: No such file or directory
Makefile:844: /absolute/path/to/server/server/example/.deps/libfoo-fred.Plo: No such file or directory
Makefile:845: /absolute/path/to/server/server/example/.deps/libfoo-plugh.Plo: No such file or directory
Makefile:846: /absolute/path/to/server/server/example/.deps/libfoo-xyzzy.Plo: No such file or directory
Makefile:847: /absolute/path/to/server/server/example/.deps/libfoo-babble.Plo: No such file or directory
Makefile:848: /absolute/path/to/server/server/example/.deps/libfoo-thud.Plo: No such file or directory
Makefile:848: /absolute/path/to/server/server/example/.deps/libfoo-flarp.Plo: No such file or directory
make: *** No rule to make target '/absolute/path/to/server/server/example/.deps/libfoo-flarp.Plo'. Stop.
How can I successfully do a make clean? I don't understand what these .Plo files are, and they are usually something I don't mess with.
Investigation
My Makefile.am
The Makefile.am which is used to generate the Makefile from which I'm trying to run the clean contains this preamble:
include $(top_srcdir)/server/include.am
include $(top_srcdir)/server/tests/include.am
...
The first $(top_srcdir)/server/include.am includes the following:
...
server_libdir = $(exec_prefix)/lib
serverdir = $(top_srcdir)/server
...
and in the specified $(top_srcdir)/server/tests/include.am, I have a lot of things including:
...
server_lib_LTLIBRARIES = libfoo.la
libfoo_la_SOURCES = \
/absolute/path/to/server/server/example/bar.c \
/absolute/path/to/server/server/example/baz.c \
/absolute/path/to/server/server/example/qux.c \
/absolute/path/to/server/server/example/quux.c \
/absolute/path/to/server/server/example/quuz.c \
/absolute/path/to/server/server/example/corge.c \
/absolute/path/to/server/server/example/grault.c \
/absolute/path/to/server/server/example/garply.c \
/absolute/path/to/server/server/example/waldo.c \
/absolute/path/to/server/server/example/fred.c \
/absolute/path/to/server/server/example/plugh.c \
/absolute/path/to/server/server/example/xyzzy.c \
...
Investigating the Makefile Contents
If I look at the Makefile directly and find the lines generating these errors, I see the following:
include /absolute/path/to/server/server/example/$(DEPDIR)/libfoo-bar.Plo
include /absolute/path/to/server/server/example/$(DEPDIR)/libfoo-baz.Plo
include /absolute/path/to/server/server/example/$(DEPDIR)/libfoo-qux.Plo
...
Where are the .Plo files generated?
Frustratingly, there seems to be the exact .Plo files that I need generated all over the place, just not in the right place; for example these files exist:
/absolute/path/to/server/tests/absolute/path/to/server/server/example/$(DEPDIR)/libfoo-bar.Plo
/absolute/path/to/server/tests/absolute/path/to/server/server/example/$(DEPDIR)/libfoo-baz.Plo
/absolute/path/to/server/tests/absolute/path/to/server/server/example/$(DEPDIR)/libfoo-qux.Plo
...
and there are 7 other directories where these .Plo files are created.
Unsuccessful Attempts
Running config.status directly
I found this message on the GNU mailing list:
Actually, these .P files are created by config.status for each
directory it creates an Automake Makefile in. (BTW, make sure
./config.status --help' lists the relevant Makfiles in the Configuration files' section).
You can request the creation of the .P files for a given directory
by running
% rm -Rf kernel/framework/.deps
% ./config.status kernel/framework/Makefile depfiles
And ran ./config.status as described but the output of make clean hasn't changed
According to this other Stack Overflow question, they were able to generate .Plo files, but I don't know what is different for them.
re-run configure with the --disable-dependency-tracking option

Linux-kernel: make isoimage fails due to missing certs. Proceed how?

I tried make isoimage but got:
kernel/Makefile:135: *** No X.509 certificates found ***
CHK kernel/config_data.h
Kernel: arch/x86/boot/bzImage is ready (#1)
rm -rf arch/x86/boot/isoimage
mkdir arch/x86/boot/isoimage
for i in lib lib64 share end ; do \
if [ -f /usr/$i/syslinux/isolinux.bin ] ; then \
cp /usr/$i/syslinux/isolinux.bin arch/x86/boot/isoimage ; \
[...]
done
arch/x86/boot/Makefile:160: rule for target „isoimage“ failed
make[1]: *** [isoimage] error 1
arch/x86/Makefile:236: rule for target „isoimage“ failed
make: *** [isoimage] error 2
while using AUR build system.
So where does one get those certs from and where to put them?
on Arch-Linux & derivates, the certs will be put in place by:
pacman -S linux
That linux package should have "mkisolinux" as a dependency, because the
make isoimage
requires it. Still, one needs to manually copy
cp /usr/lib/syslinux/bios/isolinux.bin /usr/include/syslinux/isolinux.bin
for the target to actually succeed.
"make isoimage" does work after these measures usually.
The issue is reproducible and could be easily fixed in Arch.

Makefile.am - Manual target prerequisite + addprefix?

I have a autotool project where part of the source code is downloaded dynamically from the net (because of IP rights preventing direct redistribution) and then built.
I have a Makefile.am that works but I'm not happy about some of it's aspects.
Here it is:
INCLUDES = $(all_includes) -I$(top_srcdir)/include -I$(top_builddir)
AM_CFLAGS = -fPIC -Wall ${SYMBOL_VISIBILITY}
LIBVERSION=0:0:0
REFSRC_PATH=refsrc
REFSRC_SRC=refsrc/dtx.c refsrc/globdefs.c refsrc/host.c refsrc/mathhalf.c refsrc/sp_enc.c refsrc/sp_rom.c refsrc/vad.c refsrc/err_conc.c refsrc/homing.c refsrc/mathdp31.c refsrc/sp_dec.c refsrc/sp_frm.c refsrc/sp_sfrm.c
${REFSRC_PATH}/.downloaded:
./fetch_sources.py "${REFSRC_PATH}"
for f in `ls -1 "${REFSRC_PATH}"/*.{c,h}`; do \
sed -i -e"s/round/round_l2s/" "$$f"; \
done
touch $#
${REFSRC_PATH}/dtx.c: ${REFSRC_PATH}/.downloaded
lib_LTLIBRARIES = libgsmhr.la
libgsmhr_la_SOURCES = libgsmhr.c $(REFSRC_SRC)
clean-local:
-rm -rf ${REFSRC_PATH}
So essentially, libgsmhr.c is my main wrapper, then I download the source code in a refsrc/ subdirectory and patch it a little.
First problem is that in REFSRC_SRC I would have loved to use a $(addprefix ...) instead of repeating refsrc/ in front of each .c file. But that doesn't seem to work and autoreconf complains a little.
Failure details (when removing the refsrc/ prefix from REFSRC_SRC= and using $(addprefix ${REFSRC_PATH}/, ${REFSRC_SRC}) on the dependency list):
bash$ autoreconf -i
libgsmhr/Makefile.am:19: addprefix ${REFSRC_PATH}/, ${REFSRC_SRC}: non-POSIX variable name
libgsmhr/Makefile.am:19: (probably a GNU make extension)
(configure works fine)
bash$ make
...
make[2]: Entering directory `/tmp/ram/gapk.build/libgsmhr'
CC libgsmhr.lo
CCLD libgsmhr.la
make[2]: Leaving directory `/tmp/ram/gapk.build/libgsmhr'
...
(So as you see it didn't include any of the downloaded .c files, didn't even download them at all. The compile works because libgsmhr.c is a stub that doesn't use the symbols in those file yet)
Second problem is this rule:
${REFSRC_PATH}/dtx.c: ${REFSRC_PATH}/.downloaded
I have to explicitely list the first file (dtx.c) instead of using a wildcard like:
${REFSRC_PATH}/%.c: ${REFSRC_PATH}/.downloaded
If I try to use the wildcard, then autoreconf complains and also it just doesn't work ... (pattern doesn't match somehow).
Failure detail:
bash$ autoreconf -i
libgsmhr/Makefile.am:16: `%'-style pattern rules are a GNU make extension
(configure works fine)
bash$ make
...
make[2]: *** No rule to make target `refsrc/dtx.c', needed by `dtx.lo'. Stop.
...
Sylvain
You seem to be writing a makefile in GNUMake style, but actually running some other version of Make. If it's not obvious what autoreconf is calling, you could insert a rule in the makefile:
dummy:
#echo using $(MAKE)
$(MAKE) -v
If this theory proves correct, you can either persuade autoconf to use GNUMake, or write for the version it's using.

Resources