How to install directory structure recursively in OpenEmbedded BitBake recipe? - shell

I'd like to simplify a BitBake recipe that installs a large directory structure by using some sort of recursive install routine rather than calling install many times. The source directory layout is frequently changing during development, which is causing far more recipe revisions than I want to deal with.
As an example, how would the following do_install() be simplified from this:
do_install() {
install -d ${D}/foo
install -m 0644 ${S}/foo/*.* ${D}/foo
install -d ${D}/foo/a
install -m 0644 ${S}/foo/a/*.* ${D}/foo/a
install -d ${D}/foo/b
install -m 0644 ${S}/foo/b/*.* ${D}/foo/b
install -d ${D}/foo/c
install -m 0644 ${S}/foo/c/*.* ${D}/foo/c
install -d ${D}/bar
install -m 0644 ${S}/bar/*.* ${D}/bar
install -d ${D}/bar/a
install -m 0644 ${S}/bar/a/*.* ${D}/bar/a
install -d ${D}/bar/a/bananas
install -m 0644 ${S}/bar/a/bananas/*.* ${D}/bar/a/bananas
}
To something more like this pseudocode:
do_install() {
for each subdir in ${S}/foo/
install subdir recursively to ${D}/foo/subdir
end
for each subdir in ${S}/bar/
install subdir recursively to ${D}/bar/subdir
end
}
The top-level directories in our source files (foo & bar in the above example) rarely change, so calling them out in the recipe is fine. It's the lower level directories that are frequently changing.
It may be that cp -r ends up being the way to go, but I believe I've read that it's frowned upon in BitBake recipes, so I'm wondering if BitBake provides some alternate mechanism, or if there's
some other reasonably standardized way to do this.

We used to do it in that way:
do_install() {
find ${WORKDIR}/ -type f -exec 'install -m 0755 "{}" ${D}/var/www/' \;
}

The canonical form in OE is
cp -R --no-dereference --preserve=mode,links -v SOURCE DESTINATION
see the answer here (while they looks a bit different in code, the questions are semantically equivalent)

Related

Script is running perfectly on co-workers devices but gives me 'Invalid cross-device link'

My script is running perfectly on co-workers devices (MacOSX with Docker Desktop same as me), but gives me every time the same error and it does not move or only half, the libraries in the deps directory:
OSError: [Errno 18] Invalid cross-device link: '/tmp/pip-target-dzwe_2kc/lib/python/numpy' ->
'/foo/python/numpy'
My script :
#!/bin/bash
export PKG_DIR='python'
export SIDE_DEPS_DIR='deps'
rm -rf ${PKG_DIR} && mkdir -p ${PKG_DIR}
rm -rf ${SIDE_DEPS_DIR} && mkdir -p ${SIDE_DEPS_DIR}
docker run --rm -v $(pwd):/foo -w /foo lambci/lambda:build-python3.8 \
pip3 install -r requirements.txt -t ${PKG_DIR}
# move stuff to deps
find /${PKG_DIR} -maxdepth 1 -type d \
\( -name "pandas*" -o -name "numpy*" -o -name "numpy.libs*" -o -name "scipy*" -o -name "scipy.libs*" \) -exec mv '{}' ${SIDE_DEPS_DIR} \;
# zip side dependencies
zip -r ge_deps.zip deps
# zip layer
zip -r layers-python38-great-expectations.zip python
It's a script which uses a public lambda docker image to create a lambda layer (basically a zip that contains libraries) and which removes unwanted libraries to put them in another folder deps.
The above code will use the public Docker image lambci / lambda and will install in the empty python directory, libraries which come from a python package which is called 'great-expectations' and which helps to test pipelines of data (which is specified in requirements.txt and is great-expectations==0.12.7)
I have been stuck with this problem for a while and have not found a solution.
Had this exact problem just now.
/tmp and /foo are different devices - /tmp is within the docker OS and /foo is mapped to your local OS.
pip seems to be using shutil.rename() to move the built package from tmp to the final output location (/foo). This fails because they are different devices. Ideally pip would use shutil.move() instead, which will deal with a cross-device move.
As a workaround, you can change the temp folder used by PIP by setting TMPDIR before invoking the pip command. i.e. export TMPDIR=/foo/tmp before calling pip in the docker image. So, the whole command might be something like
docker run --rm -v $(pwd):/foo -w /foo lambci/lambda:build-python3.8 \
/bin/bash -c "export TMPDIR=/foo/tmp && pip3 install -r requirements.txt -t ${PKG_DIR}"
(multiple commands soln taken from https://www.edureka.co/community/10736/how-to-run-multiple-commands-in-docker-at-once - open to better suggestions!)
This will likely be slower because it's using the local OS for temp files, but it avoids the attempted 'rename' across devices from the temp folder to the final output folder.

Setup a global variable dynamically in a Makefile

I have the following code in a Makefile, which execute one sequence of command or another based on a environmental variable.
generate :
if test -z "$$VIRTUAL_ENV"; then \
$(PYTHON) -m fades -V &>/dev/null || $(PYTHON) -m pip install --user fades; $(PYTHON) -m fades -r requirements.txt script.py;"; \
else \
python -m pip install -r requirements.txt && python script.py; \
fi
It works as expected, but I would like to do the same thing on multiple targets, to use it on other files, without having to copy this snippet of code multiple times.
My idea would be to set a variable dynamically (based on the condition that has been evaluated), containing the one command or the other, to be used over and over again, like alias in Bash.
Is that a good idea? Is it possible to set a global alias in the Makefile so it can choose between two Python interpreters based on an environmental variable?
Assuming you're using GNU make, you can do it like this:
ifdef VIRTUAL_ENV
PYCMD = python -m pip install -r requirements.txt && python
else
PYCMD = $(PYTHON) -m fades -V >/dev/null 2>&1 || $(PYTHON) -m pip install --user fades; $(PYTHON) -m fades -r requirements.txt
endif
generate:
$(PYCMD) script.py
Note I changed &>/dev/null to >/dev/null 2>&1 because the former is a bash-only feature and is not valid in POSIX sh, and make (by default) always runs /bin/sh which is (on many systems) a POSIX sh.
I don't know why you're using python in one section and $(PYTHON) in the other; it seems like you'd want to use the same in both but anyway.

create symbolic link in bitbake recipe

I have a .bbappend recipe that I need to create a symbolic link in my system.
This is how it looks like now:
bernardo#bernardo-ThinkCentre-Edge72:~/yocto/genericx86-64-rocko-18.0.0/meta-datavision/recipes-devtools/oracle-java$ cat oracle-jse-jdk_1.7.0.bbappend
FILES_${PN} += "/lib64/ld-linux-x86-64.so.2"
do_install_append() {
install -d ${D}/lib64
ln -s ${D}/lib/ld-2.26.so ${D}/lib64/ld-linux-x86-64.so.2
}
However, only the directory /lib64 is created in the sysroot. The symlink /lib64/ld-linux-x86-64.so.2 is not being generated.
What changes should I make in my recipe in order to have this symlink correctly created?
The cleanest solution is to use the "-r" flag:
do_install_append() {
install -d ${D}/lib64
ln -s -r ${D}/lib/ld-2.26.so ${D}/lib64/ld-linux-x86-64.so.2
}
From the gnu ln man page:
-r, --relative create symbolic links relative to link location
Try to avoid usage of absolute paths:
do_install_append() {
install -d ${D}/lib64
cd ${D}/lib64
ln -s ../lib/ld-2.26.so ld-linux-x86-64.so.2
}
For Yocto 2.3 to 3.4:
do_install_append() {
install -d ${D}/lib64
lnr ${D}/lib/ld-2.26.so ${D}/lib64/ld-linux-x86-64.so.2
}
For Yocto 4.0+ (or if your host system has coreutils 8.16+):
do_install_append() {
install -d ${D}/lib64
ln --relative --symbolic ${D}/lib/ld-2.26.so ${D}/lib64/ld-linux-x86-64.so.2
}
Alternatively, you can also inherit relative_symlinks which will turn any absolute symlinks into relative ones, but this is less commonly used than lnr.
Cf:
Yocto 4.0 Migration Guide (Thanks to #parsley72 for the info!)
Yocto 2.3 Migration Guide
I had a look at how other recipes create links in the rootfs, and most seem to do it this way:
ln -sf /data/etc/bluetooth/main.conf ${D}/${sysconfdir}/bluetooth/main.conf
This command in the recipe will create the following link on the device:
/# ls -al /etc/bluetooth/main.conf
lrwxrwxrwx 1 root root 29 Sep 11 15:34 /etc/bluetooth/main.conf -> /data/etc/bluetooth/main.conf
You use the full, Yocto-generated path when creating the link, but you make it point to the "final" location in the rootfs.
This way you can use "absolute" paths and won't have to change the working directory in the recipe.
As of 2022-01-19 this seems to be the only way to get it to work (adapt to the filenames needed):
do_install() {
install -d ${D}${libdir}
install -m 0644 ${S}/libmine.so.0 ${D}${libdir}/
lnr ${D}${libdir}/libmine.so.0 ${D}${libdir}/libmine.so
}
FILES_${PN} += " \
${libdir}/libmine.so.0 \
${libdir}/libmine.so \
"
FILES_SOLIBSDEV = ""
INSANE_SKIP_${PN} += "dev-so"
You can do:
ln -s ../lib/ld-2.26.so ${D}/lib64/ld-linux-x86-64.so.2
or if you don't require the symbolic-link until after your target system has booted up (i.e. it's not a dependency of other packages you are building) you can also do:
ln -s /lib/ld-2.26.so ${D}/lib64/ld-linux-x86-64.so.2
ln doesn't care if your target is valid or exists when a symbolic-link is created. It will become valid after you boot your target-system (or somehow mount this filesystem to /). But indeed, relative links are recommended.
do_install_append () {
install -d 0755 ${D}/dir
install -d 0755 ${D}/dir/subdir
cd ${D}/dir/subdir
ln -sf /source_so_the_symbilic_link <name_of_the_symbolic_link>
}
FILES_${PN} += "/dir"

Error installing flite on Mac OSX

I have downloaded the latest source distribution of flite, and went about the usual process of installing it.
$ ./configure
$ make
$ sudo make install
However, I run into a strange error when I try to install the library to my system.
$ sudo make install
Installing
mkdir -p /usr/local/bin
mkdir -p /usr/local/lib
mkdir -p /usr/local/include/flite
/usr/bin/install -c -m 644 include/*.h /usr/local/include/flite
/usr/bin/install -c -m 755 ../bin/flite_time /usr/local/bin
cp -pd ../build/i386-darwin13.1.0/lib/libflite_cmu_us_kal.a ../build/i386-darwin13.1.0/lib/libflite_cmu_time_awb.a ../build/i386-darwin13.1.0/lib/libflite_cmu_us_kal16.a ../build/i386-darwin13.1.0/lib/libflite_cmu_us_awb.a ../build/i386-darwin13.1.0/lib/libflite_cmu_us_rms.a ../build/i386-darwin13.1.0/lib/libflite_cmu_us_slt.a ../build/i386-darwin13.1.0/lib/libflite_usenglish.a ../build/i386-darwin13.1.0/lib/libflite_cmulex.a ../build/i386-darwin13.1.0/lib/libflite.a /usr/local/lib
cp: illegal option -- d
usage: cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file target_file
cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file ... target_directory
make[1]: *** [install] Error 64
make: *** [install] Error 2
How can I fix this?
There a few subtle differences between the BSD cp that Mac uses and the GNU cp of most linux distributions.
Consider the following snippet of man cp from a linux box:
-d same as --no-dereference --preserve=links
-P, --no-dereference
never follow symbolic links in SOURCE
--preserve[=ATTR_LIST]
preserve the specified attributes (default: mode,ownership,timestamps), if possible additional attributes: context,
links, xattr, all
So basically what it's trying to do is "copy the following paths, and if they're links, just copy the link, not the underlying file."
The p option exists under Mac and is equivalent to the linux behavior. The d option, however, is absent.
I've tried to figure out a way to mimic the behavior of "copy links, not targets" with the Mac cp, and as far as I can tell, there's no pleasant way to do it.
There is, fortunately, a gross work around. From man cp under Mac:
Symbolic links are always followed unless the -R flag is set, in which case symbolic links are not followed, by default.
In other words, since we know we're only copying files, you can simply replace the d flag with the R flag. The behavior is technically different (very different), but it won't matter in this specific case. You'll need to find the cp flags used in the Makefile (hopefully in a CP variable at the top of the file) and simply change them.
If you're sure the cp is the last thing to be executed in the Makefile, you could also just copy and paste it instead of changing the Makefile.
I was able to solve this problem using Corbin's suggestion. After searching the Makefile, I was able to spot where the error originated.
I am using flite-2.0.0-release and the Makefile was located in the following directory:
/flite-2.0.0-release/main/.
The last couple lines has the following:
# The libraries: static and shared (if built)
cp -pd $(flite_LIBS_deps) $(INSTALLLIBDIR)
ifdef SHFLAGS
cp -pd $(SHAREDLIBS) $(VERSIONSHAREDLIBS) $(INSTALLLIBDIR)
endif
I've changed the to the following:
# The libraries: static and shared (if built)
cp -pR $(flite_LIBS_deps) $(INSTALLLIBDIR)
ifdef SHFLAGS
cp -pR $(SHAREDLIBS) $(VERSIONSHAREDLIBS) $(INSTALLLIBDIR)
endif
By replacing the cp -pd to cp -pR, I was able to successfully install flite. I hope this advice helps.
I ran into this same issue recently, TJ Rana put me in the right direction but here's the whole process of installing flite on MacOS (original article here):
Flitevox or Flitelib is an open source small run time speech engine. Pass it text and create an audio file with a robot saying it. Really cool and useful for some projects.
Flitelib is not a native filter available in FFmpeg build nor inside the source. Although the documentation states that –enable-libflite is required for config, installing flitelib is required before installing FFmpeg. If you do try to enable the filter you’ll get this error:
$ ./configure --disable-indevs --enable-libflite --enable-cross-compile
ERROR: libflite not found
If you think configure made a mistake, make sure you are using the latest
version from Git. If the latest version fails, report the problem to the
ffmpeg-user#ffmpeg.org mailing list or IRC #ffmpeg on irc.freenode.net.
Include the log file "ffbuild/config.log" produced by configure as this will help
solve the problem.
In this post, I’ll show you how to install flite and get it working with FFmpeg
First download flitevox from source and install:
$ git clone https://github.com/festvox/flite.git
$ cd flite/
$ ./configure
$ make
$ sudo make install
If you’re running linux this installation works perfectly. If you’re running MacOS, you’ll get this error:
$ sudo make install
Password:
Installing
mkdir -p /usr/local/bin
mkdir -p /usr/local/lib
mkdir -p /usr/local/include/flite
/usr/bin/install -c -m 644 include/*.h /usr/local/include/flite
/usr/bin/install -c -m 755 ../bin/flite_time /usr/local/bin
cp -pd ../build/x86_64-darwin19.0.0/lib/libflite_cmu_us_kal.a ../build/x86_64-darwin19.0.0/lib/libflite_cmu_time_awb.a ../build/x86_64-darwin19.0.0/lib/libflite_cmu_us_kal16.a ../build/x86_64-darwin19.0.0/lib/libflite_cmu_us_awb.a ../build/x86_64-darwin19.0.0/lib/libflite_cmu_us_rms.a ../build/x86_64-darwin19.0.0/lib/libflite_cmu_us_slt.a ../build/x86_64-darwin19.0.0/lib/libflite_usenglish.a ../build/x86_64-darwin19.0.0/lib/libflite_cmu_indic_lang.a ../build/x86_64-darwin19.0.0/lib/libflite_cmu_grapheme_lang.a ../build/x86_64-darwin19.0.0/lib/libflite_cmulex.a ../build/x86_64-darwin19.0.0/lib/libflite_cmu_indic_lex.a ../build/x86_64-darwin19.0.0/lib/libflite_cmu_grapheme_lex.a ../build/x86_64-darwin19.0.0/lib/libflite.a /usr/local/lib
cp: illegal option -- d
usage: cp [-R [-H | -L | -P]] [-fi | -n] [-apvXc] source_file target_file
cp [-R [-H | -L | -P]] [-fi | -n] [-apvXc] source_file ... target_directory
make[1]: *** [install] Error 64
make: *** [install] Error 2
That’s because MacOS uses different “cp” variables then linux. I found this Stack Overflow answer with the solution: https://stackoverflow.com/a/29075638/525576 but here’s the steps to fix it.
In the folder “flite/main” you’ll need to edit the Makefile with the MacOS version of the command:
$ cd main/
$ vim Makefile
Replace the following (from TJ Rana):
# The libraries: static and shared (if built)
cp -pd $(flite_LIBS_deps) $(INSTALLLIBDIR)
ifdef SHFLAGS
cp -pd $(SHAREDLIBS) $(VERSIONSHAREDLIBS) $(INSTALLLIBDIR)
endif
to (-pd to -pR):
# The libraries: static and shared (if built)
cp -pR $(flite_LIBS_deps) $(INSTALLLIBDIR)
ifdef SHFLAGS
cp -pR $(SHAREDLIBS) $(VERSIONSHAREDLIBS) $(INSTALLLIBDIR)
endif
How we can try installing flite again:
$ sudo make install
flite should not show any errors and the installation should be complete.
Now back in ffmpeg source filter:
$ ./configure --enable-libflite --enable-cross-compile
$ make install
Installation will complete. To test if flite is working open a new terminal and type:
$ ffplay -f lavfi -i flite=text='Hello World!'
Hello world will speak!

Getting `make install` to source your bash_completion

This is the install part of my Makefile:
install:
for e in $(EXEC); do \
sudo cp --remove-destination ${CURDIR}/$$e /usr/local/bin; done
sudo cp ${CURDIR}/bin/stage2.d /etc/bash_completion.d/stage2
. /etc/bash_completion
Where "stage2" is the name of my executable.
The last line is what provides the issue. After adding a file into bash_completion.d directory I want to source the bash_completion. But calling source or . yields:
. /etc/bash_completion
/etc/bash_completion: 32: [[: not found
/etc/bash_completion: 38: [[: not found
/etc/bash_completion: 50: Bad substitution
make: *** [install] Error 2
I see two issues:
Using sudo in a Makefile rule is a bad idea. Avoid that, and call sudo make install instead.
Why would you want to source the bash_completion file in the non-interactive shell which is the make rule? It makes no sense.
As to solving them:
install:
$(INSTALL) -m 0755 -d $(DESTDIR)$(bindir)
$(INSTALL) -m 0755 -p $(EXEC) $(DESTDIR)$(bindir)/
$(INSTALL) -m 0755 -d $(DESTDIR)$(sysconfdir)/bash_completion.d
$(INSTALL) -m 0644 -p ${CURDIR}/bin/stage2.d $(DESTDIR)$(sysconfdir)/bash_completion.d/stage2
for the make rule part and
sudo make install && . /etc/bash_completion.d/stage2
for the actual running of that command. You will want to document the latter in a README, or in a line which the install: target prints when finished.
Make uses /bin/sh by default. you have to force it to use bash since [[ is not supported by normal sh.
gnu make lets you set SHELL variable [in the makefile] to force it to use bash. So you would need to add a line
SHELL=/bin/bash
at the top of your makefile

Resources