Yocto How to overwrite a file of linux rootfs depending on an IMAGE-recipe? - embedded-linux

I'm trying to add a simple line in fstab within
the final rootfs that Yocto builds.
My first approach was to add my own fstab in my layer meta-mylayer/recipes-core/base-files/base-files/fstab and the proper meta-mylayer/recipes-core/base-files/base-files/base-files_%.bbappend which only have the following line:
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
And it works, but as the title of my question says, i want to modify fstab based on the recipe-image i want to build i.e. dev-image & prod-image.
After some investigation i think i have 2 options
Modify fstab within the recipe image, extending the do_install task...
dev-image.bb
--------------
DESCRIPTION = "Development Image"
[...]
inherit core-image
do_install_append () {
echo "======= Modifying fstab ========"
cat >> ${D}${sysconfdir}/fstab <<EOF
# The line i want to Add
EOF
}
[...]
--------------
Problem is that i'm actually not seeing my modified line in my final /etc/fstab and bitbake is not showing any build error or warning about this, actually, i'm not even able to see the echo-trace i put.
My second attempt was to handle these modifications with packages and depending on the recipe-image i will be able to add the package for *-dev or *-prod. This idea was taken from Oleksandr Poznyak in this answer in summary he suggest the following:
1) Create *.bbappend recipe base-files_%s.bbappend in your layer. It
appends to poky "base-files" recipe.
2) Create your own "python do_package_prepend" function where you should
make your recipe produce two different packages
3) Add them to DEPENDS in your image recipe
And based on his example i made my own recipe:
base-files_%.bbappend
-------------------------
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
SRC_URI += "file://fstab-dev \
file://fstab-prod \
"
PACKAGES += " ${PN}-dev ${PN}-prod"
CONFFILES_${PN}-dev = "${CONFFILES_${PN}}"
CONFFILES_${PN}-prod = "${CONFFILES_${PN}}"
pkg_preinst_${PN}-dev = "${pkg_preinst_${PN}}"
pkg_preinst_${PN}-prod = "${pkg_preinst_${PN}}"
RREPLACES_${PN}-dev = "${PN}"
RPROVIDES_${PN}-dev = "${PN}"
RCONFLICTS_${PN}-dev = "${PN}"
RREPLACES_${PN}-prod = "${PN}"
RPROVIDES_${PN}-prod = "${PN}"
RCONFLICTS_${PN}-prod = "${PN}"
python populate_packages_prepend() {
import shutil
packages = ("${PN}-dev", "${PN}-prod")
for package in packages:
# copy ${PN} content to packages
shutil.copytree("${PKGD}", "${PKGDEST}/%s" % package, symlinks=True)
# replace fstab
if package == "${PN}-dev":
shutil.copy("${WORKDIR}/fstab-dev", "${PKGDEST}/${PN}-dev/etc/fstab")
else:
shutil.copy("${WORKDIR}/fstab-prod", "${PKGDEST}/${PN}-prod/etc/fstab")
}
-------------------------
And in my recipe-image(dev-image.bb) i added base-files-dev packet
dev-image.bb
--------------
DESCRIPTION = "Development Image"
[...]
inherit core-image
IMAGE_INSTALL = " \
${MY_PACKETS} \
base-files-dev \
"
[...]
--------------
Problem with this, is that i'm not familiarized with phyton indentation so probably i'm messing things up, the error log shows as follows.
DEBUG: Executing python function populate_packages
ERROR: Error executing a python function in exec_python_func() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:populate_packages(d)
0003:
File: '/home/build/share/build_2/../sources/poky/meta/classes/package.bbclass', lineno: 1138, function: populate_packages
1134:
1135: workdir = d.getVar('WORKDIR')
1136: outdir = d.getVar('DEPLOY_DIR')
1137: dvar = d.getVar('PKGD')
*** 1138: packages = d.getVar('PACKAGES').split()
1139: pn = d.getVar('PN')
1140:
1141: bb.utils.mkdirhier(outdir)
1142: os.chdir(dvar)
File: '/usr/lib/python3.6/shutil.py', lineno: 315, function: copytree
0311: destination path as arguments. By default, copy2() is used, but any
0312: function that supports the same signature (like copy()) can be used.
0313:
0314: """
*** 0315: names = os.listdir(src)
0316: if ignore is not None:
0317: ignored_names = ignore(src, names)
0318: else:
0319: ignored_names = set()
Exception: FileNotFoundError: [Errno 2] No such file or directory: '${PKGD}'
DEBUG: Python function populate_packages finished
DEBUG: Python function do_package finished
I will really appreciate any clue or sort of direction, i'm not an Yocto expert so maybe the options that i suggest are not the most elegant and probably there is a better way to do it, so be free to give me any recommendation.
Thank you very much.
UPDATE:
As always, i was not the only one trying this, the way that i make it work was thanks this answer the only inconvenience with this is that you need to rm what you want to install through a .bbappend but for now is fine for me.
I also tried to do the same with bbclasses, which for me, it is a more elegant wayto do it, but i failed... i got the following error
ERROR: base-files-dev-3.0.14-r89 do_packagedata: The recipe base-files-dev is trying to install files into a shared area when those files already exist. Those files and their manifest location are:
I tried to rm fstab within the .bbappend but the same error is showed
Maybe somebody will share what i'm doing wrong...
If you don't find this post valuable please remove...

Your recipe which base on Oleksandr doesn't work due to dropped support for variables expansion in newer Poky.
https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#migration-2.1-variable-expansion-in-python-functions
Error explicit says:
Exception: FileNotFoundError: [Errno 2] No such file or directory: '${PKGD}'
It didn't expand the variable.
P.S.
This is not a proper answer to Your question but SO blocks comments.

Related

snakemake program not running all functions?

configfile: "config.yaml"
DATA = config['DATA_DIR']
BIN = config['BIN']
JASPAR = config['DATA_DIR']
RESULTS = config['RESULTS']
# JASPAR = "{0}/JASPAR2020".format(DATA)
JASPARS, ASSEMBLIES, BATCHES, TFS, BEDS = glob_wildcards(os.path.join(DATA, "{jaspar}", "{assembly}", "TFs", "{batch}", "{tf}", "{bed}.bed"))
rule all:
input:
expand (os.path.join(RESULTS, "{jaspar}", "{assembly}", "LOLA_dbs", "JASPAR2020_LOLA_{batch}.RDS"), jaspar = JASPARS, assembly = ASSEMBLIES, batch = BATCHES)
rule createdb:
input:
files = expand(os.path.join(RESULTS, "{jaspar}", "{assembly}", "data", "{batch}", "{tf}", "regions", "{bed}.bed"), zip, jaspar = JASPARS, assembly = ASSEMBLIES, batch = BATCHES, tf = TFS, bed = BEDS)
output:
os.path.join(RESULTS, "{jaspar}", "{assembly}", "LOLA_dbs", "JASPAR2020_LOLA_{batch}.RDS")
shell:
"""
R --vanilla --slave --silent -f {BIN}/create_lola_db.R \
--args {RESULTS}/{wildcards.jaspar}/{wildcards.assembly}/data/{wildcards.batch} \
{output};
"""
Why my snakemake program is not considering "createdb" rule. It is only considering "all". Can someone please help me with this?
"snakemake only runs the first rule in a Snakefile, by default." - SOURCE
If that first rule's input isn't clearly linked to createdb, it's not going to know to run createdb. Because the only rule it wants to run would have to depend on createdb.
I suspect your problem is the input to your rule createdb shouldn't have expand() used for all the wildcards already handled by rule all, just bed?. See here.
Also other avenues to consider for debugging is investigating what you unpacked glob_wildcards() to and what files is.

Yocto Bitbake fails on ros-image-turtlebot3-core.bb:do_rootfs due to missing rpm

I like to assemble a Linux image for RaspberryPi with ROS2 and Turtlebot3 stuff. Yocto environment is already set up and worked fine until end of last year. Image file was successfully built and started as expected at target system.
My new application needs further dependencies, therefore I pulled latest changes from origin repositories few days ago. Unfortunately, this seem to broke functionality of build system. Own changes not yet applied.
Build process fails at the last stage when rootfs should be created. It claims several rpm would be missing. It turned out they're existing at different location.
Below example is about glibc-localedata-pa-pk-2.31-r0.cortexa7t2hf_neon_vfpv4.rpm but after several tries I noticed that a tons of packages are affected.
Expected: build/BUILD-poky-foxy-dunfell/deploy/rpm/cortexa7t2hf_neon_vfpv4/glibc-localedata-pa-pk-2.31-r0.cortexa7t2hf_neon_vfpv4.rpm
Existing: build/BUILD-poky-foxy-dunfell/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/glibc-locale/2.31-r0/deploy-rpms/cortexa7t2hf_neon_vfpv4/glibc-localedata-pa-pk-2.31-r0.cortexa7t2hf_neon_vfpv4.rpm
Another afftected package:
Expected:
build/BUILD-poky-foxy-dunfell/deploy/rpm/cortexa7t2hf_neon_vfpv4/libatomic1-9.3.0-r0.cortexa7t2hf_neon_vfpv4.rpm
Existing: build/BUILD-poky-foxy-dunfell/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/gcc-runtime/9.3.0-r0/deploy-rpms/cortexa7t2hf_neon_vfpv4/libatomic1-9.3.0-r0.cortexa7t2hf_neon_vfpv4.rpm
Basically rpm are expected to be at build/BUILD-poky-foxy-dunfell/deploy/rpm/cortexa7t2hf_neon_vfpv4/ but they are at build/BUILD-poky-foxy-dunfell/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/XXX/YYY/deploy-rpms/cortexa7t2hf_neon_vfpv4/
Were there any changes in buildsystem or configuration since end of 2020 or any known issues found? Any help would be appreaciated, many thanks in advance!
$ bitbake ros-image-turtlebot3-core -k
Build Configuration:
BB_VERSION = "1.46.0"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "universal"
TARGET_SYS = "arm-poky-linux-gnueabi"
MACHINE = "raspberrypi3"
DISTRO = "poky"
DISTRO_VERSION = "3.1.11"
TUNE_FEATURES = "arm vfp cortexa7 neon vfpv4 thumb callconvention-hard"
TARGET_FPU = "hard"
DISTRO_NAME = "Poky (Yocto Project Reference Distro)"
ROS_DISTRO = "foxy"
ROS_VERSION = "2"
ROS_PYTHON_VERSION = "3"
meta
meta-poky
meta-yocto-bsp = "dunfell:6a7335170ed70263c4b2c55d10fe741c237c2357"
meta-oe
meta-python
meta-perl
meta-networking = "dunfell:c56d6309efc11e9f0f4a549be45a783c73a49ba0"
meta-raspberrypi = "dunfell:59c2d6f7a8b1239bd7b587b9180c2a55f9c695a2"
meta-ros-common
meta-ros2
meta-ros2-foxy
meta-ros-backports-hardknott
meta-ros-backports-gatesgarth = "ros2foxy:638a8b93e7b328fab9ccf2321bea039a93a57dde"
meta-communication
meta-system
meta-usercode = "ros2foxy:886b2fed39384970dc7476a15009765785b52512"
Initialising tasks: 100% |##########################################################################################| Time: 0:00:16
Sstate summary: Wanted 367 Found 365 Missed 2 Current 2770 (99% match, 99% complete)
NOTE: Executing Tasks
ERROR: ros-image-turtlebot3-core-1.0-r0 do_rootfs: Error executing a python function in exec_python_func() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:do_rootfs(d)
0003:
File: '/home/user/ros_target/poky/meta/classes/image.bbclass', lineno: 247, function: do_rootfs
0243: progress_reporter.next_stage()
0244:
0245: # generate rootfs
0246: d.setVarFlag('REPRODUCIBLE_TIMESTAMP_ROOTFS', 'export', '1')
*** 0247: create_rootfs(d, progress_reporter=progress_reporter, logcatcher=logcatcher)
0248:
0249: progress_reporter.finish()
0250:}
0251:do_rootfs[dirs] = "${TOPDIR}"
File: '/home/user/ros_target/poky/meta/lib/oe/rootfs.py', lineno: 975, function: create_rootfs
0971: env_bkp = os.environ.copy()
0972:
0973: img_type = d.getVar('IMAGE_PKGTYPE')
0974: if img_type == "rpm":
*** 0975: RpmRootfs(d, manifest_dir, progress_reporter, logcatcher).create()
0976: elif img_type == "ipk":
0977: OpkgRootfs(d, manifest_dir, progress_reporter, logcatcher).create()
0978: elif img_type == "deb":
0979: DpkgRootfs(d, manifest_dir, progress_reporter, logcatcher).create()
File: '/home/user/ros_target/poky/meta/lib/oe/rootfs.py', lineno: 366, function: __init__
0362: self.manifest = RpmManifest(d, manifest_dir)
0363:
0364: self.pm = RpmPM(d,
0365: d.getVar('IMAGE_ROOTFS'),
*** 0366: self.d.getVar('TARGET_VENDOR')
0367: )
0368:
0369: self.inc_rpm_image_gen = self.d.getVar('INC_RPM_IMAGE_GEN')
0370: if self.inc_rpm_image_gen != "1":
File: '/home/user/ros_target/poky/meta/lib/oe/package_manager.py', lineno: 765, function: __init__
0761: self.primary_arch = self.d.getVar('MACHINE_ARCH')
0762:
0763: if needfeed:
0764: self.rpm_repo_dir = oe.path.join(self.d.getVar('WORKDIR'), rpm_repo_workdir)
*** 0765: create_packages_dir(self.d, oe.path.join(self.rpm_repo_dir, "rpm"), d.getVar("DEPLOY_DIR_RPM"), "package_write_rpm", filterbydependencies)
0766:
0767: self.saved_packaging_data = self.d.expand('${T}/saved_packaging_data/%s' % self.task_name)
0768: if not os.path.exists(self.d.expand('${T}/saved_packaging_data')):
0769: bb.utils.mkdirhier(self.d.expand('${T}/saved_packaging_data'))
File: '/home/user/ros_target/poky/meta/lib/oe/package_manager.py', lineno: 733, function: create_packages_dir
0729: if destdir not in seendirs:
0730: bb.utils.mkdirhier(destdir)
0731: seendirs.add(destdir)
0732: try:
*** 0733: os.link(l, dest)
0734: except OSError as err:
0735: if err.errno == errno.EXDEV:
0736: bb.utils.copyfile(l, dest)
0737: else:
Exception: FileNotFoundError: [Errno 2] No such file or directory: '/home/user/ros_target/build/BUILD-poky-foxy-dunfell/deploy/rpm/cortexa7t2hf_neon_vfpv4/glibc-localedata-pa-pk-2.31-r0.cortexa7t2hf_neon_vfpv4.rpm' -> '/home/user/ros_target/build/BUILD-poky-foxy-dunfell/work/raspberrypi3-poky-linux-gnueabi/ros-image-turtlebot3-core/1.0-r0/oe-rootfs-repo/rpm/cortexa7t2hf_neon_vfpv4/glibc-localedata-pa-pk-2.31-r0.cortexa7t2hf_neon_vfpv4.rpm'
ERROR: Logfile of failure stored in: /home/user/ros_target/build/BUILD-poky-foxy-dunfell/work/raspberrypi3-poky-linux-gnueabi/ros-image-turtlebot3-core/1.0-r0/temp/log.do_rootfs.2708
ERROR: Task (/home/user/ros_target/3rdParty/meta-ros/meta-ros-common/recipes-core/images/ros-image-turtlebot3-core.bb:do_rootfs) failed with exit code '1'
NOTE: Tasks Summary: Attempted 9034 tasks of which 9033 didn't need to be rerun and 1 failed.
Summary: 1 task failed:
/home/user/ros_target/3rdParty/meta-ros/meta-ros-common/recipes-core/images/ros-image-turtlebot3-core.bb:do_rootfs
Summary: There were 358 WARNING messages shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
Edit:
I tried to make all missing RPM available by symlinks. Since really a lot (but still not all) packages were affected, I created short script for that:
#!/bin/bash
currDir=`pwd`
#find all existing RPM in WORK directory
find build/BUILD-poky-foxy-dunfell/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi -iname "*.rpm" > ${currDir}/tmp_allRpm.log
#find all RPM that are already deployed
cd build/BUILD-poky-foxy-dunfell/deploy/rpm/cortexa7t2hf_neon_vfpv4/
ls -1 *.rpm > ${currDir}/tmp_deployedRpm.log
cd $currDir
#create list of all RPM that are not yet deployed
grep -v -f ${currDir}/tmp_deployedRpm.log ${currDir}/tmp_allRpm.log > ${currDir}/tmp_toBeLinkedRpm.log
#create symlinks
xargs -i ln -s {} build/BUILD-poky-foxy-dunfell/deploy/rpm/cortexa7t2hf_neon_vfpv4/ <${currDir}/tmp_toBeLinkedRpm.log
This seemed to solved the origin issue but lead to further weired errors. So I decided to set up new Yocto/OpenEmbedded environment using my existing config files and recipes. It turned out to be a good decision - after several hours of compilation everything was done ;) Only downside of this is that I can't tell what went wrong and when it may happen again?!

cctools-binutils-darwin/bin/ranlib failed, malformed object (unknown load command)

I am attempting to build the gitlib-libgit2 Haskell package on macos via nix in roughly the following manner.
mkdir nix-repro && cd nix-repro
cabal2nix --no-check cabal://gitlib-libgit2 > gitlib-libgit2.nix
echo '(import <nixpkgs> {}).haskellPackages.callPackage ./gitlib-libgit2.nix {}' > default.nix
nix-build
I am seeing the following warnings and error:
ld: warning: /nix/store/spx9xz1jv3yhmqw8y3agki1fvdr2x2fv-libiconv-osx-10.11.6/lib/libiconv.dylib, ignoring unexpected dylib file
ld: warning: /nix/store/spx9xz1jv3yhmqw8y3agki1fvdr2x2fv-libiconv-osx-10.11.6/lib/libiconv.dylib, ignoring unexpected dylib file
/nix/store/s33984hx2gwcg2d4dgpcm4342md19qvh-cctools-binutils-darwin/bin/ranlib: object: dist/build/libHSgitlib-libgit2-3.1.2.1-1fYQZMedHRP3aXiBXJFDO2-ghc8.6.3.a(s2_meth.o) malformed object (unknown load command 1)
`ranlib' failed in phase `Ranlib'. (Exit code: 1)
builder for '/nix/store/yrran1p69pvdq1b34jqfq7dmd95j9fh9-gitlib-libgit2-3.1.2.1.drv' failed with exit code 1
error: build of '/nix/store/yrran1p69pvdq1b34jqfq7dmd95j9fh9-gitlib-libgit2-3.1.2.1.drv' failed
What does this mean, and how can I go about fixing it?
It only fails this way on macos, as far as I can tell. I'm on nixpkgs-unstable. I am able to build this package using other tools, so the failure must be something specific to the nix tooling I'm using to build it.
$ cat $HOME/.nix-defexpr/channels/nixpkgs/.git-revision
6e5caa3f8ac48750233ef82a94825be238940825
Here's the full gitlib-libgit2.nix expression if you don't want to run cabal2nix for yourself:
{ mkDerivation, base, bytestring, conduit, conduit-combinators
, containers, directory, exceptions, fast-logger, filepath, gitlib
, gitlib-test, hlibgit2, hspec, hspec-expectations, HUnit, mmorph
, monad-loops, mtl, resourcet, stdenv, stm, stm-conduit, tagged
, template-haskell, text, text-icu, time, transformers
, transformers-base, unliftio, unliftio-core
}:
mkDerivation {
pname = "gitlib-libgit2";
version = "3.1.2.1";
sha256 = "b90e0ad2e7e0f58379e02cbe60d2900c95f0a255c34bd3461f8ee5753a6aa23e";
libraryHaskellDepends = [
base bytestring conduit conduit-combinators containers directory
exceptions fast-logger filepath gitlib hlibgit2 mmorph monad-loops
mtl resourcet stm stm-conduit tagged template-haskell text text-icu
time transformers transformers-base unliftio unliftio-core
];
testHaskellDepends = [
base exceptions gitlib gitlib-test hspec hspec-expectations HUnit
transformers
];
doCheck = false;
description = "Libgit2 backend for gitlib";
license = stdenv.lib.licenses.mit;
}

Yocto/Bitbake recipe for adding empty directory to rootfs Embedded Linux

Is there any recipe for adding a new, empty directory to the rootfs? I tried adding this into one of my bbappend file:
do_install() {
install -d ${D}/tmp/myNewDir
}
FILES_${PN} += "/tmp/myNewDir"
but I am getting a non descriptive error, Function failed: do_install
There are several ways. The image command way is already described by StackedUser.
You can also try to extend some of your recipes (as you are doing in your question). I guess that you are seeing the error because you are overwriting the do_install task. You are probably wanting to extend it, so you should add _append to the task name, i.e.:
do_install_append () {
install -d ${D}/tmp/myNewDir
}
BTW, the error "Function failed: do_install" you are hitting usually show an error code or a problematic command. Maybe there is something.
Another way is to create a simple recipe and add it to the image, here is a stub:
SUMMARY = "XXX project directory structure"
# FIXME - add proper license below
LICENSE = "CLOSED"
PV = "1.0"
S = "${WORKDIR}"
inherit allarch
do_install () {
install -d ${D}/foo/bar
}
FILES_${PN} = "/foo/bar"
In our image recipe we have something like this to create a new directory:
create_data_dir() {
mkdir -p ${IMAGE_ROOTFS}/data
}
IMAGE_PREPROCESS_COMMAND += "create_data_dir;"

cx_Freeze '#rpath/libQtDeclarative.4.dylib': doesn't exist or not a regular file

from cx_Freeze import setup, Executable
# Dependencies are automatically detected, but it might need
# fine tuning.
buildOptions = dict(packages = ["idna","lib","gui","plugins"], excludes = ["Tcl","tcl"]
import sys
base = 'Win32GUI' if sys.platform=='win32' else None
executables = [
Executable('electrum-xvg', base=base, targetName = 'Electrum XVG',icon="electrum.icns")]
setup(name='electrum-xvg',
version = '1.0',
description = '',
options = dict(build_exe = buildOptions),
executables = executables])
I have the above setup.py file which I am using to try build application on OSX Sierra. But when I use python setup.py bdist_mac it raises error
#rpath/libQtDeclarative.4.dylib
error: can't copy '#rpath/libQtDeclarative.4.dylib': doesn't exist or not a regular file
libQtDeclarative.4.dylib is present in ~/anaconda/envs/pyqtapp/lib on my system but when I used otool -D libQtDeclarative.4.dylib it raised error that no such file exists, so I used
install_name_tool -id "#rpath/libQtDeclarative.4.dylib" libQtDeclarative.4.dylib
in ~/anaconda/envs/pyqtapp/lib now when I run otool -D libQtDeclarative.4.dylib I get
libQtDeclarative.4.dylib:
#rpath/libQtDeclarative.4.dylib
but cx_Freeze still raises the error
error: can't copy '#rpath/libQtDeclarative.4.dylib': doesn't exist or not a regular file
Try explicitly setting includes (list of relative paths):
includefiles = ['README.txt', 'CHANGELOG.txt', 'helpers\uncompress\unRAR.exe', , 'helpers\uncompress\unzip.exe']

Resources