When operating within the framework known as "Yocto", how do I build source from within a monorepo, without storing it in the 'files' subdir? - embedded-linux

I have a mono-repository ("monorepo") that contains the C++ source code for my application, as well as a tree of OE/Yocto files and directories that form the bitbake recipes required to build my final image.
I wish to build and install my application into the image, but as far as I can tell, the Yocto philosophy is that source code is "fetched" (e.g. via git), from an external place, before it is built. But in my case the source code resides in the same repository. It doesn't seem to make sense to me that the entire repository is downloaded again, by bitbake, just to fetch the source for this application, so I'm looking for a better way.
I'm familiar with the idea of putting all the source in the files/ subdirectory, alongside the recipe itself. The issue I have with that is that I don't want to keep the source in the Yocto layer's recipe tree. It can be built with the SDK, for example, or even with other completely unrelated toolchains, so it should not be buried within a Yocto layer. It has its own life, outside Yocto.
I'm also familiar with the EXTERNALSRC directive, that can be used to "point" to the source code with a relative path from the build directory. For example:
EXTERNALSRC = "${TOPDIR}/../../src/myproject"
However, this is frequently not recommended as a "production" mechanism due to path issues, and it also disables devtool:
ERROR: externalsrc is currently enabled for the myproject recipe. This prevents the normal do_patch task from working. You will need to disable this first.
So I'm looking for a recommendation on how to handle compiling an application that resides in the same repository as the recipe, without putting it in files/.
EDIT: I tried something along these lines:
SRC_URI = "file://${TOPDIR}/../../src/myproject/main.c \
file://${TOPDIR}/../../src/myproject/Makefile \
"
This did not compile, but with devtool modify myproject I was able to see that it has in fact copied the source into the build directory. The problem is that it's replicated the entire directory structure from the root all the way up to the original source directory, so my source is now sitting in a location like this:
/home/david/monorepo/yocto/build/workspace/sources/myproject/home/david/monorepo/src/myproject
do_compile will need to determine and set that working directory before it will compile.
This means that the path will change depending on the user and the location of where they've checked out the monorepo. This almost works, but doesn't seem usable as-is. Is there a way to modify where bitbake's "file" fetcher puts the source when given an absolute path?
EDIT 2:
I may have found a way that works with bitbake and devtool:
FILESEXTRAPATHS_prepend := "${TOPDIR}/../../src/myproject:"
SRC_URI = "file://main.c \
file://Makefile \
"
This seems to set up the devtool directory in a much saner way (no replication of the directory tree, just symlinks to the files in oe-local-files/* directory), and the bitbake recipe also builds and installs correctly.
Is this the right way to do it?
EDIT 3: Perhaps not, as FILESEXTRAPATHS is only intended to be modified by .bbappend recipes, not base .bb recipes - any comment on that?
Best practices dictate that you accomplish this by using FILESEXTRAPATHS from within a .bbappend file [source].
EDIT 4: PierreOlivier suggests using a relative symlink in the files/ directory to the application's source directory, so that SRC_URI can find the source as if it was actually present in files/. From what I can tell with my own experiments, this does seem to work, and devtool works with this also.
Are there any implications of this approach that I should be aware of?

Related

How to use GCC PGO with source code in different folder?

I've compiled PGO-instrumented build from A/src and collected the profile. Now I want to apply this profile when building from B/src. Is this possible? GCC complains about the lack of profile since absolute paths are different, but otherwise the code is exactly the same.
See the docs on -fprofile-prefix-path:
-fprofile-prefix-path=path
This option can be used in combination with profile-generate=profile_dir and profile-use=profile_dir to inform GCC where is the base directory of built source tree. By default profile_dir will contain files with mangled absolute paths of all object files in the built project. This is not desirable when directory used to build the instrumented binary differs from the directory used to build the binary optimized with profile feedback because the profile data will not be found during the optimized build. In such setups -fprofile-prefix-path=path with path pointing to the base directory of the build can be used to strip the irrelevant part of the path and keep all file names relative to the main build directory.
So when building from A/, set the prefix path to A/, and likewise for B/.

How to use an alternate go.mod file for local development?

Currently I am working on an API which uses Serverless Framework with Go.
I'm using the Serverless-offline plugin for local testing.
This API depends on a few other repositories (which I also maintain), which I import using the go.mod file.
However I am having a hard time refining my developer workflow.
Currently, if I want to make changes in a repository which this API depends upon, I have to alter the projects go.mod to include replace directives for the purpose of testing, but then I'm having to manually change it back for deployment to production.
Basically I'm looking for a way to include replace directives, which only get applied during local development. How has everyone else dealt with this problem?
Bonus question: Is there any way to run Serverless offline in docker? I'm finding that serverless-offline running on the bare metal is causing inconsistencies between different developers environments.
You can run go commands with an alternate go.mod file with the -modfile option:
From Build commands:
The -modfile=file.mod flag instructs the go command to read (and
possibly write) an alternate file instead of go.mod in the module root
directory. The file’s name must end with .mod. A file named go.mod
must still be present in order to determine the module root directory,
but it is not accessed. When -modfile is specified, an alternate
go.sum file is also used: its path is derived from the -modfile flag
by trimming the .mod extension and appending .sum.
Create a local.go.mod file with the necessary replace directive for development and build, for example, with:
go build -modfile=local.go.mod ./...

How to configure haxe build file (HXML) to build from src directory?

I'm trying to build a pre-existing HaxePunk project in sublime (switching away from FlashDevelop).
problem: Error: Could not process argument
Here's my .hxml file:
-neko
-cp "c:/path/to/project/src"
-main Main
I've read somewhere that you shouldn't use the /src convention for your src files. That's annoying, since I want assets and binaries in their own directories separate from src files. How do I properly configure this?
You really should use the the src convention and not stuff everything within the same directory. You also don't want to make the build specific to your machine, so in you example above you don't want an absolute path but a relative one. So try the following:
#content of c:/path/to/project/build.hxml
-neko bin/output.n
-cp src
-main Main
Note that for -cp you use the relative path. The path is relative to where haxe is executed. That usually coincides with where your build.hxml file is, but it is not mandatory.
Also, you didn't specify an output file for neko. Note that you will have to create the directory bin by hand because the compiler will not do that for you and will complain if it doesn't exist.
These information are general and in no way tied with Sublime. Sublime should play just nice with these settings.

telling GNU MAKE to use a user provided directory for object files creation

I want to build a file with GNU make on a machine which I have write permission only to the tmp directory.
When I try to build I get a permission error because MAKE is trying to put the object file in the build directory which I have no write permissions for.
is it possible to provide make a specific directory where to put the object files ?
Thanks,
Itay
is it possible to provide make a specific directory where to put the object files ?
If the makefile does not allow specifying a build directory you can copy (or symlink if possible) the sources into the destination directory and build there.
Autoconf-based projects normally allow this kind of usage.
It depends on where your make rules are coming from.
Usually, rules are written to be location independent: they are written such that whether the make command will work regardless of where the source files are. This is true for the built-in rules, but also for the rules in most Makefiles, if you got any.
In that case, all you need to do is copy or move everything into /tmp and run make there.
However, if you generated your Makefiles with a tool (e.g. a ./configure script, as used by GNU autoconf), the generation process may have introduced absolute paths, so you may need to redo the generation step(s) after copying everything to /tmp.

automake: How do I copy files to the build directory?

I am autotoolizing a library project, and this project has some example programs. I want the example programs to be distributed in the dist, but not installed.
Currently the demo programs are organized like thus:
src/*.cpp (library source)
include/*.h (library headers)
demos/demo.cpp (example program)
demos/RunDemo (script to run demo)
It is important that RunDemo be runnable after building the software, without requiring the "install" step.
So far I have been able to build the "demo" exectuable using a noinst_PROGRAMS target. However, after make in a VPATH build, the following is available:
build/src/.libs/libxxx.so (etc..)
build/demos/demo
As you can see, the RunDemo script needed to execute "demo" is not copied to the $(builddir). I have tried a few things, e.g., adding RunDemo to dist_noinst_SCRIPTS, as well as adding my own copy targets and trying to hook all.. no matter what I try, I always get the output,
$ make RunDemo
make: Nothing to be done for `../../../projects/demo/RunDemo'.
I seem to be unable to create a target in the builddir that says "if this file is not in the builddir, copy it from the srcdir."
Is this possible with automake?
You can make files accessible in the build tree after the ./configure step using the AC_CONFIG_LINKS macro (provided with autoconf) in your configure.ac script. It will create a symbolic link if possible, otherwise it will copy the file.
In your case it would look like
AC_CONFIG_LINKS([demos/RunDemo:demos/RunDemo])
From the autoconf manual:
Macro: AC_CONFIG_LINKS (dest:source..., [cmds], [init-cmds])
Make AC_OUTPUT link each of the existing files source to the
corresponding link name dest. Makes a symbolic link if possible,
otherwise a hard link if possible, otherwise a copy. The dest and
source names should be relative to the top level source or build
directory
Using dist_noinst_SCRIPTS is still necessary for the file to be distributed.

Resources