Hardware acceleration RPI4 FFmpeg - ffmpeg

Question

I'm trying to get to work hardware acceleration on my Raspberry pi 4-64. I'm using FFmpeg and AFAIK hwaccel can be reached by using OpenMAX or V4L2-M2M.
After '--enable-omx' and 'enable-omx-rpi' for FFmpeg, build fails with error: 'OMX_Core.h not found'. If I will provide manually omx headers, then it will compile but FFmpeg encoding will fail due to missing libraries: bcm_host.so and libopenmaxil.so
I have tried reverting to userland by DISABLE_VC4GRAPHICS = "1", it produced bcm_host.so, but not libopenmaxil.so. I have tried different combinations of virtual providers and graphics settings but without success.
Is it possible to access omx hardware acceleration on RPI4-64?
Steps to reproduce the issue:
1.Download latest Poky distro, meta-openembedded, meta-raspberrypi
2.Enable omx, omx-rpi support for FFmpeg
3.Link headers for FFmpeg
4.Build and try to use h264_omx
How do I get missing library libopenmaxil.so and everything I need for hwaccel?
poky master: commit 5d47cdf448b6cff5bb7cc5b0ba0426b8235ec478
meta-openembedded master: commit daa50331352c1f75da3a8ef6458ae3ddf94ef863
meta-raspberrypi master: commit 8d163dd
BTW, by using V4L2-M2M, I'm getting green shadows on the resulting video. Maybe can someone point me in the right direction?

Answer 1

You have to provide some extra flags to point ffmpeg to the right header and library locations, both at compile time and at run time.
This is what I used to cross-compile ffmpeg for AArxh64:
./configure \
--arch="${HOST_ARCH}" \
--target-os="linux" \
--prefix="/usr/local" \
--sysroot="${RPI_SYSROOT}" \
--enable-cross-compile \
--cross-prefix="${HOST_TRIPLE}-" \
--toolchain=hardened \
--enable-gpl --enable-nonfree \
--enable-avresample \
--enable-libvpx --enable-libx264 --enable-libxvid \
--enable-omx --enable-omx-rpi --enable-mmal --enable-neon \
--enable-shared \
--disable-static \
--disable-doc \
--extra-cflags="$(pkg-config --cflags mmal) \
-I${RPI_SYSROOT}/usr/local/include \
-I${RPI_SYSROOT}/opt/vc/include/IL" \
--extra-ldflags="$(pkg-config --libs-only-L mmal) \
-Wl,-rpath-link,${RPI_SYSROOT}/opt/vc/lib \
-Wl,-rpath,/opt/vc/lib"
Note that pkg-config is configured for cross-compilation as well, it looks in the Raspberry Pi sysroot, not in the build machine root. This is done by setting the right environment variables here.
The -I flags specify the include paths, and the -L flags returned by pkg-config --libs-only-L are the library paths. -Wl passes a list of comma separared arguments to the linker. -rpath-link is used to find shared libraries required by other shared libraries at link time, -rpath is used to find the libraries at run time. This is required because the userland libraries are in a nonstandard location, ld will not search in /opt/vc/lib by default.
You can find the toolchains, Dockerfiles and install scripts I used on my GitHub: https://github.com/tttapa/RPi-Cpp-Toolchain/tree/master/toolchain/docker/rpi3/aarch64/aarch64-cross-build
The userland script is here: https://github.com/tttapa/RPi-Cpp-Toolchain/blob/76ac03741bc7b7da106ae89884c7bada96768a07/toolchain/docker/rpi3/aarch64/aarch64-cross-build/install-scripts/userland.sh
And the ffmpeg script is here: https://github.com/tttapa/RPi-Cpp-Toolchain/blob/76ac03741bc7b7da106ae89884c7bada96768a07/toolchain/docker/rpi3/aarch64/aarch64-cross-build/install-scripts/ffmpeg.sh
There's some more documentation about the compilation process and the files used in the repository here (though not specifically about ffmpeg).