Linux Trace Tool lttng on Raspbian: Kernel tracer not available - lttng

Question

I have installed lttng on my Raspbian running on a Raspberry Pi 3 Model B+.
I get the following error when following a first demo from the website (as root):
# lttng create my-kernel-session --output=/tmp/my-kernel-trace
Spawning a session daemon
Session my-kernel-session created.
Traces will be written in /tmp/my-kernel-trace
A process is started correctly I assume:
# ps aux | grep lttng-sessiond
root 601 0.0 0.5 129320 5404 ? Ssl 12:00 0:00 lttng-sessiond --daemonize
root 652 0.0 0.0 4372 572 pts/0 S+ 12:30 0:00 grep lttng-sessiond
The following command returns an error:
# lttng list --kernel
Error: Unable to list kernel events: Kernel tracer not available
Error: Command error
Note that the following command does not list anything:
# lsmod | grep lttng
#
Are the lttng kernel modules not loaded? Is this the cause of the problem?
By the way, I am able to trace an application as explained in https://lttng.org/docs/v2.9/#doc-tracing-your-own-user-application.
I have installed lttng following the Debian installation:
# apt-get install lttng-modules-dkms
# apt-get install liblttng-ust-dev
# apt-get install lttng-tools
# apt-get install liblttng-ust-agent-java
# apt-get install python3-lttngust
And the following is installed:
# dpkg -l | grep lttng
ii liblttng-ctl0:armhf 2.9.3-1 armhf LTTng control and utility library
ii liblttng-ust-agent-java 2.9.0-2+deb9u1 all LTTng 2.0 Userspace Tracer (Java agent library)
ii liblttng-ust-agent-java-jni:armhf 2.9.0-2+deb9u1 armhf LTTng 2.0 Userspace Tracer (Java agent JNI interface)
ii liblttng-ust-ctl2:armhf 2.9.0-2+deb9u1 armhf LTTng 2.0 Userspace Tracer (trace control library)
ii liblttng-ust-dev:armhf 2.9.0-2+deb9u1 armhf LTTng 2.0 Userspace Tracer (development files)
ii liblttng-ust-java 2.9.0-2+deb9u1 all LTTng 2.0 Userspace Tracer (Java support library)
ii liblttng-ust-java-jni:armhf 2.9.0-2+deb9u1 armhf LTTng 2.0 Userspace Tracer (JNI interface)
ii liblttng-ust-python-agent0:armhf 2.9.0-2+deb9u1 armhf LTTng 2.0 Userspace Tracer (Python agent native library)
ii liblttng-ust0:armhf 2.9.0-2+deb9u1 armhf LTTng 2.0 Userspace Tracer (tracing libraries)
ii lttng-modules-dkms 2.9.0-1 all Linux Trace Toolkit (LTTng) kernel modules (DKMS)
ii lttng-tools 2.9.3-1 armhf LTTng control and utility programs
ii python3-lttngust 2.9.0-2+deb9u1 all LTTng 2.0 Userspace Tracer (Python 3 UST agent)
Some more info on my Linux distribution:
# uname -a
Linux realtimepi 4.14.52-rt34-v7+ #5 SMP PREEMPT RT Wed Aug 1 14:22:59 CEST 2018 armv7l GNU/Linux
And:
# cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)"
NAME="Raspbian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
Did I incorrectly install lttng, is there a mismatch with my Linux distribution, are the lttng kernel modules not correctly loaded, or is there a different problem?
Thanks,
Bart

Answer 1

From the uname -a output I can see that you are running a PREEMPT RT Linux kernel as you probably know because your hostname is realtimepi.
When you are running a kernel that was not packaged by your distribution you can't simply install the lttng-modules kernel modules as the Linux kernel header files installed on your machine would not match the ones expected by the modules package.
To use the lttng-modules, you will need to compile them from source. Have a look at the documentation on that subject: https://lttng.org/docs/v2.10/#doc-building-from-source. In your case, i believe you only need to build the modules from source.
Make sure you uninstall the lttng-modules-dkms package and then restart the lttng-sessiond process before trying again.

Answer 2

There is a problem with the RT kernel. This causes the following warning when compiling lttng-modules:
depmod: WARNING: /lib/modules/4.14.66-rt40-v7/extra/lttng-tracer.ko needs unknown symbol __migrate_disabled
Although this is reported as a warning, it causes the error.
If you adapt the source code of the lttng-modules 2.10 branch as explained in this bug report and compile lttng-modules again, the warning is absent and the kernel events can be listed.
Thank you fdeslaur for this answer!