Compile Linux Kernel Modules into LLVM .bc bitcode - makefile

Question

Background
I'm trying to compile certain drivers within the Linux kernel: drm (drivers/gpu/drm/drm_drv.o) and radeon (drivers/gpu/drm/radeon/) gpu drivers. I'm using LLVM for the purposes of static analysis (tracking the arguments used in copy_to/from_user() invocations).
So far, I'm able to compile the actual modules using the Makefile as shown below:
make CC=clang CFLAGS=-emit-llvm drivers/gpu/drm/radeon/
But this does not actually emit any llvm bitcode -- I need the .bc files to run my pass with opt.
I only know how to generate .bc files when using clang directly (like below), but not with Makefiles...
clang -emit-llvm hello.c -c -o hello.bc
Since that worked, I grabbed the verbose output of the GNU make operation, changed gcc to clang, and ran it to create the .bc file, which also worked:
clang -emit-llvm [[tons of CFLAGS]] -c -o drm_drv.bc drivers/gpu/drm/drm_drv.c
The only problem with that is I can only process a single C file in the kernel module at a time. Also it's very tedious to do this approach...
Main Problem
Which brings me to my main question: How would you go about emitting llvm .bc bitcode files using the kernel's Makefiles?
Or, if .bc bitcode creation must be done on a per-file basis, then how would I link them all together at the end so that I can run an LLVM opt pass on the aggregate of all the .bc files in a kernel module?

Answer 1

The best way to get LLVM IR in bitcode form out of Clang is to do LTO with the -flto command line flag.
If you have multiple translation units you can combine them together using the llvm-lto tool to "link" the bitcode files. Typically it generates code, but you can get it to drop the merged LLVM IR module with the -save-merged-module flag.
But none of this is really a supported interface. If this is a significantly useful workflow, you should talk to the LLVM developers about supporting something more akin to ld's -r.

Answer 2

I used clang to instrument ext2 module. There are 3 things that you want to do:
1) convert .c into .bc
2) run your optimiser on .bc file, and create an ext2-opt.o file
3) create ext2-instrumented.ko file from the ext2-opt.o file.
along with this, you also require a linux version compiled with clang. I could get linux 4.17 compiled with clang version 3.8.1 after disabling few modules. you can get it here
now, lets move to step 1 - as you said, run Make in verbose mode
make V=1 M=fs/ext2
and grab all options that are spit out by Makefile. the default compiler will be gcc. replace gcc with clang, add --emit-llvm command, and replace .o with .bc. the resultant compilataion command on my machine looks like so:
cd ../../ && clang -emit-llvm -Wp,-MD,fs/ext2/.all.o.d -nostdinc -isystem /usr/local/bin/../lib/clang/3.8.1/include -I./arch/x86/include -Iarch/x86/include/generated/uapi -Iarch/x86/include/generated -Iinclude -I./arch/x86/include/uapi -Iarch/x86/include/generated/uapi -I./include/uapi -Iinclude/generated/uapi -include ./include/linux/kconfig.h -D__KERNEL__ -Qunused-arguments -Wno-unknown-warning-option -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -std=gnu89 -no-integrated-as -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -m64 -mtune=generic -mno-red-zone -mcmodel=kernel -funit-at-a-time -DCONFIG_X86_X32_ABI -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -DCONFIG_AS_CFI_SECTIONS=1 -DCONFIG_AS_FXSAVEQ=1 -DCONFIG_AS_SSSE3=1 -DCONFIG_AS_CRC32=1 -DCONFIG_AS_AVX=1 -DCONFIG_AS_AVX2=1 -DCONFIG_AS_SHA1_NI=1 -DCONFIG_AS_SHA256_NI=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -O2 -Wframe-larger-than=1024 -fno-stack-protector -Wno-unused-variable -Wno-format-invalid-specifier -Wno-gnu -Wno-asm-operand-widths -Wno-initializer-overrides -fno-builtin -Wno-tautological-compare -mno-global-merge -fno-omit-frame-pointer -fno-optimize-sibling-calls -g -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -Werror=implicit-int -Werror=strict-prototypes -Werror=date-time -Wno-initializer-overrides -Wno-unused-value -Wno-format -Wno-unknown-warning-option -Wno-sign-compare -Wno-format-zero-length -Wno-uninitialized -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(all)" -D"KBUILD_MODNAME=KBUILD_STR(ext2)" -mcmodel=kernel -c -o fs/ext2_instrumented/all.bc fs/ext2/all.c && cd -
this will create a .bc file for you. note that I have changed the resultant .bc codes directory to ext2_instrumented, i.e. my bc file is not created in fs/ext2 but in another folder called fs/ext2_instrumented. this is my work folder. I want this folder to not have any .c files, only .bc files. I need this because the default KBuild system looks for .c files in a folder. More on this later.
Step 2: Run all optimisation passes on your resultant ext2-instrumented.bc file using opt command like so:
opt -load $(DIR)/build/FSlice.so -constprop -sccp -mergereturn -sink -licm -reg2mem all.bc -o all.inst.bc
llvm-link -o all.inst2.bc $(DIR)/build/libFSlice.bc all.inst.bc
clang -mcmodel=kernel -c all.inst2.bc -o all.o
this will result into a .o file, which we will now compile into a .ko file in the next step:
Step 3:
make -C /lib/modules/$(shell uname -r)/build M=$(PWD) ext2instrumented.ko
Kbuild system is tough to understand, and even tougher to make modifications on. I would recommend use this 2 folder hack (1st folder to create .bc file and second folder to create .o and .ko files). To summarise, here is my resultant makefile in ext2_instrumented folder:
DIR=/home/fslice/fslice
obj-m += ext2instrumented.o
ext2instrumented-objs := all.o
all:
cd ../../ && clang -emit-llvm -Wp,-MD,fs/ext2/.all.o.d -nostdinc -isystem /usr/local/bin/../lib/clang/3.8.1/include -I./arch/x86/include -Iarch/x86/include/generated/uapi -Iarch/x86/include/generated -Iinclude -I./arch/x86/include/uapi -Iarch/x86/include/generated/uapi -I./include/uapi -Iinclude/generated/uapi -include ./include/linux/kconfig.h -D__KERNEL__ -Qunused-arguments -Wno-unknown-warning-option -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -std=gnu89 -no-integrated-as -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -m64 -mtune=generic -mno-red-zone -mcmodel=kernel -funit-at-a-time -DCONFIG_X86_X32_ABI -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -DCONFIG_AS_CFI_SECTIONS=1 -DCONFIG_AS_FXSAVEQ=1 -DCONFIG_AS_SSSE3=1 -DCONFIG_AS_CRC32=1 -DCONFIG_AS_AVX=1 -DCONFIG_AS_AVX2=1 -DCONFIG_AS_SHA1_NI=1 -DCONFIG_AS_SHA256_NI=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -O2 -Wframe-larger-than=1024 -fno-stack-protector -Wno-unused-variable -Wno-format-invalid-specifier -Wno-gnu -Wno-asm-operand-widths -Wno-initializer-overrides -fno-builtin -Wno-tautological-compare -mno-global-merge -fno-omit-frame-pointer -fno-optimize-sibling-calls -g -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -Werror=implicit-int -Werror=strict-prototypes -Werror=date-time -Wno-initializer-overrides -Wno-unused-value -Wno-format -Wno-unknown-warning-option -Wno-sign-compare -Wno-format-zero-length -Wno-uninitialized -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(all)" -D"KBUILD_MODNAME=KBUILD_STR(ext2)" -mcmodel=kernel -c -o fs/ext2_instrumented/all.bc fs/ext2/all.c && cd -
opt -load $(DIR)/build/FSlice.so -constprop -sccp -mergereturn -sink -licm -reg2mem all.bc -o all.inst.bc
llvm-link -o all.inst2.bc $(DIR)/build/libFSlice.bc all.inst.bc
clang -mcmodel=kernel -c all.inst2.bc -o all.o
make -C /lib/modules/$(shell uname -r)/build M=$(PWD) ext2instrumented.ko
clean:
make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean
rm -rf *.bc