raw_spin_lock(): unexpected null pointer exception - linux-kernel

So I adapted some of the /kernel/sched/rt.c code to write my own simple CPU scheduler, and I'm getting a null pointer dereference exception when I try to acquire a lock. This is despite me printk()'ing all of the relevant pointers, and seeing that they're not NULL.
//Snippet from my adaptation of update_curr_rt()
//wrr_rq is a struct wrr_rq*
printk("Before loop, wrr_rq pointer is %p\n",wrr_rq);
printk("Before loop, &wrr_rq->wrr_runtime_lock is %p\n",&wrr_rq->wrr_runtime_lock);
for_each_sched_wrr_entity(wrr_se) {
printk("1\n");
wrr_rq = wrr_rq_of_se(wrr_se);
printk("2\n");
raw_spin_lock(&wrr_rq->wrr_runtime_lock);
printk("3\n");
[ 263.595176] Before loop, wrr_rq is 00000000aebb4d6d
[ 263.596283] Before loop, &wrr_rq->wrr_runtime_lock is 0000000015dee87f
[ 263.597764] 1
[ 263.598141] wrr_rq_of_se: called
[ 263.598888] 2
[ 263.599268] BUG: kernel NULL pointer dereference, address: 0000000000000068
[ 263.600836] #PF: supervisor write access in kernel mode
[ 263.602027] #PF: error_code(0x0002) - not-present page
...
[ 263.656134] RIP: 0010:_raw_spin_lock+0x7/0x20
I've printed all the relevant pointers and seen they're not NULL (and have values quite a bit above 0), but I still get this exception. I tried using the elixir browser to see what is happening with the raw_spin_lock() macro, and it doesn't seem like anything crazy is happening...
In addition, the runqueue lock is already held when this code is called (the runqueue lock is acquired by task_sched_runtime()).
Any thoughts appreciated.
Thanks.

Credit to #0andriy: It turns out that kernel NULL pointers when printed by %p get hashed to some other unique value that may not be NULL, and so when I printed things with %px I saw they were in fact NULL.

Related

How to debug/find cma allocation failure reason?

Is there any open source debug methods/patch available for CMA failure debug ?
How to know the reason of CMA allocation failure
1.When the cma_allocation fails it dump backtrace of failure.
eg.
[ 35.360001] page:bef55be8 count:58 mapcount:56 mapping:bc4001dc index:0x3
[ 35.366855] flags: 0x8019040c(referenced|uptodate|arch_1|mappedtodisk|unevictable|mlocked)
[ 35.375173] raw: 8019040c bc4001dc 00000003 00000037 0000003a b9eb1a98 b9eb1a98 00000000
[ 35.383299] raw: be008c00
[ 35.385916] page dumped because: VM_BUG_ON_PAGE(PageLRU(page) || PageUnevictable(page))
[ 35.393995] page->mem_cgroup:be008c00
[ 35.397668] ------------[ cut here ]------------
[ 35.402281] kernel BUG at mm/vmscan.c:1350!
[ 35.406458] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP ARM
[ 37.778079] Backtrace:
[ 37.780531] [<80360610>] (shrink_page_list) from [<803617c8>]
(reclaim_clean_pages_from_list+0x14c/0x1a8)
[ 37.790093] r10:b9c6fb88 r9:b9c6fb9c r8:b9c6fb0c r7:8141e100 r6:81216588 r5:b9c6fb9c
[ 37.797914] r4:bf05ffb8
[ 37.800444] [<8036167c>] (reclaim_clean_pages_from_list) from [<80352b2c>] (alloc_contig_range+0x17c/0x4e0)
[ 37.810178] r10:00000000 r9:8121e384 r8:814790c4 r7:b9c6e000 r6:0006a000 r5:00081a00
[ 37.817999] r4:b9c6fb9c
[ 37.820529] [<803529b0>] (alloc_contig_range) from [<803bd8c8>] (cma_alloc+0x154/0x5dc)
[ 37.828527] r10:00040000 r9:00017c00 r8:fffffff4 r7:00017c00 r6:8147bf24 r5:00009e00
[ 37.836347] r4:00069e00
[ 37.838878] [<803bd774>] (cma_alloc) from [<80694188>] (dma_alloc_from_contiguous+0x40/0x44)
[ 37.847310] r10:00000000 r9:80607f30 r8:b9c6fd64 r7:00017c00 r6:17c00000 r5:81216588
[ 37.855131] r4:00000001
[ 37.857661] [<80694148>] (dma_alloc_from_contiguous) from [<80218720>] (__alloc_from_contiguous+0x54/0x144)
[ 37.867396] [<802186cc>] (__alloc_from_contiguous) from [<80218854>] (cma_allocator_alloc+0x44/0x4c)
[ 37.876523] r10:00000000 r9:b9c6fe08 r8:81216588 r7:00c00000 r6:b94d0140 r5:80607f30
[ 37.884343] r4:00000001
[ 37.886870] [<80218810>] (cma_allocator_alloc) from [<80217e28>] (__dma_alloc+0x19c/0x2e4)
[ 37.895125] r5:bd2da400 r4:014000c0
[ 37.898695] [<80217c8c>] (__dma_alloc) from [<80218000>] (arm_dma_alloc+0x4c/0x54)
[ 37.906258] r10:00000080 r9:17c00000 r8:80c01778 r7:bd2da400 r6:8148ff6c r5:00c00000
[ 37.914079] r4:00000707
[ 37.916608] [<80217fb4>] (arm_dma_alloc) from [<80607f30>]
[ 37.924690] r5:81490278 r4:81216588
You can debug cma allocation failure using this backtrace.
Continuously check /proc/pagetypeinfo before allocation and after allocation, it will give you hint if pages are back or not to the initial stage.
To get pages information refer below link:-
link
stable kernel bug here is the patch link
According to link :-
This cma mechanism imposes following weaknesses.
Allocation failure
CMA could fail to allocate contiguous memory due to following reasons.
1-1. Direct pinning
Any kernel thread could pin any movable pages for a long time. If a movable page which needs to be migrated for a contiguous memory allocation is already pinned by someone, migration could not be completed. In consequence, contiguous allocation could be fail if the page is not be unpinned longtime.
1-2. Indirect pin
If a movable page have dependency with an object, the object would increase reference count of the movable page to assert it is safe to use the page. If a movable page which is needs to be migrated for a contiguous memory allocation is in the case, the page could not be free to be used by contiguous allocation.
In consequence, contiguous allocation could be failed.
In short, cma doesn't guarantee success and fast latency of contiguous memory allocation. And, the core cause is the fact that cma chosen 2nd-class client(movable pages) were not nice(hard to migrate / discard) enough

What are the reasons for DMA timeouts?

I see a timeout happening when trying to allocate a DMA memory region. While I can remove this bug by using GFP_ATOMIC instead of GFP_KERNEL als gfp flags, so that the DMA allocation becomes non-interruptible, I wonder what the reasons for the occurrence of such a timeout? The requested memory region is not known to the system bus? The bus is saturated?
[ 87.400000] [<c0138eec>] (schedule_bug) from [<c05a0774>] (schedule+0x3c/0x528)
[ 87.410000] [<c05a0774>] (__schedule) from [<c05a0d0c>] (schedule+0xac/0xcc)
[ 87.410000] [<c05a0d0c>] (schedule) from [<c05a3e20>] (schedule_timeout+0x20/0x2b8)
[ 87.420000] [<c05a3e20>] (schedule_timeout) from [<c05a1804>] (wait_for_common+0xf8/0x1a8)
[ 87.430000] [<c05a1804>] (wait_for_common) from [<c012c540>] (flush_work+0x174/0x1ac)
[ 87.450000] [<c012c540>] (flush_work) from [<c01a0648>] (drain_all_pages+0x108/0x130)
[ 87.460000] [<c01a0648>] (drain_all_pages) from [<c01d6d34>] (start_isolate_page_range+0xbc/0x284)
[ 87.470000] [<c01d6d34>] (start_isolate_page_range) from [<c01a3310>] (alloc_contig_range+0xdc/0x330)
[ 87.480000] [<c01a3310>] (alloc_contig_range) from [<c01d7658>] (cma_alloc+0x170/0x308)
[ 87.490000] [<c01d7658>] (cma_alloc) from [<c011142c>] (__alloc_from_contiguous+0x40/0xd8)
[ 87.500000] [<c011142c>] (__alloc_from_contiguous) from [<c0111500>] (cma_allocator_alloc+0x3c/0x44)
[ 87.510000] [<c0111500>] (cma_allocator_alloc) from [<c010f86c>] (__dma_alloc+0x1d4/0x2fc)
[ 87.520000] [<c010f86c>] (__dma_alloc) from [<c010fa0c>] (arm_dma_alloc+0x3c/0x48)
[ 87.530000] [<c010fa0c>] (arm_dma_alloc) from [<c03ae018>] (tsg_ioctl+0x3e4/0x954)
Most likely, the reason is that you are calling sleep capable function (arm_dma_alloc) from atomic context, where sleep is forbidden. Probably you making this allocation under spin_lock.
The spin_lock calls preempt_disable, what means there is no rescheduling possible. That is how spin_lock works. Anyhow, if you will call sleeping primitive inside spin_lock / spin_unlock and reschedule will happen, the kernel scheduler subsystem will warn you, that you have rescheduled in atomic context (this is what you are seeing in this message).

spinlock lockup suspected reasons

What could be reasons for the following message:
BUG: spinlock lockup suspected on CPU#0, sh/11786
lock: kmap_lock+0x0/0x40, .magic: dead4ead, .owner: sh/11787, .owner_cpu: 1
Blockquote
BUG: spinlock lockup suspected on CPU#0, sh/11786
This indicates the CPU0 is lockup, and the thread/Process would be sh (or start by sh, I am not sure). You should have a look at the stack strace info dumped by the kernel. For example:
127|uid=0 gid=1007#nutshell:/var # [ 172.285647] BUG: spinlock lockup on CPU#0, swapper/0, 983482f0
[ 172.291523] [<8003cb44>] (unwind_backtrace+0x0/0xf8) from [<801853e4>] (do_raw_spin_lock+0x100/0x164)
[ 172.300768] [<801853e4>] (do_raw_spin_lock+0x100/0x164) from [<80350508>] (_raw_spin_lock_irqsave+0x54/0x60)
[ 172.310618] [<80350508>] (_raw_spin_lock_irqsave+0x54/0x60) from [<7f3cf4a0>] (mlb_os81092_interrupt+0x18/0x68 [os81092])
[ 172.321636] [<7f3cf4a0>] (mlb_os81092_interrupt+0x18/0x68 [os81092]) from [<800abee0>] (handle_irq_event_percpu+0x50/0x184)
[ 172.332781] [<800abee0>] (handle_irq_event_percpu+0x50/0x184) from [<800ac050>] (handle_irq_event+0x3c/0x5c)
[ 172.342622] [<800ac050>] (handle_irq_event+0x3c/0x5c) from [<800ae00c>] (handle_level_irq+0xac/0xfc)
[ 172.351767] [<800ae00c>] (handle_level_irq+0xac/0xfc) from [<800ab82c>] (generic_handle_irq+0x2c/0x40)
[ 172.361090] [<800ab82c>] (generic_handle_irq+0x2c/0x40) from [<800552e8>] (mx3_gpio_irq_handler+0x78/0x140)
[ 172.370843] [<800552e8>] (mx3_gpio_irq_handler+0x78/0x140) from [<800ab82c>] (generic_handle_irq+0x2c/0x40)
[ 172.380595] [<800ab82c>] (generic_handle_irq+0x2c/0x40) from [<80036904>] (handle_IRQ+0x4c/0xac)
[ 172.389402] [<80036904>] (handle_IRQ+0x4c/0xac) from [<80035ad0>] (__irq_svc+0x50/0xd0)
[ 172.397416] [<80035ad0>] (__irq_svc+0x50/0xd0) from [<80036bb4>] (default_idle+0x28/0x2c)
[ 172.405603] [<80036bb4>] (default_idle+0x28/0x2c) from [<80036e9c>] (cpu_idle+0x9c/0x108)
[ 172.413793] [<80036e9c>] (cpu_idle+0x9c/0x108) from [<800088b4>] (start_kernel+0x294/0x2e4)
[ 172.422181] [<800088b4>] (start_kernel+0x294/0x2e4) from [<10008040>] (0x10008040)
[1]This would tell you the function call relationships. Notice the info:
[ 172.310618] [<80350508>] (_raw_spin_lock_irqsave+0x54/0x60) from [<7f3cf4a0>] (mlb_os81092_interrupt+0x18/0x68 [os81092])
This tells mlb_os81092_interrupt function try to use the spin_lock_irqsave to lock something. So we can just found this spinlock is used to lock what, and try to analyse or and logs to detect which one is holding the lock. Then found the method to avoid it.
[2]Also because the CPU0 is locked, and there can be MP system, you should make sure whether there is the a irq which may use the critical resource, if the handler of irq is assigned to other CPUs(like the CPU1), is's OK, but if CPU0 deals with the handler of irq, this would cause the deadlock if you use the spin_lock not the spin_lock_irqsave, so check it.

CLR IL-significance of square bracket on .locals init

I am trying to generate a dynamic assembly using Reflection & Emit in .NET. I am getting an error, "Common Language Runtime detected an invalid program." I created another program which has the functionality I want using hard-coded types. The functionality I am trying to write will ultimately use dynamic types, but I can use ILDasm to see the IL I need to generate. I am comparing the IL I am generating with the IL which the compiler generates. In the .locals init declaration of one method I see there is an extra item in the compiler-generated code,
compiler-generated:
.locals init ([0] class [System.Core]System.Linq.Expressions.ParameterExpression CS$0$0000,
[1] class [System.Core]System.Linq.Expressions.ParameterExpression[] CS$0$0001)
mine:
.locals init (class [System.Core]System.Linq.Expressions.ParameterExpression V_0,
class [System.Core]System.Linq.Expressions.ParameterExpression[] V_1)
I don't understand the significance of the "[0]" and "[1]" in the compiler-generated code. Can anyone tell me what it means?
As a more general question, I can follow most ILDasm output without too much trouble. But every so often I run across a problematic expression. For instance, in this line from ILDasm
callvirt instance class [EntityFramework]System.Data.Entity.ModelConfiguration.EntityTypeConfiguration`1<!!0> [EntityFramework]System.Data.Entity.DbModelBuilder::Entity<class DynamicEdmxTrial.HardFooAsset>()
the "!!0" probably refers to the generic type of the Entity<>, but I don't know for sure, and I wonder if there is a key to ILDasm output that would explain its more obscure output to me.
The specification is freely available here. It takes a little getting used to, but most details are easily found once you figure out the structure.
!! is listed in II.7.1 Types:
Type ::= | Description | Clause
‘!’ Int32 | Generic parameter in a type definition, | §II.9.1
| accessed by index from 0 |
| ‘!!’ Int32 | Generic parameter in a method | §II.9.2
| definition, accessed by index from 0 |
...
In other words, inside a method that C# would call f<T, U>(), !!0 would be T, and !!1 would be U.
However, the [0] is a good question. The spec does not seem to address it. The .locals directive is described in II.15.4.1.3 The .locals directive, which lists the syntax as
MethodBodyItem ::= ...
| .locals [ init ] ‘(’ LocalsSignature ‘)’
LocalsSignature ::= Local [ ‘,’ Local ]*
Local ::= Type [ Id ]
There is nothing that seems to allow [0] there unless it is part of a Type, and Type does not allow anything starting with [ either. My guess is that this is an undocumented peculiarity specific to Microsoft's implementation, intended to help the human reader see that location 0 is local variable CS$0$0000, for when the generated instructions access local variables by index.
Experimenting with ILAsm shows that this is exactly what it means. Taking a simple C# program:
static class Program {
static void Main() {
int i = 0, j = 1;
}
}
and compiling and then disassembling it (csc test.cs && ildasm /text test.exe >test.il) shows:
....
.locals init (int32 V_0,
int32 V_1)
IL_0000: nop
IL_0001: ldc.i4.0
IL_0002: stloc.0
IL_0003: ldc.i4.1
IL_0004: stloc.1
IL_0005: ret
....
Modifying the .locals to
.locals init ([0] int32 V_0, [0] int32 V_1)
gives a useful warning message:
test.il(41) : warning : Local var slot 0 is in use
And indeed, declaring variables of different types, then reordering them using [2], [1], [0], assembling and immediately disassembling the result, shows that the variables got reordered.

How to trace function call in Erlang ?

I have a function in my_sup.erl like this:
init([ems_media_sup]) ->
{ok, {{simple_one_for_one, ?MAX_RESTART, ?MAX_TIME}, [
{ems_media_sup, {ems_media, start_link, []}, temporary, 2000, worker, [ems_media]}]
}};
But there is no function named start_link/1 in ems_media.erl, I want to know why there is no error when run
supervisor:start_link(?MODULE, [ems_media_sup])
So, How to know what happened next after call init([ems_media_sup])
That's because my_sup is of type simple_one_for_one - so it will only start child processes when explicitly asked to do so through supervisor:start_child/2.
If the supervisor had been of any other type (one_for_one, one_for_all or rest_for_one) it would have attempted to start all children in the child specification at startup, but a simple_one_for_one supervisor is for creating large numbers of children that only vary by their argument list, so in that case the child specification in the init function only plays the role of a template.

Resources