Suppose that a Linux driver code acquires a spinlock, inside the critical section a function call force the process running on top of the driver to sleep. Knowing that to hold spinlock disables preemption on the relevant processor, is it possible for the process to wake up, and consequently to permit the driver code to release the spinlock ?
No, it is not allowed to sleep while holding a spinlock. Code that does this is buggy.
The only way the process could be woken is if code running on another core did something to wake it up (which means that yes, it will certainly deadlock if there is only one core).
while spinlock other process can't do anything to wake it up, you could try to use semaphores for this type of switching
Related
We develop a user-space process running on Linux 3.4.11 in an embedded MIPS system. The process creates multiple (>10) threads using pthreads. The process has a SIGSEGV signal handler which, among other things, generates a log message which goes to our log file. As part of this flow, it acquires a semaphore (bad, I know...).
During our testing the process appeared to hang. We're currently unable to build gdb for the target platform, so I wrote a CLI tool that uses ptrace to extract the register values and USER data using PTRACE_PEEKUSR.
What surprised me to see is that all of our threads were inside our crash handler, trying to acquire the semaphore. This (obviously?) indicates a deadlock on the semaphore, which means that a thread died while holding it. When I dug up the stack, it seemed that almost all of the threads (except one) were in a blocking call (recv, poll, sleep) when the signal handler started running. Manual stack reconstruction on MIPS is a pain so we have not fully done it yet. One thread appeared to be in the middle of a malloc call, which to me indicates that it crashed due to a heap corruption.
A couple of things are still unclear:
1) Assuming one thread crashed in malloc, why would all other threads be running the SIGSEGV handler? As I understand it, a SIGSEGV signal is delivered to the faulting thread, no? Does it mean that each and every one of our threads crashed?
2) Looking at the sigcontext struct for MIPS, it seems it does not contain the memory address which was accessed (badaddr). Is there another place that has it? I couldn't find it anywhere, but it seemed odd to me that it would not be available.
And of course, if anyone can suggest ways to continue the analysis, it would be appreciated!
Yes, it is likely that all of your threads crashed in turn, assuming that you have captured the thread state correctly.
siginfo_t has a si_addr member, which should give you the address of the fault. Whether your kernel fills that in is a different matter.
In-process crash handlers will always be unreliable. You should use an out-of-process handler, and set kernel.core_pattern to invoke it. In current kernels, it is not necessary to write the core file to disk; you can either read the core file from standard input, or just map the process memory of the zombie process (which is still available when the kernel invokes the crash handler).
I have ARM board at remote location. Some time I had a kernel panic error in it. At this same time there is no option to hardware restart. bus no one is available at this place to restart it.
I want to restart my board automatically after kernel panic error. so what to do in kernel.
If your hardware contains watchdog timer, then compile the kernel with watchdog support and configure it. I suggest to follow this blog http://www.jann.cc/2013/02/02/linux_watchdog.html
Caution :: I never tried this. If the problem is solved, request you to update here.
You can modify the panic() function kernel/panic.c to call the kernel_restart(*cmd) at the point you want it to restart (like probably after printing the required debug information).
I am assuming you are bringing up a board, so Please note that you need to supply the ops for the associated functions in machine_restart() - (called by kernel_restart) in accordance to the MACH . If you are just using the board as is , then i guess rebuilding the kernel with kernel_restart(*cmd) should do.
The panic() is usually due to events that the kernel can not recover from. If you do not have a watchdog, you need to look at your hardware to see if a GPIO, etc is connected to the RESET line. If so, you can toggle this pin to reboot the CPU. Trying to alter panic() may just make things worse, depending on the root cause and the type of features you use.
You may hook arm_pm_restart with your custom restart functionality. You can test it with the shell command reboot, if present. panic() should call the same routine. With current ARM Linux versions
You may wish to turn off the MMU and block interrupts in this routine. It will make it more resilient when called from panic(). As you are going to reset, you can copy the routine to any physical address you like.
The watchdog maybe better; it may catch cases where even panic() may not be called. You may have a watchdog and not realize it. Many Cortex-A CPUs, have one built in. It is fairly rare for hardware not to have a watchdog.
However, if you don't have the watchdog, you can use the GPIO mechanism above; hardware should usually provide someway for software to restart the device (and peripherals). The panic() maybe due to some mis-behaving device tromping memory, latched up DRAM/Flash, etc. Toggling a RESET line maybe better than a watchdog in this case; if the RESET is also connected to other hardware, besides the CPU.
Related: How to debug kernel freeze, How to change watchdog timer
AFAIK, a simple way to restart the board after kernel panic is to pass a kernel parameter (from the bootloader usually)
panic=1
The board will then auto-reboot '1' second(s) after a panic.
Search the Documentation for more.
Some examples from the documentation:
...
panic= [KNL] Kernel behaviour on panic: delay <timeout>
timeout > 0: seconds before rebooting
timeout = 0: wait forever
timeout < 0: reboot immediately
Format: <timeout>
...
oops=panic Always panic on oopses. Default is to just kill the
process, but there is a small probability of
deadlocking the machine.
This will also cause panics on machine check exceptions.
Useful together with panic=30 to trigger a reboot.
...
As suggested in previous comments watchdog timer is your friend here. If your hardware contains watchdog timer, Enable it in kernel option and configure it.
Other alternative is use Phidget. If you usb connection available at remote location. Phidget controller/software is used to control your board using USB. Check for board support.
I would like to know what is going on during sleep and wakeup process on OSx Kernel.
Does a Kernel extension receive a new address space and start all over again its initialization process or the kernel simply puts the extension back in the same address space?
Does internal kernel extensions (IOKit drivers for example) also behave the same? Perhaps they are loaded into a different location in the memory?
Basically the question is: will my driver, which obtained an interface to a IOService, will be able to use its address after sleep without a problem.
On sleep, memory is "frozen", and on resume, it's restored to its original state. So unless you actively participate in power management, your kext won't notice anything has changed. If you're dealing directly with hardware, you will HAVE to care about power management, though, as your device will have power-cycled and will need to be reinitialised.
I have a character device driver which is causing a system deadlock on a multicore system. The write call has a critical section protected by a spin lock (spin_lock_irqsave). The ISR must obtain this lock to finish its task as well. If the ISR is called on one core while the write is executing the critical section on another, a panic occurs due to a watchdog timer detecting a hard lockup on the core for the ISR. The write process never returns to finish executing. Shouldn't the write process continue to execute on its core, release the lock which will allow the other core in its ISR to then run?
The critical section requires about 5us to complete. The hard lock occurs after 5 seconds.
I assume I'm doing something wrong but do not know what.
Appreciate any help!
Turns out the critical section was calling wait_for_completion_timeout. Even though the timeout was zero, it still slept and didn't wake up to release the spin lock if the interrupt occurred in the blocking section. Using try_wait_for_completion in this case resolved the issue.
I would have posted source but it spans many modules and has architecture abstractions for portability between operating systems. Would have been a mess.
I have intensive processing that I need to perform in a device driver, at DISPATCH_LEVEL or lower IRQL.
How do I create a kernel-thread?
What IRQL does it run at? Can I control this?
How is it scheduled? Because I am thinking from a user-mode perspective here, what priority does it run at?
What kernel functions can I use to provide locking / synchronization?
you can create system thread with this As you can see one of its parameters is a start routine which can hold custom code - in it you can use KeRaiseIrql and KeLowerIrql. By default threads will run in PASSIVE_LEVEL. "Locks, Deadlocks, and Synchronization" is a very helpful paper regarding synchronization in kernel on windows and everyone who has to do some tinkering with the windows kernel should read or at least skim it