Is it possible to run multiple instances of a same interrupt simultaneously on a multi processor system in linux?
If not possible, why do we need to synchronize between interrupt handlers using spin locks?
Thanks
Venkatesh
On a SMP architecture Advanced Programmable Interrupt Controller(APIC) is used to route the interrupts from peripherals to the CPU's.
the APIC, based on
1. the routing table (where interrupt affinity is set to a particular processor),
2. priority of the interrupt,
3. the load on the CPU's
For example, consider a interrupt is received at IRQ line 32, this goes through APIC,the interrupt is routed to a particular CPU, for now consider CPU0, this interrupt line is masked until the ISR is handled, which means you will not get a interrupt of the same type if ISR execution is in progress
Once ISR is handled, only then the interrupt line is unmasked for future interrupts
Is it possible to run multiple instances of a same interrupt simultaneously on a multi processor system in linux?
The interrupt handlers are generally serialized. Meaning, that only one instance of the handler would be running(on either of the processors). While this is running, if same type of interrupt is again generated, it is processed only after the current one is done, thus serialized. While "this" handler is being executed by one of the core, other core might service handler of a different instance.
Why do we need to synchronize between interrupt handlers using spin locks?
The spinlocks are used even in such cases as the data has to be protected against some other threads(for example bottom halved, user read/write handler functions, etc).
The scenario could be something like this :
my_ISR()
{
lock(&l);
// data is accessed here
unlock(&l);
}
my_other_thread()
{
lock(&l);
// same data is accessed here
unlock(&l);
}
Related
I'm trying to make sure that a unique user process executes as soon as possible after a particular hardware interrupt occurs.
One mechanism I'm aware of for doing this is to write a small kernel module that exports a device while sleeping inside the read handler. The module also registers an irq handler, which does nothing but wake the process. Then from the user's perspective, reads to that device block until the relevant interrupt occurs.
(1) On a modern CPU with a mainline kernel, can you reliably expect sub millisecond latency between the kernel seeing the interrupt and the user process regaining control with this?
(2) Are there any lower latency mechanisms on a mainline kernel?
Apply the PREEMPT_RT patch to the kernel and compile it configuring full preemptability through make menuconfig.
This will allow you to have threaded interrupts (i.e., interrupt handlers executed as kernel threads). Then, you can assign maximum priority (i.e., RT prio > 50) to your specific interrupt handler (check its PID using ps aux) and to your specific process, and a lower priority to anything else.
Imagine that a device function holds a spinlock to control access to the device. While the lock is held, the device issues an interrupt, which causes an interrupt handler to run. The interrupt handler, before accessing the device, must also obtain the lock.
Suppose that the interrupt handler executes in the same processor as the code that took out the lock originally.
Knowing that to hold spinlock disables preemption on the relevant processor, is it possible that the code that holds the spinlock be executed on another processor (because of preemption on this processor)? (We suppose that this is a SMP machine)
Is it possible that the code that holds the spinlock be executed on another processor (because of preemption on this processor)?
No, the code just keeps waiting for the interrupt handler to return.
Just use spin_lock_irq*(), or spin_lock_bh() if you also want to protect against softirqs/tasklets.
For a shared interrupt line,I can have several interrupt handlers. The kernel will sequentially invoke all the handlers for that particular shared line.
As far as I know, each handler, when invoked informs the kernel whether it was the correct handler to be invoked or not.
My questions is how is this determined,is there a way it checks a memory mapped register that tells status of a particular device or is there some other hardware mechanism ? How does the handler know that the corresponding device is indeed the one that issued the interrupt or not ?
Is this information relayed through the interrupt controller that is between the devices and the processor interrupt line ??
The kernel will sequentially invoke all the handlers for that particular shared line.
Exactly. Say Dev1 and Dev2 shares the IRQ10. When an interrupt is generated for IRQ10, all ISRs registered with this line will be invoked one by one.
In our scenario, say Dev2 was the one that generated the interrupt. If Dev1's ISR is registered first, than its ISR (i.e Dev1's ISR) only called first. In that ISR, the interrupt status register will be verified for interrupt. If no interrupt bit is set (which is the case, cause Dev2 raised the interrupt) then we can confirm that interrupt was not generated by Dev1 - so Dev1's ISR should return to the kernel IRQ_NONE - which means:"I did not handled that interrupt", so on the kernel continues to the next ISR (i.e Dev2's ISR), which in turn, will indeed verify that its corresponding device generated the interrupt, thus, this handler should handle it and eventually return IRQ_HANDLED - which means:"I handled this one".
See the return values IRQ_NONE/IRQ_HANDLED for more information.
How does the handler know that the corresponding device issued the interrupt or not ?
By reading the Interrupt status register only.
Is this information relayed through the interrupt controller that is between the devices and the processor interrupt line ??
I'm not sure about this. But the OS will take care of calling ISRs based on the return values from ISR.
Does the CPU disable all interrupts on local CPU before calling the interrupt handler?
Or does it only disable that particular interrupt line, which is being served?
x86 disables all local interrupts (except NMI of course) before jumping to the interrupt vector. Linux normally masks the specific interrupt and re-enables the rest of the interrupts (which aren't masked), unless a specific flags is passed to the interrupt handler registration.
Note that while this means your interrupt handler will not race with itself on the same CPU, it can and will race with itself running on other CPUs in an SMP / SMT system.
Normally (at least in x86), an interrupt disables interrupts.
When an interrupt is received, the hardware does these things:
1. Save all registers in a predetermined place.
2. Set the instruction pointer (AKA program counter) to the interrupt handler's address.
3. Set the register that controls interrupts to a value that disables all (or most) interrupts. This prevents another interrupt from interrupting this one.
An exception is NMI (non maskable interrupt) which can't be disabled.
Yes, that's fine.
I'd like to also add what I think might be relevant.
In many real-world drivers/kernel code, "bottom-half" (bh) handlers are used pretty often- tasklets, softirqs. These bh's run in interrupt context and can run in parallel with their top-half (th) handlers on SMP (esp softirq's).
Of course, recently there's a move (mainly code migrated from the PREEMPT_RT project) towards mainline, that essentially gets rid of the 'bh' mechanism- all interrupt handlers will run with all interrupts disabled. Not only that, handlers are (can be) converted to kernel threads- these are the so-called "threaded" interrupt handlers.
As of today, the choice is still left to the developer- you can use the 'traditional' th/bh style or the threaded style.
Ref and Details:
http://lwn.net/Articles/380931/
http://lwn.net/Articles/302043/
Quoting Intels own, surprisingly well-written "Intel® 64 and IA-32 Architectures Software Developer’s Manual", Volume 1, pages 6-10:
If an interrupt or exception handler is called
through an interrupt gate, the processor clears the interrupt enable (IF) flag in the EFLAGS register to prevent
subsequent interrupts from interfering with the execution of the handler. When a handler is called through a trap
gate, the state of the IF flag is not changed.
So just to be clear - yes, effectively the CPU "disables" all interrupts before calling the interrupt handler. Properly described, the processor simply triggers a flag which makes it ignore all interrupt requests. Except probably non-maskable interrupts and/or its own software exceptions (please someone correct me on this, not verified).
We want ISR to be atomic and no one should be able to preempt the ISR.
Therefore, An ISR disables the local interrupts ( i.e. the interrupt on the current processor) and once the ISR calls ret_from_intr() function ( i.e. we have finished the ISR) , interrupts are again enabled on the current processor.
If an interrupt occurs, it will now be served by the other processor ( in SMP system) and ISR related to that interrupt will start running.
In SMP system , We also need to include the proper synchronization mechanism ( spin lock) in an ISR.
I am actually reading Windows Internals 5th edition and i am enjoying, although isn't a easy book to read and understand.
I am confused about IRQLs and IDT Table.
I read that windows implement custom priorization levels with IRQL and the Plug and Play Manager maps IRQ from devices to IRQL.
Alright, so, IRQLs are used for Software and Hardware interrupts, and for exceptions is used the Exception Dispatch handler.
When one device generates an interrupt, the interrupt controller pass this information to the CPU with the IRQ.
So Windows takes this IRQ and translates to IRQL to schedule when to execute the routine (routine that IDT[IRQ_VALUE] is pointing to?
Is that what is happening?
Yes, on a very high level.
Everything starts with a kernel trap. Kernel trap handler handles interrupts, exceptions, system service calls and virtual memory pager.
When an interrupt happens (line based - using dedicated pin or message based- writing to an address) windows uses IRQL to determine the priority of the interrupt and uses this to see if the interrupt can be served or not during that time. HAL does the job of translating the IRQ to IRQL.
It then uses IRQ to get an index of the IDT to find the appropriate ISR routing to invoke. Note there can be multiple ISR associated for a given IRQ. All of them execute in order.
Each processor has its own IDT so you could potentially have multiple ISR's running at the same time.
Exception dispatch, as I mentioned before, is also handled by the kernel trap but the procedure for it is different. It usually starts by checking for any exception handlers by stack unwinding, then checking for debugger port etc.