I am trying to debug a custom Linux serial driver that is having some issues missing some receive data. It has one interrupt for 4 serial ports, and baud rate is 115200. Firstly I would like to see how to measure how long the interrupt handler takes. I have used perf, but things are just in percent and not seconds. Secondly does anyone see any issues with the below code that can be improved to speed things up?
void serial_interrupt(int irq, void *dev_id)
{
...
// Need to loop through each port to see which port caused the interrupt.
list_for_each(lpNode, &serial_ports)
{
struct serial_port_module *ser_dev = list_entry(lpNode, struct serial_port_module, port_list);
lnIsr = ioread8(ser_dev->membase + ser_dev->chan_num * PORT_OFFSET + SERIAL_ISR);
if (lnIsr & IPM512_RX_INT)
{
while (serialdata_is_data_available(ser_dev)) // equals a ioread8()
{
lcIn = ioread8(ser_dev->membase + ser_dev->chan_num * PORT_OFFSET + SERIAL_RBR);
kfifo_in(&ser_dev->rx_fifo, &lcIn, sizeof(lcIn));
// Notify if anyone is doing a blocking read.
wake_up_interruptible(&ser_dev->read_queue);
}
}
}
}
Use the ftrace API to try to track down your latency issues. It's woth the time to get to know: https://www.kernel.org/doc/Documentation/trace/ftrace.txt
If this is too heavy-weight, what about adding some simple instrumentation yourself? getnstimeofday(struct timespec *ts) is relatively lightweight... with a little code you could output in a sysfs debug file the worst case execution times, some stats on latency of call to this function, worst-case number of bytes available per interrupt... if this number gets near your hardware FIFO size, you're in trouble.
One optimization would be to read the data in batches into a buffer, as long as data is available, then input the entire buffer, then wake up any readers.
while(data_available(dev))
{
buf[cnt++] = ioread8();
}
kfifo_in(fifo, buf, cnt);
wake_up_interruptible();
But execution time of code this simple is not likely to be an issue. You're probably suffering from missed interrupts or unexpected latency of the interrupt handling.
Related
I am trying to implement a cache based covert channel in C but noticed something weird. The physical address between the sender and the receiver is shared by using the mmap() call that maps to the same file with the MAP_SHARED option. Below is the code for the sender process which flushes an address from the cache to transmit a 1 and loads an address into the cache to transmit a 0. It also measures the latency of a load in both cases:
// computes latency of a load operation
static inline CYCLES load_latency(volatile void* p) {
CYCLES t1 = rdtscp();
load = *((int *)p);
CYCLES t2 = rdtscp();
return (t2-t1);
}
void send_bit(int one, void *addr) {
if(one) {
clflush((void *)addr);
load__latency = load_latency((void *)addr);
printf("load latency = %d.\n", load__latency);
clflush((void *)addr);
}
else {
x = *((int *)addr);
load__latency = load_latency((void *)addr);
printf("load latency = %d.\n", load__latency);
}
}
int main(int argc, char **argv) {
if(argc == 2)
{
bit = atoi(argv[1]);
}
// transmit bit
init_address(DEFAULT_FILE_NAME);
send_bit(bit, address);
return 0;
}
The load operation takes around 0 - 1000 cycles (during a cache-hit and a cache-miss) when issued by the same process.
The receiver program loads the same shared physical address and measures the latency during a cache-hit or a cache-miss, the code for which has been shown below:
int main(int argc, char **argv) {
init_address(DEFAULT_FILE_NAME);
rdtscp();
load__latency = load_latency((void *)address);
printf("load latency = %d\n", load__latency);
return 0;
}
(I ran the receiver manually after the sender process terminated)
However, the latency observed in this scenario is very much different as compared to the first case. The load operation takes around 5000-1000 cycles.
Both the processes have been pinned to the same core-id by using the taskset command. So if I'm not wrong, during a cache-hit, both processes will experience a load latency of the L1-cache on a cache-hit and DRAM on a cache-miss. Yet, these two processes experience a very different latency. What could be the reason for this observation, and how can I have both the processes experience the same amount of latency?
The initial access to an mmaped region will page-fault (lazy mapping/allocation by the kernel), unless you use mmap(MAP_POPULATE), or mlock, or touch some other cache line of the page first.
You're probably timing a page fault if you only do one time measurement per mmap, or per run of a whole program.
(Also you don't seem to be doing anything to warm up the CPU frequency, so once core cycle could be many reference cycles. Some of the time for an L3 miss is fixed in terms of memory clock cycles, but another part of it scales with core/uncore clock.)
Also note that unless you run the 2nd process immediately (e.g. from the same shell command), the OS will get a chance to put that core into a deep sleep. On Intel CPUs at least, that empties L1d and L2 so it can power them down in the deeper C states. Probably also the TLBs.
It's also strange that you you cast away volatile in load = *((int *)p);
Assigning the load result to a global(?) variable inside the timed region is also pretty questionable; that could also soft page fault. If so, RDTSCP will have to wait for it, because the store can't retire.
(But on TLB hit for the store, it doesn't have to wait for it commit to cache, since there's no mfence before rdtscp. A store can retire while the store is still in the store buffer. In fact it must retire before the store buffer entry is known to be non-speculative so it can commit.)
I am trying to make asynchronous kernel calls to my GPGPU using CUDAfy .NET.
When I pass values to the kernel and copy them back to the host, I do not always get the value I expect.
I have a structure Foo with a byte Bar:
[Cudafy]
public struct Foo {
public byte Bar;
}
And I have a kernel I want to call:
[Cudafy]
public static void simulation(GThread thread, Foo[] f)
{
f[0].Bar = 3;
thread.SyncThreads();
}
I have a single thread with streamID = 1 (I tried using multiple threads, and noticed the issue. Reducing to a single thread didn't seem to fix the issue though).
//allocate
streamID = 1;
count = 1;
gpu.CreateStream(streamID);
Foo[] sF = new Foo[count];
IntPtr hF = gpu.HostAllocate<Foo>(count);
Foo[] dF = gpu.Allocate<Foo>(sF);
while (true)
{
//set value
sF[0].Bar = 1;
byte begin = sF[0].Bar;
//host -> pinned
GPGPU.CopyOnHost<Foo>(sF, 0, hF, 0, count);
sF[0].Bar = 2;
lock (gpu)
{
//pinned -> device
gpu.CopyToDeviceAsync<Foo>(hF, 0, dF, 0, count, streamID);
//run
gpu.Launch().simulation(dF);
//device -> pinned
gpu.CopyFromDeviceAsync<Foo>(dF, 0, hF, 0, count, streamID);
}
//WAIT
gpu.SynchronizeStream(streamID);
//pinned -> host
GPGPU.CopyOnHost<Foo>(hF, 0, sF, 0, count);
byte end = sF[0].Bar;
}
//de-allocate
gpu.Free(dF);
gpu.HostFree(hF);
gpu.DestroyStream(streamID);
First I create a stream on the GPU.
I am creating a regular structure Foo array of size 1 (sF) and setting it's Bar value to 1. Then I create pinned memory on the host (hF) for Foo as well. I also create memory on the device for Foo (dF).
I initialize the structure's Bar value to 1 then I copy it to the pinned memory (As a check, I set the value to 2 for the structure after copying to pinned, you'll see why later). Then I use a lock to ensure I have full access to the GPU and I queue a copy to dF, a run for the kernel, and a copy from dF. At this point I don't know when this will all actually run on the GPU... so I can call SynchronizeStream to wait on the host until the device is done.
When it's done, I can copy the pinned memory (hF) to the shared memory (sF). When I get the value, it's usually a 3 (which was set on the device) or a 1 (which means either the value wasn't set in the kernel, or the new value wasn't copied to the pinned memory). I do know that the pinned memory is copied to the structure because the structure never has the value of 2.
Over many runs, a small percentage is runs results in something other than begin=1 and end=3. It would always be begin=1, end=1 and it happens about 5-10% of the time.
I have no idea why this happens. I know it generally highlights a race condition, but by calling the sync calls, I would expect the async calls to work in a predictable fashion.
Why would I be encountering this kind of issue with this code?
Thank you so much!
-Phil
I just figured out the issue that was occurring. While the launch was being done asynchronously... I didn't include the stream for the launch.
Changing my launch to be:
gpu.Launch(gridsize,blocksize,streamID).simulation(dF);
resolved the problem. It seems that the launches were occurring on stream 0 and the stream 1 and 2 were being synced. So sometimes the data gets set, sometimes it doesn't. A race condition.
I have created a kernel thread using kthread_run in a kernel module.
The thread is very simple, just like bellow.
static int my_thread_func(void * data)
{
int a;
DBG_PRINT("policy:%lu; prio:%d", current->policy, current->prio);
while (!kthread_should_stop())
{
a++;
}
}
However, after I loaded the module, the system did not response any more.
So I wonder what's the schedule policy and priority of this kernel thread.
Then I try to print out the schedule policy and priority of this kernel thread,
and got bellow output.
policy:0; prio:120
policy:0 means SCHED_NORMAL;
prio:120 this is also not high.
While the thread does not have a SCHED_FIFO or SCHED_RR schedule policy, Why can it hog up the cpu?
And I also found that if I insert some sleep code in the loop body of the thread, the system could remain responsive.
And I also found when I run a userspace program implemented as bellow, the system remained responsive, too.
int main(int argc, char *argv[])
{
int a;
while (1) a++;
return 0;
}
So who can tell me, why the kernel thread could hog up the cpu.
When you say the priority is 120, are you observing the priority of kthreadd, or the actual kernel thread that was created for you?
Please see http://lxr.free-electrons.com/source/kernel/kthread.c#L310 for the function that is used to create new kernel threads.
Excerpt:
struct task_struct *kthread_create_on_node(int (*threadfn)(void *data),
void *data, int node,
const char namefmt[],
...)
{
...
if (!IS_ERR(task)) {
static const struct sched_param param = { .sched_priority = 0 };
...
/*
* root may have changed our (kthreadd's) priority or CPU mask.
* The kernel thread should not inherit these properties.
*/
sched_setscheduler_nocheck(task, SCHED_NORMAL, ¶m);
set_cpus_allowed_ptr(task, cpu_all_mask);
}
...
}
It appears that the sched_priority is set to 0.
Now, please look at http://lxr.free-electrons.com/source/include/linux/sched/prio.h#L9
Excerpt:
/*
* Priority of a process goes from 0..MAX_PRIO-1, valid RT
* priority is 0..MAX_RT_PRIO-1, and SCHED_NORMAL/SCHED_BATCH
* tasks are in the range MAX_RT_PRIO..MAX_PRIO-1. Priority
* values are inverted: lower p->prio value means higher priority.
*
* The MAX_USER_RT_PRIO value allows the actual maximum
* RT priority to be separate from the value exported to
* user-space. This allows kernel threads to set their
* priority to a value higher than any user task. Note:
* MAX_RT_PRIO must not be smaller than MAX_USER_RT_PRIO.
*/
Please note the second paragraph: This allows kernel threads to set their priority to a value higher than any user task.
Summary:
It appears that newly created kernel threads are set to sched_priority of 0 even with SCHED_NORMAL. Priority of 0 is not a normal priority for SCHED_NORMAL, and thus the kernel thread will take priority over any other thread (that isn't using an RT policy---and I don't think any processes in the kernel use RT by default).
Addendum:
Note: I am not 100% sure if this is the reason. But, if you look at the comments in the kernel for kernel threads, they all seem to imply that a kernel thread keeps running UNLESS:
The kernel thread itself yields or calls do_exit.
Someone else calls kthread_should_stop().
Which to me, sounds like the kernel thread runs as long as it wants, until it decides to stop, or someone else explicitly tells it to stop.
Hi I have a problem in sending message in project, I am using pic16f877a and sim300. the main function runs repeatedly. Some characters are missed in the sent sms.
my program is like this...
void main()//main function
{
Serial_init(); // initialization of serial communication
Send_SMS();
}
void Serial_init()
{
TRISC=0XC0;
TXSTA=0x24;
SPBRG=129; // set baud rate 9600 Hz for 20MHz fosc
RCSTA=0x90;
TXIF=1;
}
void Send_SMS(void)
{
USART_puts("AT\0");
putch1(0x0D);
Delay_ms4M(200);
USART_puts("AT+CMGF=1\0"); // switch into text mode
putch1(0x0D);// ascii of Carriage Return
Delay_ms4M(200);
USART_puts("AT+CMGS=\"9741153218\"\0"); // send sms to the number
putch1(0x0D);
Delay_ms4M(200);
USART_puts("Hi this is working LOL\0"); // SMS text
putch1(0x0A); // new line
Delay_ms4M(200);
putch1(0x0D);
Delay_ms4M(100);
putch1(0x1A); // ascii of 'substitute' i.e end of file
}
void USART_puts(const unsigned char *string)
{
while(*string)
putch1(*string++);
}
void putch1(unsigned char data)
{
while(TXIF==0);
TXREG=data;
}
Please help
additional details: all other programs run properly, but if I call send_sms function, "main" runs repeatedly and several messages are sent with missed characters.
IMHO :
Your chip is resetting. This is the highest probable cause.
Either it is faulty or you have set Watchdog Timer to on somewhere.
For missing characters :
a) Chip resets in the midst of a data transfer.
b) Roule of thumb for usart:
Stop stuffing bytes to usart. Send each byte with a small leading delay like 10-20 microseconds.
The communication is asynchronous, which means the receiver has to synchronize at the beginning of each communication unit which is a byte. To do that receiver brutely uses resources to detect start bit, the length (in time) of it etc. So if you try to send a byte train, you will stall the receiver.
Have you tried the code with another 16F877a ? (to check chip failure)...
I'm writing a block device driver for a hot-pluggable PCI memory device on 2.6.43.2-6.fc15 (so LDD3 is out of date with respect to a lot of functions) and I'm having trouble getting the block device de-registration to go smoothly. When the device is removed, I go to tear down the gendisk and request_queue, but it hangs on blk_cleanup_queue(). Presumably there's some queue-related process I have neglected to carry out before that, but I can't see any major consistent differences with other block drivers from that kernel tree that i am using for reference (memstick, cciss, etc). What are the steps I should carry out before going to tidy up the queue and gendisk?
I am implementing .open, .release, .ioctl in the block_ops as well as a mydev_request(struct request_queue *q) attached with blk_init_queue(mydev_request, &mydev->lock), but I'm not sure exactly how to tidy the queue either when requests occur or when de-registering the block device.
This is caused by not ending the requests that you fetch off the queue. To fix it, end the request as follows:
while ((req = blk_fetch_request(q)) != NULL )
{
res = mydev_submit_request_sg(mydev, req);
if (res)
__blk_end_request_all(req, res);
else
__blk_end_request_cur (req, res);
}