You have done it a thousand times: you unplug some USB equipment and any device associated with that USB equipment is removed by the driver. Any program that uses some previously opened file handle will get an error. Somehow most Linux drivers take care of that.
I am currently struggling to implement the same in a simple driver. My driver creates a character device. When the device is opened, I set the private_data member of struct file to the address of some management data that exists once per character device. That management data also includes a mutex that I use to synchronize operations like read, write, and ioctl.
The problem now arises when the USB equipment is unplugged. I must not free the memory where that management data lives. First, any currently running read, write, or ioctl should finish. Any such running call will likely also hold a lock on the mutex and will attempt to unlock it. So the memory where the mutex lives will be accessed.
Any read, write, or ioctl call subsequent to unplugging the equipment should fail, so every such call must read some variable telling whether the USB equipment is still plugged in or not. Again, that variable must live somewhere and the memory where it lives must stay allocated as long as there are open file handles.
Long story short, it seems to me that I must do some sort reference counting: The management data must stay allocated until all file handles have been closed. I could implement that myself, but I have the feeling that I would reinvent the wheel. Such a thing must already exist, I'm not the first to have this problem.
Does Linux internally keep track of the number of open file handles? Can I define a callback that is called when all file handles have been closed? Is that even a viable thing? What is the proper way to remove a character device from the system?
Global variables shall not be avoided, since any number of USB devices can be attached.
As suggested by 0andriy, I have used reference counting to track the number of open file handles. Per character device, I have added a new struct containing a mutex, a kref, the cdev struct, and a pointer to further data. I use kref_get to increase the reference counter when a new file handle is opened. Accordingly, I use kref_put when file handles are released or the USB equipment is unplugged.
Since all I/O operations (read, write, ioctl, etc.) use the mutex, I can safely change the pointer to NULL when the USB equipment is unplugged. The I/O operations then start reporting errors.
The kref reference counter only returns to zero when the USB equipment is unplugged and all file handles are closed. So then I can also free the memory of the new struct.
This seems to work. It was a surprise that the cdev struct is referenced by file handles. It seems to be needed as long as there are open file handles.
But I'm still surprised that I had to go down this route. It seems redundant to implement that in every driver.
UPDATE: As it turns out, doing reference counting yourself is dangerous. In particular counting the number of open file handles is wrong. There is a whole reference-count-based garbage collection system in Linux and you need to use it.
For example, Linux calls cdev_get when a process opens a character device and cdev_put when the process exists and the file handle is still open. Unfortunately, the call to cdev_put would occur after the file handle's release function. As we would free the memory after the last file handle has been released, we would end up freeing the underlying memory of the cdev and the mutex before cdev_put is called.
The solution is to assign a parent to the cdev via cdev_set_parent. The parent kobject will have its reference counter increased whenever cdev_get is called and decreased after cdev_put is called. The kobject then can have its own release function which frees any memory required by the cdev. Basically I replaced the kref in my struct with a kobject.
Lesson learned: don't do reference counting on your own. Use the existing hierarchy of kobjects, which is the kernel's way of doing garbage collection.
UPDATE2: It turns out, we reinvented the wheel again. The device struct includes a release hook, which is called when the internal kobject reached a reference count of zero. This is the mechanism typically used in the kernel to release resources associated with an device - including the device struct itself.
I was looking for a release hook in the cdev struct. There was none. But it turns out I should have looked one step up in the hierarchy for such a hook.
Long story short: Use cdev_device_add in combination with the device release hook. cdev_device_add will internally call cdev_set_parent, so the device becomes the parent of the cdev. Those are the mechanism other kernel drivers (e.g. evdev) use to release their resources.
Related
This is all in Linux 4.14.73. I'd upgrade if I could but I cant.
I'm trying to trigger an LED flash in a standard LED class instance from another kernel space driver. I know all about the "bad form" of not accessing files from Kernel Space so I figure there must be some way already defined way for accessing Sysfs attributes from Kernel Space.
The LED is defined here:
/sys/class/leds/fpga_led0
Its trigger is set to [oneshot] so it has a device attribute called "shot" exposed. To get a single LED flash all I need to do from the command line is this:
echo 1 > /sys/class/leds/fpga_led0/shot
I can easily write a User Space program to open the "shot" attribute and write a "1" string to it. There are various published methods of forcing file operations into a kernel driver. Most of them are fairly limited. I've yet to see one that exposes file seek operations which are key to repeatedly writing to an attribute without wasting time opening and closing the file. To be clear, this is not setting values at boot time. In this case I have one driver that needs to send a value to another driver's Sysfs entry at a specific moment in its own operation. Is there a standard, accepted way of sending a value from one running kernel driver to the Sysfs attribute of another kernel driver?
I have managed to create a virtual IOPCIDevice which attaches to IOResources and basically does nothing. I'm able to get existing drivers to register and match to it.
However when it comes to IO handling, I have some trouble. IO access by functions (e.g. configRead, ioRead, configWrite, ioWrite) that are described in IOPCIDevice class can be handled by my own code. But drivers that use memory mapping and IODMACommand are the problem.
There seems to be two things that I need to manage: IODeviceMemory(described in the IOPCIDevice) and DMA transfer.
How could I create a IODeviceMemory that ultimately points to memory/RAM, so that when driver tries to communicate to PCI device, it ultimately does nothing or just moves the data to RAM, so my userspace client can handle this data and act as an emulated PCI device?
And then could DMA commands be directed also to my userspace client without interfering to existing drivers' source code that use IODMACommand.
Thanks!
Trapping memory accesses
So in theory, to achieve what you want, you would need to allocate a memory region, set its protection bits to read-only (or possibly neither read nor write if a read in the device you're simulating has side effects), and then trap any writes into your own handler function where you'd then simulate device register writes.
As far as I'm aware, you can do this sort of thing in macOS userspace, using Mach exception handling. You'd need to set things up that page protection fault exceptions from the process you're controlling get sent to a Mach port you control. In that port's message handler, you'd:
check where the access was going to
if it's the device memory, you'd suspend all the threads of the process
switch the thread where the write is coming from to single-step, temporarily allow writes to the memory region
resume the writer thread
trap the single-step message. Your "device memory" now contains the written value.
Perform your "device's" side effects.
Turn off single-step in the writer thread.
Resume all threads.
As I said, I believe this can be done in user space processes. It's not easy, and you can cobble together the Mach calls you need to use from various obscure examples across the web. I got something similar working once, but can't seem to find that code anymore, sorry.
… in the kernel
Now, the other problem is you're trying to do this in the kernel. I'm not aware of any public KPIs that let you do anything like what I've described above. You could start looking for hacks in the following places:
You can quite easily make IOMemoryDescriptors backed by system memory. Don't worry about the IODeviceMemory terminology: these are just IOMemoryDescriptor objects; the IODeviceMemory class is a lie. Trapping accesses is another matter entirely. In principle, you can find out what virtual memory mappings of a particular MD exist using the "reference" flag to the createMappingInTask() function, and then call the redirect() method on the returned IOMemoryMap with a NULL backing memory argument. Unfortunately, this will merely suspend any thread attempting to access the mapping. You don't get a callback when this happens.
You could dig into the guts of the Mach VM memory subsystem, which mostly lives in the osfmk/vm/ directory of the xnu source. Perhaps there's a way to set custom fault handlers for a VM region there. You're probably going to have to get dirty with private kernel APIs though.
Why?
Finally, why are you trying to do this? Take a step back: What is it you're ultimately trying to do with this? It doesn't seem like simulating a PCI device in this way is an end to itself, so is this really the only way to do what greater goal you're ultimately trying to achieve? See: XY problem
I am learning linux network driver recently, and I wonder that if I have many network cards in same type on my board, how does the kernel drive them? Does the kernel need to load the same driver many times? I think it's not possible, insmod won't do that, so how can I make all same kind cards work at same time?
regards
The state of every card (I/O addresses, IRQs, ...) is stored into a driver-specific structure that is passed (directly or indirectly) to every entry point of the driver which can this way differenciate the cards. That way the very same code can control different cards (which means that yes, the kernel only keeps one instance of a driver's module no matter the number of devices it controls).
For instance, have a look at drivers/video/backlight/platform_lcd.c, which is a very simple LCD power driver. It contains a structure called platform_lcd that is private to this file and stores the state of the LCD (whether it is powered, and whether it is suspended). One instance of this structure is allocated in the probe function of the driver through kzalloc - that is, one per LCD device - and stored into the platform device representing the LCD using platform_set_drvdata. The instance that has been allocated for this device is then fetched back at the beginning of all other driver functions so that it knows which instance it is working on:
struct platform_lcd *plcd = to_our_lcd(lcd);
to_our_lcd expands to lcd_get_data which itself expands to dev_get_drvdata (a counterpart of platform_set_drvdata) if you look at include/linux/lcd.h. The function can then know the state of the device is has been invoked for.
This is a very simple example, and the platform_lcd driver does not directly control any device (this is deferred to a function pointer in the platform data), but add hardware-specific parameters (IRQ, I/O base, etc.) and you get how 99% of the drivers in Linux work.
The driver code is only loaded once, but it allocates a separate context structure for each card. Typically you will see a struct pci_driver with a .probe function pointer. The probe function is called once for each card by the PCI support code, and it calls alloc_etherdev to allocate a network interface with space for whatever private context it needs.
I am interested in developing kernel module that binds two block devices into a new block device in such manner that first block device contains data at mount time, and the other is considered empty. Every write is being made to second partition, so on next mount the base filesystem remains unchanged. I know of solutions like UnionFS, but those are filesystem-based, while i want to develop it a layer lower, block-based.
Can anyone tell me how could i open ad read/write block device from kernel module? Possibly without using userspace program for reading/writing merged block devices. I found similar topic here, but the answer was rather unsatysfying because filp_* functions are rather for reading small config files, not for (large) block device I/O.
Since interface for creating block devices is standarized i was thinking of direct (or almost direct) acces to functions implementing source devices, as i will be requested to export similar functions anyway. If i could do that i would simply create some proxy-functions calling appropriate functions on source devices. Can i somehow obtain pointer to a gendisk structure that belongs to different driver?
This serves only my own purposes (satisfying quriosity being main of them) so i am not worried about messing my kernel up seriously.
Or does somebody know if module like that already exists?
The source code in the device mapper driver will suit your needs. Look at the code in the Linux source in Linux/drivers/md/dm-*.
You don't need to access the other device's gendisk structure, but rather its request queue. You can prepare I/O requests and push it down the other device's queue, and it will do the rest itself.
I have implemented a simple block device that opens another block device. Take a look in my post describing it:
stackbd: Stacking a block device over another block device
Here are some examples of functions that you need for accessing another device's gendisk.
The way to open another block device using its path ("/dev/"):
struct block_device *bdev_raw = lookup_bdev(dev_path);
printk("Opened %s\n", dev_path);
if (IS_ERR(bdev_raw))
{
printk("stackbd: error opening raw device <%lu>\n", PTR_ERR(bdev_raw));
return NULL;
}
if (!bdget(bdev_raw->bd_dev))
{
printk("stackbd: error bdget()\n");
return NULL;
}
if (blkdev_get(bdev_raw, STACKBD_BDEV_MODE, &stackbd))
{
printk("stackbd: error blkdev_get()\n");
bdput(bdev_raw);
return NULL;
}
The simplest example of passing an I/O request from one device to another is by remapping it without modifying it. Notice in the following code that the bi_bdev entry is modified with a different device. One can also modify the block address (*bi_sector) and the data itself.
static void stackbd_io_fn(struct bio *bio)
{
bio->bi_bdev = stackbd.bdev_raw;
trace_block_bio_remap(bdev_get_queue(stackbd.bdev_raw), bio,
bio->bi_bdev->bd_dev, bio->bi_sector);
/* No need to call bio_endio() */
generic_make_request(bio);
}
Consider examining the code for the dm / md block devices in drivers/md - these existing drivers create a block device that stores data on other block devices.
In fact, you could probably implement your idea as another "RAID personality" in md, and thereby make use of the existing userspace tools for setting up the devices.
You know, if you're a GPL'd kernel module you can just call open(), read(), write(), etc. from kernel mode right?
Of course this way has certain caveats including requiring forking from kernel mode to create a space for your handle to live.
I've written a Linux device driver for a PCI device. This device performs DMA operations. An issue arise when the program crashes when a DMA operation is running.
Indeed, when crashing, the device_remove() function is called by the system (as if close() were called). This function does the cleanup on the memory regions used by the PCI device, frees the allocated memory correctly. I mean it works correctly under normal circumstances.
But if a DMA is running, when it will actually terminate, it won't be able to perform the DMA cleanup because it does not have access anymore to the device data that have been freed. A simple solution would be to wait in the close() function. (This is my understanding, but maybe the last part of the DMA function is never executed?)
Is it a good idea to wail for the DMA to actually terminate in the device_remove() (aka close()) function of a device driver? Are there other means to deal with this issue?
Yes, wait should work but :
Unless you are trying to test surprise removal behavior of your PCI device, I think a call to remove() should fail when you have DMA going on to/from the device. Also, I don't think close() can be treated the same way as remove(). The latter is going to completely remove all device related data structures from memory (For example: see one of the network device drivers). So, in other words, what I am trying to say is : wait() on close() but fail() on remove()
Also depending on your situation, you might also want to take a look at reference counting for freeing up of the device related resources.