Is it possible to modify a v4l2_buffer allocated through MMAP in userspace such that the physical address of the requested buffer points towards a desired physical memory?
The goal is that we have a sensor writing data in a circular buffer within a reserved memory range and we would like to avoid copying the data (as they are made available by the sensor which triggers an IRQ as soon as the buffer is filled).
Thank you for any help you could provide!
Related
I would first like to give a brief description of the scenario that I am working on.
What I am trying to accomplish is to load image data from my user space application and transfer it over PCIe to a custom acceleration engine located inside a FPGA board.
The specifications of my host machine are:
Intel Xeon Processor with 16G ram.
64 Bit Debian Linux with kernel version 4.18.
The FPGA is a Virtex 7 KC705 development board.
The FPGA uses a PCIe controller (bridge) for the communication between the PCIe infrastructure and the AXI interface of the FPGA.
In addition, the FPGA is equiped with a DMA engine which is supposed to read data through the PCIe controller from the kernel memory and forward them to the accelerator.
Since in future implementations I would like to make multiple kernel allocations up to 256M, I have configured my kernel to support CMA and DMA Contiguous Allocator.
According to dmesg I can verify that my system reserves at startup the CMA area.
Regarding the acceleration procedure:
The driver initially allocates 4M kernel memory by using the dma_alloc_coherent() with GFP_KERNEL flag. This allocation is inside the range of the CMA.
Then from my user space application I call mmap with READ_PROT/WRITE_PROT and MAP_SHARED/MAP_LOCKED flags to map the previously allocated CMA memory and load the image data in there.
Once the image data is loaded I forward the dma_addr_t physical address of the CMA allocated memory and I start the DMA to transfer the data to the accelerator. When the acceleration is completed the DMA is supposed to write the processed data back to the same CMA kernel allocated memory.
On completion the user space application reads the processed data from the CMA memory and saves it to a .bmp file. When I check the "processed" image it is the same as the original one. I suppose that the processed data were never written to the CMA memory.
Is there some kind of memory protection that does not allow writing to the CMA memory when using GFP_KERNEL flag?
An interesting fact is that when I allocate kernel memory with dma_alloc_coherent but with either GFP_ATOMIC or GFP_DMA the processed data are written correctly to the kernel memory but unfortunately the allocated memory does not belong to the range of the CMA area.
What is wrong in my implementation?
Please let me know if you need more information!
In order to use mmap() I have adopted the debugfs file operations method.
Initially, I open a debugfs file as follows:
shared_image_data_file = open("/sys/kernel/debug/shared_image_data_mmap_value", O_RDWR);
The shared_image_data_mmap_value is my debugfs file which is created in my kernel driver and the shared_image_data_file is just an integer.
Then, I call mmap() from userspace as follows:
kernel_address = (unsigned int *)mmap(0, (4 * MBYTE), PROT_READ | PROT_WRITE, MAP_SHARED | MAP_LOCKED, shared_image_data_file, 0);
When I call the mmap() function in user space the mmap file operation of my debugfs file executes the following function in the kernel driver:
dma_mmap_coherent(&dev->dev, vma, shared_image_data_virtual_address, shared_image_data_physical_address, length);
The shared_image_data_virtual_address is a pointer of type uint_64_t while the shared_image_data_physical_address is of type dma_addr_t and they where created earlier when I used the following code to allocate memory in kernel space:
shared_image_data_virtual_address = dma_alloc_coherent(&dev->dev, 4 * MBYTE, &shared_image_data_physical_address, GFP_KERNEL);
The address that I pass to the DMA of the FPGA is the shared_image_data_physical_address.
I hope that the above are helpful.
Thank you!
Of course in general the physical memory behind user mode allocations (e. g. pointers returned by malloc()) might very well move to some different physical address at the discretion of the kernel (for example if the memory is swapped to disk and is later recalled to some other segment of physical memory). However the user mode code will be unaffected as on those occasions the kernel will update the page tables to point to the new location with no change to the virtual address being used by the user.
If this (perhaps discontiguous) physical memory is to be the target of some DMA then there are the user mode mlock() and munlock() calls to prevent this this memory from being paged out and you can call get_user_pages() etc. to get the physical addresses to feed to the DMA hardware and so on. That’s all well and good- but that’s not my case.
Rather I have done some kernel mode allocations (using __get_free_pages()) to obtain physically contiguous memory as our PCIe card-based DMA engine does not support scatter-gather. All of this has been working fine for some time- until I encountered a motherboard on which I see hard crashes.
I have thus far not worried that the kernel might at any time just move the physical memory locations for these ranges (resulting in my DMA card writing data to the wrong locations).
But with the new crashes on this motherboard now have me questioning that assumption.
Can that kernel memory allocated with __get_free_pages() too be moved by the kernel at any time to a different physical address range?
And if so is there any kernel facility to prevent these moves?
I have an SoC which has both DSP and ARM cores on it and I would like to create a section of shared memory that both my userspace software, and DSP software are able to access. What would be the best way to allocate a buffer like this in Linux? Here is a little background, right now what I have is a kernel module in which I use kmalloc() to get a kernel buffer, I then use the __pa() macro from asm/page.h to get the physical address of my kernel buffer. I save this address as a sysfs entry so that my userspace code can get the physical address of this buffer. I can then write this address to the DSP so it knows where the shared memory location is, and I can also mmap /dev/mem or my own kernel module so that I can access this buffer from userspace (I could also use the read/write fileops).
For some reason I feel like this is overboard but I cannot find the best way to do what I am trying to do.
Would it be possible to just mmap \dev\mem a section of memory and just read and write to this section? My feeling is that this would not 'lock' this section of memory from the kernel, thus the kernel could still read/write to this memory without me knowing. Is this the case. After reading the memory management chapter of LDD3 I see that mmap creates a new VMA of the mapping. Would this lock this area of memory so that other processes would not get allocated this section of memory?
Any and all help is appreciated
Depending on the kind of DMA you're using, you need to allocate the buffer with dma_alloc_coherent(), or use standard allocations and the dma_map_* functions.
(You must not use __pa(); physical addresses are not necessarily the same as DMA bus addresses.)
To map the buffers to user space, use dma_mmap_coherent() for coherent buffers, or map the memory pages manually for streaming buffers.
For a similar requirement of mine, I had reserved about 16 MB of memory towards the end of ram and used it in both kernel and user space. Suppose you have 128 MB ram, you can set BOOTMEM argument as 112 MB in your boot loader. I am assuming you are using uboot. This will reserve 16 MB towards the end of the ram. Now in kernel and user space you can map this area and use it as shared memory.
I am developing a Linux kernel driver on 3.4. The purpose of this driver is to provide a mmap interface to Userspace from a buffer allocated in an other kernel module likely using kzalloc() (more details below). The pointer provided by mmap must point to the first address of this buffer.
I get the physical address from virt_to_phys(). I give this address right shifted by PAGE_SHIFT to remap_pfn_range() in my mmap fops call.
It is working for now but it looks to me that I am not doing the things properly because nothing ensure me that my buffer is at the top of the page (correct me if I am wrong). Maybe mmap()ing is not the right solution? I have already read the chapter 15 of LDD3 but maybe I am missing something?
Details:
The buffer is in fact a shared memory region allocated by the remoteproc module. This region is used within an asymetric multiprocessing design (OMAP4). I can get this buffer thanks to the rproc_da_to_va() call. That is why there is no way to use something like get_free_pages().
Regards
Kev
Yes, you're correct: there is no guarantee that the allocated memory is at the beginning of a page. And no simple way for you to both guarantee that and to make it truly shared memory.
Obviously you could (a) copy the data from the kzalloc'd address to a newly allocated page and insert that into the mmap'ing process' virtual address space, but then it's not shared with the original datum created by the other kernel module.
You could also (b) map the actual page being allocated by the other module into the process' memory map but it's not guaranteed to be on a page boundary and you'd also be sharing whatever other kernel data happened to reside in that page (which is both a security issue and a potential source of kernel data corruption by the user-space process into which you're sharing the page).
I suppose you could (c) modify the memory manager to return every piece of allocated data at the beginning of a page. This would work, but then every time a driver wants to allocate 12 bytes for some small structure, it will in fact be allocating 4K bytes (or whatever your page size is). That's going to waste a huge amount of memory.
There is simply no way to trick the processor into making the memory appear to be at two different offsets within a page. It's not physically possible.
Your best bet is probably to (d) modify the other driver to allocate the specific bits of data that you want to make shared in a way that ensures alignment on a page boundary (i.e. something you write to replace kzalloc).
Is the DMA address returned from this call the same as the physical address? LDD3 says the DMA address should be treated as opaque by the driver. I want to mmap this DMA buffer so user-space can read/write directly to it. Question is what PFN should I specify for the remap_pfn_range (which to my pleasant surprise now (kernel 3.4+) works for conventional memory same as for I/O memory). Can I just cast the DMA address to unsigned long and convert that to PFN? Isn't this a violation of what LDD3 said about opaqueness?
Does dma_alloc_coherent always use __get_free_pages internally? Does this mean the region is potentially always over-allocated (since first function takes bytes but second function allocates in units of pages)?
Is there a way to setup a single streaming mapping for multiple consecutive pages obtained from call to __get_free_pages? dma_map_page applies to only single pages.
No, the address returned is a virtual address, otherwise you wouldn't be able to access it from kernel space. It's dma_handle which represents the physical address, but it's opaque. You need to use virt_to_phys on the address it returns and then pass this to remap_pfn_range.
I don't believe it does (it's likely to be platform dependent though), but it does allocate pages. If you want smaller amounts of memory for DMA you should use dma_pool_create and then allocate regions from there.
You can use dma_map_single instead of dma_map_page.
I'd recommend consulting DMA-API.txt for more details on some of this stuff.