Windows has a FILE_FLAG_NO_BUFFERING flag that allows you to specify whether or not you want your I/O to be cached by the file system.
That's fine, but what if I want to use the cache if possible, but avoid modifying it?
In other words, how do you tell Windows the following?
Read this file from the cache if it's already cached, but my data doesn't exhibit locality, so do not put it into the cache!
The SCSI standard defines a Disable Page Out bit that does precisely this, so I'm wondering how (if at all) it is possible to use that feature from Windows (with cooperation of the file system cache too, of course)?
Edit: TL;DR:
What's the equivalent of FILE_FLAG_WRITE_THROUGH for reads?
About the closest Windows provides to what you're asking is FILE_FLAG_WRITE_THROUGH.
I see two flags that look suspiciously like what you are asking for:
FILE_FLAG_RANDOM_ACCESS
FILE_FLAG_SEQUENTIAL_SCAN
The later's doc clearly suggests that it won't retain pages in cache, though it will probably read-ahead sequentially. The former's doc is completely opaque, but would seem to imply what you want. If the pattern is quite random, hanging onto pages for later reuse would be a waste of memory.
Keep in mind that, for files, the Windows kernel always will use some pages of 'cache' to hold the I/O. It has nowhere else to put it. So it's not meaningful to say 'don't cache it,' as opposed to 'evict the old pages of this file before evicting some other pages.'
Related
I'm looking for a way to flush the L1-L2 cache using a kernel module.
Is there a way to completely flush the whole cluster cache (4 core configuration) or even better, write back the dirty cache lines into main memory?
It sounds weird that you want to flush your caches from a kernel module. That should be done by the core-kernel part and as a driver you should not have to worry about that.
Is there any specific reason you need to do that in a driver?
I think you want to have a look at 3.9 of "Understanding the Linux Virtual Memory Manager" [1] from Mel Gorman. I think what you are looking for is flush_cache_page(...)
[1] https://www.kernel.org/doc/gorman/
Well it seems that it is actually different the way that the caches are flushed in different architectures. Nevertheless, I didn't find an implementation that works. BUT, what I did was to find the Page table entry (PTE) of the particular page that I want to flush, and changed the memory attributes to Non-Cacheable. Then, the data went directly to the DRAM. (ARMv8)
Cheers
I am kind of pissed off by the browsers memory use. I would like to limit the total memory used by Chrome, opera, firefox etc. to 800MB for example.
It looks like a job for cgroups.
I've read about cgexec and it would do what I want...
However, I would also like to "prepare" a group called "internet", using a similar method as described here :
https://wiki.archlinux.org/index.php/cgroups#Persistent_group_configuration
And since it's mentioned :
Note: when using Systemd >= 205 to manage cgroups, you can ignore this file entirely.
I'm a bit scared. (and Google finds results relevent for the situation before systemd, but it's a blur for the current situation)
Since Systemd looks like it's becoming the new standard, how to do it with a long term support ?
(...And am I missing/messing something here, because it's quite unclear to me to be honest)
I generally think it is a bad idea since Chrome will probably crash when it will not able to allocate any more memory.
Alternatively, it will swap its data to the disk which is even worse.
Chrome's high memory consumption is what makes it fast.
If you insist on creating a cgroup for your browsers, I suggest creating a script that first creates the cgroups if it does not exist, then runs the application given in the script's parameters.
One possible solution to the problem of Why does WriteFile call ReadFile and how do I avoid it?. Is to write to file using CcPreparePinWrite and PIN_CALLER_TRACKS_DIRTY_DATA. Basically what this does is to make the cache manager map a file section into memory without having to read it from disk, since the entire section is assumed to be overwritten.
The PIN_CALLER_TRACKS_DIRTY_DATA flag is commonly used in cases where a file system is managing a log file that is written to but not read from. Because the existing file data will be overwritten and not read, the cache manager may return pages of zeros instead of faulting in the actual pages of file data from disk.
This is all great in theory. Though it seems quite complicated to achieve in practice. Especially since these are kernel-mode functions that cannot be called from a user-mode application.
Is there any way to achieve this behaviour using the regular WriteFile API? Or is there any good resource that further explain how to make use of the Cache Manager Routines?
I see some variables named 'dirty' in some source code at work and some other code. What does it mean? What is a dirty flag?
Generally, dirty flags are used to indicate that some data has changed and needs to eventually be written to some external destination. It isn't written immediate because adjacent data may also get changed and writing bulk of data is generally more efficient than writing individual values.
There's a deeper issue here - rather than "What does 'dirty mean?" in the context of code, I think we should really be asking - is 'dirty' an appropriate term for what is generally intended.
'Dirty' is potentially confusing and misleading. It will suggest to many new programmers corrupt or erroneous form data. The work 'dirty' implies that something is wrong and that the data needs to be purged or removed. Something dirty is, after all undesirable, unclean and unpleasant.
If we mean 'the form has been touched' or 'the form has been amended but the changes haven't yet been written to the server', then why not 'touched' or 'writePending' rather than 'dirty'?
That I think, is a question the programming community needs to address.
Dirty could mean a number of things, you need to provide more context. But in a very general sense a "dirty flag" is used to indicate whether something has been touched / modified.
For instance, see usage of "dirty bit" in the context of memory management in the wiki for Page Table
"Dirty" is often used in the context of caching, from application-level caching to architectural caching.
In general, there're two kinds of caching mechanisms: (1) write through; and (2) write back. We use WT and WB for short.
WT means that the write is done synchronously both to the cache and to the backing store. (By saying the cache and the backing store, for example, they can stand for the main memory and the disk, respectively, in the context of databases).
In contrast, for WB, initially, writing is done only to the cache. The write to the backing store is postponed until the cache blocks containing the data are about to be modified/replaced by new content.
The data is the dirty values. When implementing a WB cache, you can set dirty bits to indicate whether a cache block contains dirty value or not.
If one process does a write() of size (and alignment) S (e.g. 8KB), then is it possible for another process to do a read (also of size and alignment S and the same file) that sees a mix of old and new data?
The writing process adds a checksum to each data block, and I'd like to know whether I can use a reading process to verify the checksums in the background. If the reader can see a partial write, then it will falsely indicate corruption.
What standards or documents apply here? Is there a portable way to avoid problems here, preferably without introducing lots of locking?
When a function is guaranteed to complete without there being any chance of any other process/thread/anything seeing things in a half finished state, it's said to be atomic. It either has or hasn't happened, there is no part way. While I can't speak to Windows, there are very few file operations in POSIX (which is what Linux/BSD/etc attempt to stick to) that are guaranteed to be atomic. Reading and writing are not guaranteed to be atomic.
While it would be pretty unlikely for you to write 2 bytes to a file and another process only see one of those bytes written, if by dumb luck your write straddled two different pages in memory and the VM system had to do something to prepare the second page, it's possible you'd see one byte without the other in a second process. Usually if things are page aligned in your file, they will be in memory, but again you can't rely on that.
Here's a list someone made of what is atomic in POSIX, which is pretty short, and I can't vouch for it's authenticity. (I can't think of why unlink isn't listed, for example).
I'd also caution you against testing what appears to work and running with it, the moment you start accessing files over a network file system (NFS on Unix, or SMB mounts in Windows) a lot of things that seemed to be atomic before no longer are.
If you want to have a second process calculating checksums while a first process is writing the file, you may want to open a pipe between the two and have the first process write a copy of everything down the pipe to the checksumming process. That may be faster than dealing with locking.