After some theoretical discussion today I decided to do some research, but I did not find anything conclusive.
Here's the problem:
We have written a tool that reads around 10Gb of image files from a data set of several terabytes. We want to speed up the execution time by minimizing I/O overhead. The idea would be to "pre-warm" the disk cache, as we known beforehand what directory we will be reading from as the tool executes. Is there any API or method to give this hint to Windows so that it can start pre-warming the disk cache, speeding up future disk access as the files are already in RAM (of which there is plenty on the machines we run the tool on)?
I know Windows does readahead on a single file, but what if I have a directory with thousands of files?
I haven't found any direct win32 APIs or command line tools to do this directly.
What if I start a low priority background thread, opening all the files for reading and closing them?
I could of course memory map all the files and pin them in RAM, but that would probably run the risk of starving the main worker thread of I/O.
The general idea here is that the tool "bursts" I/O requests, as each thread will do I/O and CPU processing in sequence, hence we could use the "idle" I/O time to preload the remaining files into RAM.
(I could of course benchmark, and I will, but I would like to understand a bit more of how this works in order to be more scientific and less cargo culty).
Related
When using memory mapped files I'm getting in situations where Windows stalls since new memory is allocated and processed faster than it can be written to disk using memory mappedfiles.
The only solution I see is to throttle my processing while the MiMappedPageWriter and the KeBalanceSetManager are doing their jobs. I would be completely fine if the application is running slower instead of a complete OS freeze.
It already helped to use SetWorkingSetSizeEx using a hard limit, because the MiMappedPageWriter is starting earlier to page-out to disk, but still on some drives the data is allocated faster. For example an SSD with 250MB/s does not manage it, but with 500MB/s it is getting better. But I have to support a wide range of hardware and cannot rely on fast drives.
I found that there once was a performance counter, for example: Memory\Mapped File Bytes Written/sec, that I could use not monitor periodically (see: https://docs.microsoft.com/en-us/windows-server/management/windows-performance-monitor/memory-performance-counter-mapped-file-bytes-written-sec) but it seems that all the links have gone.
I have searched on many places, but couldn't find the performance counters for this.
Is there still a source for this?
I have a few Rust programs that read data from a file, do some operations, and write data on another file.
Simple enough, but I've been having a big issue in that my programs saturate the HDD max I/O and can only be executed when no other process is in use.
To be more precise, I'm currently using BufReader and BufWriter with a buffer size of 64 KB which is fantastic in and of itself to read/write a file as quickly as possible. But reading at 250MB/s and writing at the same time at 250MB/s has a tendency to overflow what the HDD can manage. Suffice to say that I'm all for speed and whatnot, but I realized that those Rust programs are asking for too much resources from the HDD and seems to be stalled by the Operating System (Windows) to let other processes work in peace. The files I'm reading/writing are generally a few Gigabytes
Now I know I could just add some form of wait() between each read/write operation on the disk but, I don't know how to find out at which speed I'm currently reading/writing and am looking for a more optimal solution. Plus even after reading the docs, I still can't find an option on BufReader/BufWriter that could limit HDD I/O operations to some arbitrary value (let's say 100MB/s for example).
I looked through the sysinfo crate but it does not seem to help in finding out current and maximum I/O for the HDD.
Am I out of luck and should I delve deeper in systems programming to find a solution ? Or is there already something that might teach how to prioritize my calls to the HDD or to simply limit my calls to some arbitrary value calculated from the currently available I/O rate of the HDD ?
After reading a bit more on the subject, apart from trying to read/write a lot of data and calculate from its performance, it seems like you can't find out HDD max I/O rate during the execution of the program and can only guess a constant at which HDD I/O rate can't go higher. (see https://superuser.com/questions/795483/how-to-limit-hdd-write-speed-for-chosen-programs/795488#795488)
But, you can still monitor disk activity, and with the number guessed earlier, you can use wait() more accurately than always limiting yourself at a constant speed. (here is a crate for Rust : https://github.com/myfreeweb/systemstat).
Prioritizing the process with the OS might be overkill since I'm trying to slip between other processes and share whatever resources are available at that time.
What Windows API can I use to monitor I/O performance metrics for a specific file or set of files? Performance counters seem to offer only higher level objects such as LogicalDisk and PhysicalDisk. I'm looking for something that Windows Resource Monitor uses under Disk->Disk Activity, i.e read/write bps and response time.
I did a quick search for "Perfmon individual files" and didn't see anything promising.
But I'm not sure measuring the performance of individual files will be all that meaningful. I/O activity is coalesced in the I/O stack in several places, the result being that at different levels the OS can't distinguish file I/O for one file versus another.
Assuming the app isn't doing any buffering/caching on it's own, the first place can be in buffering that happens in "C" (or similar) runtime libraries. Another place where coalescing occurs is in the file system (I'm assuming NTFS). I/O for file directories can be coalesced across multiple files in the same directory. I/O can be coalesced based on the file system's block size. So if multiple MFT entries share a block they can all be read/written at once. NTFS also implements caching and other I/O optimizations (read ahead). The performance of the cache can be affected by other processes running at the same time by either accessing the same file(s) you want to measure (helping to keep the file in cache) or by accessing other files (helping to evict your file from cache).
Coalescing also happens below the file system at the logical disk level. Single I/Os may service multiple files.
At the disk driver level single I/O requests may again involve multiple files. Additionally the driver (or more likely the drive firmware) can reorder disk I/Os based on knowledge it has about the drive "geometry" to gain additional throughput at the (possible) expense of response time. In this case I/O to your files may suffer compared to what it would see if the other processes weren't doing I/O at the same time.
Many disks implement caching in DRAM. This cache will also be affected by other processes in the same way that Window's cache is. Again affecting measured performance due to other process's activity.
If you still want to measure though, one way to circumvent the limitations in Perfmon is to put files or sets of files on different drives. The drives don't necessarily have to be different physical drives, they could be VHDs, or some other kind of virtual disk-on-physical disk. I know the Volume Snapshot Service (VSS) SDK has a little utility to create virtual drives out of files.
But putting your files on their own physical disks will probably give much more consistent results.
I'm passing 1-2 MB of data from one process to another, using a plain old file. Is it significantly slower than going through RAM entirely?
Before answering yes, please keep in mind that in modern Linux at least, when writing a file it is actually written to RAM, and then a daemon syncs the data to disk from time to time. So in that way, if process A writes a 1-2 MB into a file, then process B reads them within 1-2 seconds, process B would simply read the cached memory. It gets even better than that, because in Linux, there is a grace period of a few seconds before a new file is written to the hard disk, so if the file is deleted, it's not written at all to the hard disk. This makes passing data through files as fast as passing them through RAM.
Now that is Linux, is it so in Windows?
Edit: Just to lay out some assumptions:
The OS is reasonably new - Windows XP or newer for desktops, Windows Server 2003 or newer for servers.
The file is significantly smaller than available RAM - let's say less than 1% of available RAM.
The file is read and deleted a few seconds after it has been written.
When you read or write to a file Windows will often keep some or all of the file resident in memory (in the Standby List). So that if it is needed again, it is just a soft-page fault to map it into the processes' memory space.
The algorithm for what pages of a file will be kept around (and for how long) isn't publicly documented. So the short answer is that if you are lucky some or all of it may still be in memory. You can use the SysInternals tool VMmap to see what of your file is still in memory during testing.
If you want to increase your chances of the data remaining resident, then you should use Memory Mapped Files to pass the data between the two processes.
Good reading on Windows memory management:
Mysteries of Windows Memory Management Revealed
You can use FILE_ATTRIBUTE_TEMPORARY to hint that this data is never needed on disk:
A file that is being used for temporary storage. File systems avoid writing data back to mass storage if sufficient cache memory is available, because typically, an application deletes a temporary file after the handle is closed. In that scenario, the system can entirely avoid writing the data. Otherwise, the data is written after the handle is closed.
(i.e. you need use that flag with CreateFile, and DeleteFile immediately after closing that handle).
But even if the file remains cached, you still have to copy it twice: from your process A to the cache (the WriteFile call), and from cache to the proces B (ReadFile call).
Using memory mapped files (MMF, as josh poley already suggested) has the primary advantage of avoiding one copy: the same physical memory pages are mapped into both processes.
A MMF can be backed by virtual memory, which means basically that it always stays in memory unless swapping becomes necessary.
The major downside is that you can't easily grow the memory mapping to changing demands, you are stuck with the initial size.
Whether that matters for an 1-2 MB data transfer depends mostly on how you acquire and what you do with the data, in many scenarios the additional copy doesn't really matter.
I have an application which needs to create many small files in maximum performance (less than 1% of them may be read later), and I want to avoid using asynchronous file API to keep the simplicity of my code. The size of total files written cannot be pre-determined, so I figured that to achieve maximum performance, I would need:
1.Windows to utilize all unused RAM for cache (especially for file writes), with no regard of relibility or other issues. If I have more than 1GB of unused RAM and I create one million of 1KB files, I expect Windows to report "DONE" immediately, even if it has written nothing to disk yet.
OR
2.A memory-based file system backed by real on-disk file system. Whenever I write files, it should first write everything in memory only, and then update on-disk file system in background. No delay in synchronous calls unless there isn't enough free memory. Note it's different from tmpfs or RAM disk implementations on Windows, since they require fixed amount of memory and utilize page file when there isn't enough RAM.
For option 1, I have tested VHD files - while it does offer as high as 50% increase of total performance under some configurations, it still does flushing-to-disk and cause writing to wait unnecessarily. And it appears there is no way I can disable journaling or further tweak the write caching behavior.....
For option 2, I have yet found anything similar..... Do such things exist?