Reading Data from a Physical Hard Drive - winapi

I am trying to develop a program that goes and finds 2 connected unformatted physical drives and read bytes. The program currently runs in the administrator mode since that's the only way I guess the program can see unformatted hard drives. I am using visual studio 2015 and it runs in windows 7 machine.
The problem is that it can only read multiples of 512 (512 is the sector size). Currently the unformatted hard drives are located in disk 2 and 3 slots (they are both SSDs). It first reads 512 bytes (works without an issue) and doesn't do any more reads if it's a formatted hard drive. If it's an unformatted hard drive it goes ahead and read more bytes. If it's hard drive A it then reads the next 1024 bytes and it works (read_amount = 1024). If it's hard drive B it then reads the next 1025 bytes and it doesn't work (read_amount = 0). I am not sure why it can't read a multiple of a 512/sector sizes. My understanding is that when you call "CreateFile()" function with dwFlagsAndAttributes = FILE_ATTRIBUTE_NORMAL, I should be able to read sizes that are not multiples of sector sizes (if you use FILE_FLAG_NO_BUFFERING then you can only read multiples of 512 and I am NOT using that flag). See my code below.
// Hard_Drive_Read.cpp : Defines the entry point for the console application.
// This program assumes you have EXACTLY TWO unformatted hard drives connected to your computer.
#include <Windows.h>
#include <io.h>
#include <fcntl.h>
#include <fstream>
#include <iostream>
#include <iomanip>
using namespace std;
int main(int argc, char *argv[])
{
if (argc != 3)
{
cout << "Need to enter 2 arguments" << endl;
exit(0);
}
int frames_to_process = atoi(argv[2]);
if (frames_to_process < 1)
{
cout << "invalid argument 2" << endl;
exit(0);
}
//HANDLE hDisk_A;
//HANDLE hDisk_B;
LPCTSTR dsksrc = L"\\\\.\\PhysicalDrive";
wchar_t dsk[512] = L"";
bool channel_A_found = false;
bool channel_B_found = false;
char frame_header_A[1024];
char frame_header_B[1025];
HANDLE hDisk;
char buff_read[512];
DWORD read_amount = 0;
for (int i = 0; i < 4; i++)
{
swprintf(dsk, 511, L"%s%d", dsksrc, i);
hDisk = CreateFile(dsk, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
if (hDisk == INVALID_HANDLE_VALUE)
{
printf("%s%d%s", "couldn't open the drive ", i, "\n");
CloseHandle(hDisk);
}
else
{
printf("%s%d%s", "successfully open the drive ", i, "\n");
BOOL read_success_1 = ReadFile(hDisk, buff_read, 512, &read_amount, NULL);
cout << "read amount 1 - " << read_amount << endl;
if ((read_success_1 == TRUE) && (read_amount == 512))
{
if ((buff_read[510] == (char)0x55) && (buff_read[511] == (char)0xAA)) // test for a formatted drive; is there other identifiers?
{
cout << i << " is a formatted drive" << endl;
}
else
{
cout << "Not a formatted drive, trying to find sync " << endl;
ofstream writeBinary_Test;
if (i == 2)
{
writeBinary_Test.open("file_A_test.bin", ofstream::out | ofstream::binary);
ReadFile(hDisk, frame_header_A, 1024, &read_amount, NULL);
cout << "read amount " << read_amount << endl;
writeBinary_Test.write(frame_header_A, 1024);
writeBinary_Test.close();
}
else if(i == 3)
{
writeBinary_Test.open("file_B_test.bin", ofstream::out | ofstream::binary);
ReadFile(hDisk, frame_header_B, 1025, &read_amount, NULL);
cout << "read amount " << read_amount << endl;
writeBinary_Test.write(frame_header_B, 1025);
writeBinary_Test.close();
}
LARGE_INTEGER distanceToMove;
SetFilePointerEx(hDisk, distanceToMove, NULL, FILE_BEGIN);
}
}
else
{
}
}
if (channel_A_found && channel_B_found)
{
cout << "both drives found" << endl;
break;
}
}
if ((channel_A_found == false) || (channel_B_found == false))
{
cout << "Couldn't Find Hard Drive A or Drive B or Both" << endl;
cout << "Exiting the program" << endl;
exit(0);
}
CloseHandle(hDisk);
return 0;
}
Eventually, I want to use SetFilePointerEx() to move around the hard drive and I the program has to work with and data size (not multiples of 512). Therefore, it's imperative I can read sizes that's not multiples of 512. Any ideas of how to fix this program? Am I using my flags properly?
Any help is much appreciated!

The documentation for CreateFile says:
Volume handles can be opened as noncached at the discretion of the particular file system, even when the noncached option is not specified in CreateFile. You should assume that all Microsoft file systems open volume handles as noncached. The restrictions on noncached I/O for files also apply to volumes.
Although it doesn't spell it out explicitly, this applies to drives as well as to volumes.
In practice, this isn't a problem. It is straightforward to write a helper function that returns an arbitrary amount of data from an arbitrary offset, while performing only aligned reads.

It's imperative I can read sizes that's not multiples of 512.
That is not possible. For direct access of a disk, you can only read and write multiples of the sector size. Furthermore, you must align your read and write operations. That is the file pointer must be at a multiple of the sector size.
If you want to present an interface that allows arbitrary seeking, reading and writing, then you will need to implement your own buffering on top of the aligned raw disk access.

Related

Programmatically Change Windows Laptop's Refresh Rate

I'm thinking of coding something up that will change a laptop's refresh rate based on whether or not the device is plugged in.
From my research, these are two links I came across. One is 20 years old and the other is from Microsoft, but I don't see any mentions of refresh rate specifically.
https://www.codeproject.com/Articles/558/Changing-your-monitor-s-refresh-rate
https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-changedisplaysettingsa?redirectedfrom=MSDN
Does anyone have any insight into how to do this? I'm not too particular about what language would have to be used for it, so let me know whatever would be most viable. Of course I'd also have to be able to check a change in state for plugged in/unplugged, but I haven't gotten to that point yet.
I'm mostly targeting Windows 10 since that's what my device is on.
You can use EnumDisplaySettings to enumerate the information of the current display device, and then set the display by ChangeDisplaySettingsA.
If you want to modify the refresh rate, you only need to modify the dmDisplayFrequency parameter of DEVMODEA.
Here is the sample:
#include <Windows.h>
#include <iostream>
using namespace std;
int main(int argc, const char* argv[])
{
DEVMODE dm;
ZeroMemory(&dm, sizeof(dm));
dm.dmSize = sizeof(dm);
if (0 != EnumDisplaySettings(NULL, ENUM_CURRENT_SETTINGS, &dm))
{
cout << "DisplayFrequency before setting = " << dm.dmDisplayFrequency << endl;
dm.dmDisplayFrequency = 60; //set the DisplayFrequency
LONG ret = ChangeDisplaySettingsEx(NULL, &dm, NULL, 0, NULL);
std::cout << "ChangeDisplaySettingsEx returned " << ret << '\n';
if (0 != EnumDisplaySettings(NULL, ENUM_CURRENT_SETTINGS, &dm))
{
cout << "DisplayFrequency after setting = " << dm.dmDisplayFrequency << endl;
}
switch (ret)
{
case DISP_CHANGE_SUCCESSFUL:
std::cout << "display successfully changed\n";
break;
case DISP_CHANGE_BADDUALVIEW:
std::cout << "The settings change was unsuccessful because the system is DualView capable\n";
break;
case DISP_CHANGE_BADFLAGS:
std::cout << "An invalid set of flags was passed in.\n";
break;
case DISP_CHANGE_BADMODE:
std::cout << "The graphics mode is not supported.\n";
break;
case DISP_CHANGE_BADPARAM:
std::cout << "An invalid parameter was passed in. This can include an invalid flag or combination of flags.\n";
break;
case DISP_CHANGE_FAILED:
std::cout << "The display driver failed the specified graphics mode.\n";
break;
case DISP_CHANGE_NOTUPDATED:
std::cout << "Unable to write settings to the registry.\n";
break;
case DISP_CHANGE_RESTART:
std::cout << "The computer must be restarted for the graphics mode to work.\n";
break;
}
}
system("pause");
}
This example is not always successful. Whether you can modify the refresh rate depends on whether your monitor supports it. This is the output of successful setup:

Full duplex named pipe lockup when written to

I'm trying to use one NamedPipe for bi-direction IPC. In my mind (and I can't find more information on MSDN), one full-duplex pipe would be sufficient. Here's my code.
//Compiled with these commands during my test:
//g++ -DCLIENT -o client.exe xxx.cpp
//g++ -DSERVER -o server.exe xxx.cpp
#include <iostream>
#include <windows.h>
using namespace std;
DWORD WINAPI ReadingThread(LPVOID a)
{
HANDLE pipe = (HANDLE)a;
BOOL result;
char buffer[256];
DWORD numBytesRead;
while (true)
{
result = ReadFile(pipe, buffer, sizeof(buffer) - 1, &numBytesRead, NULL);
if (result)
{
buffer[numBytesRead] = 0;
cout << "[Thread] Number of bytes read: " << numBytesRead << endl;
cout << "[Thread] Message: " << endl
<< buffer << endl
<< endl;
}
else
{
cout << "[Thread] Failed to read data from the pipe. err=" << GetLastError() << endl;
break;
}
}
return 0;
}
int main(int argc, const char **argv)
{
#ifdef CLIENT
cout << "[Main] Connecting to pipe..." << endl;
HANDLE pipe = CreateFileA("\\\\.\\pipe\\PipeTest", GENERIC_READ | GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
#else
cout << "[Main] Creating an instance of a named pipe..." << endl;
HANDLE pipe = CreateNamedPipeA("\\\\.\\pipe\\PipeTest", PIPE_ACCESS_DUPLEX, PIPE_TYPE_BYTE, 1, 0, 0, 0, NULL);
#endif
if (pipe == NULL || pipe == INVALID_HANDLE_VALUE)
{
cout << "[Main] Failed to acquire pipe handle." << endl;
return 1;
}
#ifdef CLIENT
#else
cout << "[Server] Waiting for a client to connect to the pipe..." << endl;
BOOL result = ConnectNamedPipe(pipe, NULL);
if (!result)
{
cout << "[Server] Failed to make connection on named pipe." << endl;
CloseHandle(pipe);
return 1;
}
cout << "[Server] Client is here!" << endl;
{
const char *buf = "Hello pipe!\n";
WriteFile(pipe, buf, strnlen(buf, 30), 0, 0);
}
#endif
CreateThread(0, 0, ReadingThread, pipe, 0, 0);
cout << "[Main] Ready to send data." << endl;
while (true)
{
char buffer[128];
DWORD numBytesWritten = 0;
BOOL result;
cin >> buffer;
if (!strcmp(buffer, "q"))
{
break;
}
cout << "[Main] Writing data to pipe..." << endl;
result = WriteFile(pipe, buffer, strnlen(buffer, _countof(buffer)), &numBytesWritten, 0);
if (result)
{
cout << "[Main] Written " << numBytesWritten << " bytes to the pipe." << endl;
}
else
{
cout << "[Main] Failed to write data to the pipe. err=" << GetLastError() << endl;
}
}
CloseHandle(pipe);
cout << "[Main] Done." << endl;
return 0;
}
I can get the "Hello pipe!" message from server-side to client-side. And I'm expecting to type some string on either program's terminal and press enter, and see it on the other side.
However after the hello message, both program will stuck on the WriteFile call. Meanwhile the thread is stuck at the ReadFile call. How can I make it work, or did I left something out?
when file created for synchronous I/O (flag FO_SYNCHRONOUS_IO present in FILE_OBJECT ) all I/O operations on file is serialized - new operation will be wait in I/O manager before passed to driver, until current(if exist) not complete. in concurrent can execute only single I/O request. if we do blocked read in dedicated thread - all another I/O request on this file will be blocked until read not complete. this related not only to write. even query file name/attributes will block here. as result render reading in separate not help here - we block on first write attemp. solution here use asynchronous files - this let any count of I/O operation execute in concurrent.
Named Pipes in Windows are HALF DUPLEX. As demonstrated on Windows 10. The MSDN Documentation is Wrong. A request has been submitted to Microsoft to correct their documentation.
While a pipe can be opened on the client to be "Generic Read | Generic Write" you can NOT do both at the same time.
And Overlapped IO submitted after the First Overlapped IO will break the pipe.
You can submit overlapped io. Then Wait for it to finish. Then submit the next overlapped io. You can not simultaneously Submit overlapped Reads AND overlapped Writes.
This is by definition, "Half Duplex".

How to prevent flushing to disk of a memory map opened on a windows temporary delete-on-close file

UPDATE 2 / TL;DR
Is there some way to prevent dirty pages of a windows FILE_FLAG_DELETE_ON_CLOSE temporary file from being flushed as a result of closing memory maps opened on these files?
Yes. If you do not need to do anything with the files themselves after their initial creation and you implement some naming conventions, this is possible through the strategy explained in this answer.
Note: I am still quite interested in finding out the reasons for why there is so much difference in the behavior depending on how maps are created and the order of disposal/unmapping.
I have been looking into some strategies for an inter-process shared memory data structure that allows growing and shrinking its committed capacity on windows by using a chain of "memory chunks."
One possible way is to use pagefile backed named memory maps as the chunk memory. An advantage of this strategy is the possibility to use SEC_RESERVE to reserve a big chunk of memory address space and incrementally allocate it using VirtualAlloc with MEM_COMMIT. Disadvantages appear to be (a) the requirement to have SeCreateGlobalPrivilege permissions to allow using a shareable name in the Global\ namespace and (b) the fact that all committed memory contributes to the system commit charge.
To circumvent these disadvantages, I started investigating the use of temporary file backed memory maps. I.e. memory maps over files created using the FILE_FLAG_DELETE_ON_CLOSE | FILE_ATTRIBUTE_TEMPORARY flags combination. This appears to be a recommended strategy that according to e.g. this blog post should prevent flushing the mapped memory to disk (unless memory pressure causes dirty mapped pages to be paged out).
I am however observing that closing the map/file handle before the owning process exits, causes dirty pages to be flushed to disk. This occurs even if the view/file handle is not the one through which the dirty pages were created and when these views/file handles were opened after the pages were 'dirtied' in a different view.
It appears that changing the order of disposal (i.e. unmapping the view first or closing the file handle first) has some impact on when the disk flush is initiated, but not on the fact that flushing takes place.
So my questions are:
Is there some way to use temporary file backed memory maps and prevent them from flushing dirty pages when the map/file is closed, taking into account that multiple threads within a process/multiple processes may have open handles/views to such a file?
If not, what is/could be the reason for the observed behavior?
Is there an alternative strategy that I may have overlooked?
UPDATE
Some additional info: When running the "arena1" and "arena2" parts of the sample code below in two separate (independent) processes, with "arena1" being the process that creates the shared memory regions and "arena2" the one that opens them, the following behavior is observed for maps/chunks that have dirty pages:
If closing the view before the file handle in the "arena1" process, it flushes each of these chunks to disk in what seems a (partially) synchronous process (i.e. it blocks the disposing thread for several seconds), independent of whether or not the "arena2" process was started.
If closing the file handle before the view, disk flushes only occur for those maps/chunks that are closed in the "arena1" process while the "arena2" process still has an open handle to those chunks, and they appear to be 'asynchronous', i.e. not blocking the application thread.
Refer to the (c++) sample code below that allows reproducing the problem on my system (x64, Win7):
static uint64_t start_ts;
static uint64_t elapsed() {
return ::GetTickCount64() - start_ts;
}
class PageArena {
public:
typedef uint8_t* pointer;
PageArena(int id, const char* base_name, size_t page_sz, size_t chunk_sz, size_t n_chunks, bool dispose_handle_first) :
id_(id), base_name_(base_name), pg_sz_(page_sz), dispose_handle_first_(dispose_handle_first) {
for (size_t i = 0; i < n_chunks; i++)
chunks_.push_back(new Chunk(i, base_name_, chunk_sz, dispose_handle_first_));
}
~PageArena() {
for (auto i = 0; i < chunks_.size(); ++i) {
if (chunks_[i])
release_chunk(i);
}
std::cout << "[" << ::elapsed() << "] arena " << id_ << " destructed" << std::endl;
}
pointer alloc() {
auto ptr = chunks_.back()->alloc(pg_sz_);
if (!ptr) {
chunks_.push_back(new Chunk(chunks_.size(), base_name_, chunks_.back()->capacity(), dispose_handle_first_));
ptr = chunks_.back()->alloc(pg_sz_);
}
return ptr;
}
size_t num_chunks() {
return chunks_.size();
}
void release_chunk(size_t ndx) {
delete chunks_[ndx];
chunks_[ndx] = nullptr;
std::cout << "[" << ::elapsed() << "] chunk " << ndx << " released from arena " << id_ << std::endl;
}
private:
struct Chunk {
public:
Chunk(size_t ndx, const std::string& base_name, size_t size, bool dispose_handle_first) :
map_ptr_(nullptr), tail_(nullptr),
handle_(INVALID_HANDLE_VALUE), size_(0),
dispose_handle_first_(dispose_handle_first) {
name_ = name_for(base_name, ndx);
if ((handle_ = create_temp_file(name_, size)) == INVALID_HANDLE_VALUE)
handle_ = open_temp_file(name_, size);
if (handle_ != INVALID_HANDLE_VALUE) {
size_ = size;
auto map_handle = ::CreateFileMappingA(handle_, nullptr, PAGE_READWRITE, 0, 0, nullptr);
tail_ = map_ptr_ = (pointer)::MapViewOfFile(map_handle, FILE_MAP_ALL_ACCESS, 0, 0, size);
::CloseHandle(map_handle); // no longer needed.
}
}
~Chunk() {
if (dispose_handle_first_) {
close_file();
unmap_view();
} else {
unmap_view();
close_file();
}
}
size_t capacity() const {
return size_;
}
pointer alloc(size_t sz) {
pointer result = nullptr;
if (tail_ + sz <= map_ptr_ + size_) {
result = tail_;
tail_ += sz;
}
return result;
}
private:
static const DWORD kReadWrite = GENERIC_READ | GENERIC_WRITE;
static const DWORD kFileSharing = FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE;
static const DWORD kTempFlags = FILE_ATTRIBUTE_NOT_CONTENT_INDEXED | FILE_FLAG_DELETE_ON_CLOSE | FILE_ATTRIBUTE_TEMPORARY;
static std::string name_for(const std::string& base_file_path, size_t ndx) {
std::stringstream ss;
ss << base_file_path << "." << ndx << ".chunk";
return ss.str();
}
static HANDLE create_temp_file(const std::string& name, size_t& size) {
auto h = CreateFileA(name.c_str(), kReadWrite, kFileSharing, nullptr, CREATE_NEW, kTempFlags, 0);
if (h != INVALID_HANDLE_VALUE) {
LARGE_INTEGER newpos;
newpos.QuadPart = size;
::SetFilePointerEx(h, newpos, 0, FILE_BEGIN);
::SetEndOfFile(h);
}
return h;
}
static HANDLE open_temp_file(const std::string& name, size_t& size) {
auto h = CreateFileA(name.c_str(), kReadWrite, kFileSharing, nullptr, OPEN_EXISTING, kTempFlags, 0);
if (h != INVALID_HANDLE_VALUE) {
LARGE_INTEGER sz;
::GetFileSizeEx(h, &sz);
size = sz.QuadPart;
}
return h;
}
void close_file() {
if (handle_ != INVALID_HANDLE_VALUE) {
std::cout << "[" << ::elapsed() << "] " << name_ << " file handle closing" << std::endl;
::CloseHandle(handle_);
std::cout << "[" << ::elapsed() << "] " << name_ << " file handle closed" << std::endl;
}
}
void unmap_view() {
if (map_ptr_) {
std::cout << "[" << ::elapsed() << "] " << name_ << " view closing" << std::endl;
::UnmapViewOfFile(map_ptr_);
std::cout << "[" << ::elapsed() << "] " << name_ << " view closed" << std::endl;
}
}
HANDLE handle_;
std::string name_;
pointer map_ptr_;
size_t size_;
pointer tail_;
bool dispose_handle_first_;
};
int id_;
size_t pg_sz_;
std::string base_name_;
std::vector<Chunk*> chunks_;
bool dispose_handle_first_;
};
static void TempFileMapping(bool dispose_handle_first) {
const size_t chunk_size = 256 * 1024 * 1024;
const size_t pg_size = 8192;
const size_t n_pages = 100 * 1000;
const char* base_path = "data/page_pool";
start_ts = ::GetTickCount64();
if (dispose_handle_first)
std::cout << "Mapping with 2 arenas and closing file handles before unmapping views." << std::endl;
else
std::cout << "Mapping with 2 arenas and unmapping views before closing file handles." << std::endl;
{
std::cout << "[" << ::elapsed() << "] " << "allocating " << n_pages << " pages through arena 1." << std::endl;
PageArena arena1(1, base_path, pg_size, chunk_size, 1, dispose_handle_first);
for (size_t i = 0; i < n_pages; i++) {
auto ptr = arena1.alloc();
memset(ptr, (i + 1) % 256, pg_size); // ensure pages are dirty.
}
std::cout << "[" << elapsed() << "] " << arena1.num_chunks() << " chunks created." << std::endl;
{
PageArena arena2(2, base_path, pg_size, chunk_size, arena1.num_chunks(), dispose_handle_first);
std::cout << "[" << ::elapsed() << "] arena 2 loaded, going to release chunks 1 and 2 from arena 1" << std::endl;
arena1.release_chunk(1);
arena1.release_chunk(2);
}
}
}
Please refer to this gist that contains the output of running the above code and links to screen captures of system free memory and disk activity when running TempFileMapping(false) and TempFileMapping(true) respectively.
After the bounty period expired without any answers that provided more insight or solved the mentioned problem, I decided to dig a little deeper and experiment some more with several combinations and sequences of operations.
As a result, I believe I have found a way to achieve memory maps shared between processes over temporary, delete-on-close files, that are not being flushed to disk when they are closed.
The basic idea involves creating the memory map when a temp file is newly created with a map name that can be used in a call to OpenFileMapping:
// build a unique map name from the file name.
auto map_name = make_map_name(file_name);
// Open or create the mapped file.
auto mh = ::OpenFileMappingA(FILE_MAP_ALL_ACCESS, false, map_name.c_str());
if (mh == 0 || mh == INVALID_HANDLE_VALUE) {
// existing map could not be opened, create the file.
auto fh = ::CreateFileA(name.c_str(), kReadWrite, kFileSharing, nullptr, CREATE_NEW, kTempFlags, 0);
if (fh != INVALID_HANDLE_VALUE) {
// set its size.
LARGE_INTEGER newpos;
newpos.QuadPart = desired_size;
::SetFilePointerEx(fh, newpos, 0, FILE_BEGIN);
::SetEndOfFile(fh);
// create the map
mh = ::CreateFileMappingA(mh, nullptr, PAGE_READWRITE, 0, 0, map_name.c_str());
// close the file handle
// from now on there will be no accesses using file handles.
::CloseHandle(fh);
}
}
Thus, the file handle is only used when the file is newly created, and closed immediately after the map is created, while the map handle itself remains open, to allow opening the mapping without requiring access to a file handle. Note that a race condition exists here, that we would need to deal with in any "real code" (as well as adding decent error checking and handling).
So if we got a valid map handle, we can create the view:
auto map_ptr = MapViewOfFile(mh, FILE_MAP_ALL_ACCESS, 0, 0, 0);
if (map_ptr) {
// determine its size.
MEMORY_BASIC_INFORMATION mbi;
if (::VirtualQuery(map_ptr, &mbi, sizeof(MEMORY_BASIC_INFORMATION)) > 0)
map_size = mbi.RegionSize;
}
When, some time later closing a mapped file: close the map handle before unmapping the view:
if (mh == 0 || mh == INVALID_HANDLE_VALUE) {
::CloseHandle(mh);
mh = INVALID_HANDLE_VALUE;
}
if (map_ptr) {
::UnmapViewOfFile(map_ptr);
map_ptr = 0;
map_size = 0;
}
And, according to the test I have performed so far, this does not cause flushing dirty pages to disk on close, problem solved. Well partially anyway, there may still be a cross-session map name sharing issue.
If I take it correctly, commenting out Arena2 part of code shall reproduce the issue without the need for second process. I have tried this:
I edited base_path as follows for convenience:
char base_path[MAX_PATH];
GetTempPathA(MAX_PATH, base_path);
strcat_s(base_path, MAX_PATH, "page_pool");
I edited n_pages = 1536 * 128 to bring the used memory to 1.5GB, compared to your ~800mb.
I have tested TempFileMapping(false) and TempFileMapping(true), one at a time, for the same results.
I have tested with Arena2 commented out and intact, for the same results.
I have tested on Win8.1 x64 and Win7 x64, for ±10% same results.
In my tests, code runs in 2400ms ±10%, only 500ms ±10% spent on deallocating. That's clearly not enough for a flush of 1.5GB on a low-spinning silent HDDs I have there.
So, the question is, what are you observing? I'd suggest that you:
Provide your times for comparison
Use a different computer for tests, paying attention to excluding software issues such as "same antivirus"
Verify that you're not experiencing a RAM shortage.
Use xperf to see what's happening during the freeze.
Update
I have tested on yet another Win7 x64, and times are 890ms full, 430ms spent on dealloc. I have looked into your results, and what is VERY suspicious is that almost exactly 4000ms is spent in freeze each time on your machine. That can't be a mere coincidence, I believe. Also, it's rather obvious now the the problem is somehow bound to a specific machine you're using. So my suggestions are:
As stated above, test on another computer yourself
As stated above, Use XPerf, it will allow you to see what exactly happens in user mode and kernel mode during the freeze (I really suspect some non-standard driver in the middle)
Play with number of pages and see how it affects the freeze length.
Try to store files on a different disk drive on the same computer where you have tested initially.

Why Intel Pin cannot identify the image/routine of some executed instructions?

I am creating a large pintool and I have two questions:
The tool (abridged below to the relevant part only) sometimes cannot identify the image/routine for particular executed instructions. Does anybody know when/why can that happen?
The tool (when instrumenting a Barnes-Hut benchmark) always terminates with an out-of-memory (OOM) error after running for a while (although the benchmark, when run standalone, completes successfully). Which tools to use to debug/trace the OOM error of Pin-instrumented applications?
int main(int argc, char *argv[])
{
PIN_InitSymbols();
if( PIN_Init(argc, argv) )
{
return 0;
}
INS_AddInstrumentFunction(Instruction, 0);
PIN_StartProgram();
return 0;
}
VOID Instruction(INS ins, VOID *v)
{
INS_InsertPredicatedCall( ins,
IPOINT_BEFORE,
(AFUNPTR) handle_ins_execution,
IARG_INST_PTR,
.....);
}
VOID handle_ins_execution (ADDRINT addr, ...)
{
PIN_LockClient();
IMG img = IMG_FindByAddress(addr);
RTN rtn = RTN_FindByAddress(addr);
PIN_UnlockClient();
if( IMG_Valid(img) ) {
std::cerr << "From Image : " << IMG_Name( img ) << std::endl;
} else {
std::cerr << "From Image : " << "(UKNOWN)" << std::endl;
}
if( RTN_Valid(rtn) ) {
std::cerr << "From Routine : " << RTN_Name(rtn) << std::endl;
} else {
std::cerr << "From Routine : " << "(UKNOWN)" << std::endl;
}
}
I recently asked this on the PinHeads forum, and I'm awaiting a response. What I have read in the documentation is that the IMG_FindByAddress function operates by looking "for each image, check if the address is within the mapped memory region of one of its segments." It may be possible that instructions are executed that are not within the valid ranges.
The best way to know what image it is in for cases like this is to look at the context. My pintool (based on DebugTrace) continues to run even without knowing what image it is in. You can look at the log entries before and after this occurs. I see this all the time in dydl on OSX.

Cause of serial port transmitting bad data and WriteFile return wrong number of bytes written?

Bugfix update:
As of Jun, 2013 FTDI did acknowledge to me that the bug was real. They have since released a new version of their driver (2.8.30.0, dated 2013-July-12) that fixes the problem. The driver made it through WHQL around the first of August, 2013 and is available via Windows Update at this time.
I've re-tested running the same test code and am not able to reproduce the problem with the new driver, so at the moment the fix seems to be 'upgrade the driver'.
The original question:
I've got an 8 port USB-serial device (from VsCOM) that is based on the FTDI FT2232D chip. When I transmit at certain settings from one of the ports, AND I use the hardware handshaking to stop and start the data flow from the other end, I get two symptoms:
1) The output data sometimes becomes garbage. There will be NUL characters, and pretty much any random thing you can think of.
2) The WriteFile call will sometimes return a number of bytes GREATER than the number I asked it to write. That's not a typo. I ask for 30 bytes to be transmitted and the number of bytes sent comes back 8192 (and yes, I do clear the number sent to 0 before I make the call).
Relevant facts:
Using FTDI drivers 2.8.24.0, which is the latest as of today.
Serial port settings are 19200, 7 data bits, odd parity, 1 stop bit.
I get this same behavior with another FTDI based serial device, this time a single port one.
I get the same behavior with another 8 port device of the same type.
I do NOT get this behavior when transmitting on the built-in serial ports (COM1).
I have a very simple 'Writer' program that just transmits continuously and a very simple 'Toggler' program that toggles RTS once per second. Together these seem to trigger the issue within 60 seconds.
I have put an issue into the manufacturer of the device, but they've not yet had much time to respond.
Compiler is mingw32, the one included with the Qt installer for Qt 4.8.1 (gcc 4.4.0)
I'd like to know first off, if there's anything that anyone can think of that I could possibly do to trigger this behavior. I can't conceive of anything, but there's always things I don't know.
Secondly, I've attached the Writer and Toggler test programs. If anyone can spot some issue that might trigger the program, I'd love to hear about it. I have a lot of trouble thinking that there is a driver bug (especially from something as mature as the FTDI chip), but the circumstances force me to think that there's at least SOME driver involvement. At the least, no matter what I do to it, it shouldn't be returning a number of bytes written greater than what I asked it to write.
Writer program:
#include <iostream>
#include <string>
using std::cerr;
using std::endl;
#include <stdio.h>
#include <windows.h>
int main(int argc, char **argv)
{
cerr << "COM Writer, ctrl-c to end" << endl;
if (argc != 2) {
cerr << "Please specify a COM port for parameter 2";
return 1;
}
char fixedbuf[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";
std::string portName = "\\\\.\\";
portName += argv[1];
cerr << "Transmitting on port " << portName << endl;
HANDLE ph = CreateFileA( portName.c_str(),
GENERIC_READ | GENERIC_WRITE,
0, // must be opened with exclusive-access
NULL, // default security attributes
OPEN_EXISTING, // must use OPEN_EXISTING
0, // overlapped I/O
NULL ); // hTemplate must be NULL for comm devices
if (ph == INVALID_HANDLE_VALUE) {
cerr << "CreateFile " << portName << " failed, error " << GetLastError() << endl;
return 1;
}
COMMCONFIG ccfg;
DWORD ccfgSize = sizeof(COMMCONFIG);
ccfg.dwSize = ccfgSize;
GetCommConfig(ph, &ccfg, &ccfgSize);
GetCommState(ph, &(ccfg.dcb));
ccfg.dcb.fBinary=TRUE;
ccfg.dcb.fInX=FALSE;
ccfg.dcb.fOutX=FALSE;
ccfg.dcb.fAbortOnError=FALSE;
ccfg.dcb.fNull=FALSE;
// Camino is 19200 7-O-1
ccfg.dcb.BaudRate = 19200;
ccfg.dcb.Parity = ODDPARITY;
ccfg.dcb.fParity = TRUE;
ccfg.dcb.ByteSize = 7;
ccfg.dcb.StopBits = ONESTOPBIT;
// HW flow control
ccfg.dcb.fOutxCtsFlow=TRUE;
ccfg.dcb.fRtsControl=RTS_CONTROL_HANDSHAKE;
ccfg.dcb.fInX=FALSE;
ccfg.dcb.fOutX=FALSE;
COMMTIMEOUTS ctimeout;
DWORD tout = 10;// 10 ms
ctimeout.ReadIntervalTimeout = tout;
ctimeout.ReadTotalTimeoutConstant = tout;
ctimeout.ReadTotalTimeoutMultiplier = 0;
ctimeout.WriteTotalTimeoutMultiplier = tout;
ctimeout.WriteTotalTimeoutConstant = 0;
SetCommConfig(ph, &ccfg, sizeof(COMMCONFIG));
SetCommTimeouts(ph, &ctimeout);
DWORD nwrite = 1;
for(;;) {
nwrite++;
if (nwrite > 30) nwrite = 1;
DWORD nwritten = 0;
if (!WriteFile(ph, fixedbuf, nwrite, &nwritten, NULL)) {
cerr << "f" << endl;
}
if ((nwritten != 0) && (nwritten != nwrite)) {
cerr << "nwrite: " << nwrite << " written: " << nwritten << endl;
}
}
return 0;
}
Toggler program:
#include <iostream>
#include <string>
using std::cerr;
using std::endl;
#include <stdio.h>
#include <windows.h>
int main(int argc, char **argv)
{
cerr << "COM Toggler, ctrl-c to end" << endl;
cerr << "Flips the RTS line every second." << endl;
if (argc != 2) {
cerr << "Please specify a COM port for parameter 2";
return 1;
}
std::string portName = "\\\\.\\";
portName += argv[1];
cerr << "Toggling RTS on port " << portName << endl;
HANDLE ph = CreateFileA( portName.c_str(),
GENERIC_READ | GENERIC_WRITE,
0, // must be opened with exclusive-access
NULL, // default security attributes
OPEN_EXISTING, // must use OPEN_EXISTING
0, // overlapped I/O
NULL ); // hTemplate must be NULL for comm devices
if (ph == INVALID_HANDLE_VALUE) {
cerr << "CreateFile " << portName << " failed, error " << GetLastError() << endl;
return 1;
}
COMMCONFIG ccfg;
DWORD ccfgSize = sizeof(COMMCONFIG);
ccfg.dwSize = ccfgSize;
GetCommConfig(ph, &ccfg, &ccfgSize);
GetCommState(ph, &(ccfg.dcb));
ccfg.dcb.fBinary=TRUE;
ccfg.dcb.fInX=FALSE;
ccfg.dcb.fOutX=FALSE;
ccfg.dcb.fAbortOnError=FALSE;
ccfg.dcb.fNull=FALSE;
// Camino is 19200 7-O-1
ccfg.dcb.BaudRate = 19200;
ccfg.dcb.Parity = ODDPARITY;
ccfg.dcb.fParity = TRUE;
ccfg.dcb.ByteSize = 7;
ccfg.dcb.StopBits = ONESTOPBIT;
// no flow control (so we can do manually)
ccfg.dcb.fOutxCtsFlow=FALSE;
ccfg.dcb.fRtsControl=RTS_CONTROL_DISABLE;
ccfg.dcb.fInX=FALSE;
ccfg.dcb.fOutX=FALSE;
COMMTIMEOUTS ctimeout;
DWORD tout = 10;// 10 ms
ctimeout.ReadIntervalTimeout = tout;
ctimeout.ReadTotalTimeoutConstant = tout;
ctimeout.ReadTotalTimeoutMultiplier = 0;
ctimeout.WriteTotalTimeoutMultiplier = tout;
ctimeout.WriteTotalTimeoutConstant = 0;
SetCommConfig(ph, &ccfg, sizeof(COMMCONFIG));
SetCommTimeouts(ph, &ctimeout);
bool rts = true;// true for set
for(;;) {
if (rts)
EscapeCommFunction(ph, SETRTS);
else
EscapeCommFunction(ph, CLRRTS);
rts = !rts;
Sleep(1000);// 1 sec wait.
}
return 0;
}
I don't have a good answer yet from FTDI, but I've got the following suggestions for anyone dealing with this issue:
1) Consider switching to a non-FTDI usb-serial converter. This is what my company did, but certainly this isn't an option for everyone (we're putting the chip in our own product). We're using Silicon Labs chips now, but I think there are one or two other vendors as well.
2) Per Hans Passant in the comments - reconsider the use of RTS/CTS signalling. If the writes don't fail due to blocking, then you shouldn't trigger this bug.
3) Set all writes to infinite timeout. Again, no fail due to blocking, no triggering of the bug. This may not be appropriate for all applications, of course.
Note that if pursuing strategy #3, if Overlapped IO is used for writes, then CancelIo and it's newer cousin CancelIoEx could be used to kill off the writes if necessary. I have NOT tried doing so, but I suspect that such cancels might also result in triggering this bug. If they were only used when closing the port anyway, then it might be you could get away with it, even if they do trigger the bug.
If anyone else is still seeing this -- update your FTDI driver to 2.8.30.0 or later, as this is caused by a driver bug in earlier versions of the FTDI driver.

Resources