Win32API ReadFile Call worked and now never returns after call - winapi

When the code executes the ReadFile the call never returns. Previously the way I saved the Handle and passed it from function to function was not properly done. I changed the way the handle was stored and it works with all of the other calls in the program except the read. I have looked and compared all of the variables used and they check out.
Here is the code. instance->Master has the handle from the create. I added a GetCommState call before the read and it executes correctly.
'''
ATCA_STATUS swi_silab_receive_byte(ATCASWIMaster_t* instance, uint8_t* data)
{
uint8_t retries = 3;
DWORD NoBytesRead = 0; // Bytes read by ReadFile()
uint8_t SerialBuffer; //Buffer to send and receive data
DWORD Byte_count = (DWORD)sizeof(SerialBuffer);
ATCA_STATUS status;
while ((retries > 0) && (NoBytesRead < 1))
{
//Read data and store in a buffer
status = GetCommState(instance->hMaster, &instance->dcbMaster);
if (status == 0)
printf(" init GetcommState failed\n");
status = ReadFile(instance->hMaster, &SerialBuffer, Byte_count, &NoBytesRead, NULL);
retries--;
}
if (status == FALSE)
{
printf_s("\nError! in ReadFile()\n\n");
return ATCA_TIMEOUT;
}
else
{
printf("Read Success Serial Buffer = %x\n", &SerialBuffer);
*data = SerialBuffer;
//printf("Read Success Data = %x\n", *data);
return ATCA_SUCCESS;
}
}
'''

I am happy to state there is no issue in the code. Thanks to Zhu Song who made a comment about reading the ReadFile remarks. If there is no data to read then readfile will just wait. A check with the logic analyzer showed the write executes but doesn't actually write, hence the read is not actually able to read.
Thanks to everyone who commented

According to ReadFile:
The ReadFile function returns when one of the following conditions occur:
The number of bytes requested is read.
A write operation completes on the write end of the pipe.
An asynchronous handle is being used and the read is occurring asynchronously.
An error occurs.
To cancel all pending asynchronous I/O operations, use either:
CancelIo—this function only cancels operations issued by the calling thread for the specified file handle.
CancelIoEx—this function cancels all operations issued by the threads for the specified file handle.

Related

Capture output from console program with overlapping and events

I know lots of similar questions on this topic have been asked before but so far I have been unable to find a solution that actually works. I want to start a console program from my program and capture its output. My implementation should be in a way that is compatible with WaitForMultipleObjects(), i.e. I want to get notified whenever there is new data to read in the pipe.
My implementation is based on this example from MSDN. However, I had to modify it a little because I need overlapped I/O in order to be able to wait for ReadFile() to finish. So I'm using named pipes created using Dave Hart's MyCreatePipeEx() function from here.
This is my actual code. I have removed error checks for readability reasons.
HANDLE hReadEvent;
HANDLE hStdIn_Rd, hStdIn_Wr;
HANDLE hStdOut_Rd, hStdOut_Wr;
SECURITY_ATTRIBUTES saAttr;
PROCESS_INFORMATION piProcInfo;
STARTUPINFO siStartInfo;
OVERLAPPED ovl;
HANDLE hEvt[2];
DWORD mask, gotbytes;
BYTE buf[4097];
saAttr.nLength = sizeof(SECURITY_ATTRIBUTES);
saAttr.bInheritHandle = TRUE;
saAttr.lpSecurityDescriptor = NULL;
MyCreatePipeEx(&hStdOut_Rd, &hStdOut_Wr, &saAttr, 0, FILE_FLAG_OVERLAPPED, FILE_FLAG_OVERLAPPED);
MyCreatePipeEx(&hStdIn_Rd, &hStdIn_Wr, &saAttr, 0, FILE_FLAG_OVERLAPPED, FILE_FLAG_OVERLAPPED);
SetHandleInformation(hStdOut_Rd, HANDLE_FLAG_INHERIT, 0);
SetHandleInformation(hStdIn_Wr, HANDLE_FLAG_INHERIT, 0);
memset(&piProcInfo, 0, sizeof(PROCESS_INFORMATION));
memset(&siStartInfo, 0, sizeof(STARTUPINFO));
siStartInfo.cb = sizeof(STARTUPINFO);
siStartInfo.hStdError = hStdOut_Wr;
siStartInfo.hStdOutput = hStdOut_Wr;
siStartInfo.hStdInput = hStdIn_Rd;
siStartInfo.dwFlags |= STARTF_USESTDHANDLES;
CreateProcess(NULL, "test.exe", NULL, NULL, TRUE, 0, NULL, NULL, &siStartInfo, &piProcInfo);
hReadEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
for(;;) {
int i = 0;
hEvt[i++] = piProcInfo.hProcess;
memset(&ovl, 0, sizeof(OVERLAPPED));
ovl.hEvent = hReadEvent;
if(!ReadFile(hStdOut_Rd, buf, 4096, &gotbytes, &ovl)) {
if(GetLastError() == ERROR_IO_PENDING) hEvt[i++] = hReadEvent;
} else {
buf[gotbytes] = 0;
printf("%s", buf);
}
mask = WaitForMultipleObjects(i, hEvt, FALSE, INFINITE);
if(mask == WAIT_OBJECT_0 + 1) {
if(GetOverlappedResult(hStdOut_Rd, &ovl, &gotbytes, FALSE)) {
buf[gotbytes] = 0;
printf("%s", buf);
}
} else if(mask == WAIT_OBJECT_0) {
break;
}
}
The problem with this code is the following: As you can see, I'm reading in chunks of 4kb using ReadFile() because I obviously don't know how much data the external program test.exe will output. Doing it this way was suggested here:
To read a variable amount of data from the client process just issue
read requests of whatever size you find convenient and be prepared to
handle read events that are shorter than you requested. Don't
interpret a short but non-zero length as EOF. Keep issuing read
requests until you get a zero length read or an error.
However, this doesn't work. The event object passed to ReadFile() as part of the OVERLAPPED structure will only trigger once there are 4kb in the buffer. If the external program just prints "Hello", the event won't trigger at all. There need to be 4kb in the buffer for hReadEvent to actually trigger.
So I thought I should read byte by byte instead and modified my program to use ReadFile() like this:
if(!ReadFile(hStdOut_Rd, buf, 1, &gotbytes, &ovl)) {
However, this doesn't work either. If I do it like this, the read event is not triggered at all which is really confusing me. When using 4096 bytes, the event does indeed trigger as soon as there are 4096 bytes in the pipe, but when using 1 byte it doesn't work at all.
So how am I supposed to solve this? I'm pretty much out of ideas here. Is there no way to have the ReadFile() event trigger whenever there is some new data in the pipe? Can't be that difficult, can it?
Just for the record, while there are some problems with my code (see discussion in comments below the OP), the general problem is that it's not really possible to capture the output of arbitrary external programs because they will typically use block buffering when their output is redirected to a pipe, which means that output will only arrive at the capturing program once that buffer is flushed so real time capturing is not really possible.
Some workarounds have been suggested though:
1) (Windows) Here is a workaround that uses GetConsoleScreenBuffer() to capture the output from arbitrary console programs but it currently only supports one console page of output.
2) (Linux) On Linux, it's apparently possible to use pseudo-terminals to force the external program to use unbuffered output. Here is an example.

WaitForSingleObject() gets signalled without occurrence of awaited event

I'm writing a Class for COM ports in C++ using win-api. Righ now I test the functionality on RS232 with connected Rx and Tx pins.
I've encountered somewhat weird problem. I use separate thread for reading from the COM port. Within the thread I use SetCommMask, WaitCommEvent and WaitForSingleObject to wait for char arrival into buffer. However the WaitForSingleObject tends to exit without actually receiving any chars.
I assumed this was caused by wrong use of mentioned functions, but then I discovered, that premature exit does not occur every time (first time always works as intended).
In second go the thread enters the waiting state and exits a while later proceeding to ReadFile, where it waits indefinitely, because the buffer is empty, no data is to be sent and Total timeout is not used.
I've already been advised to simply use ReadFile and process only the data I acquire, however I use another thread to check if the communication channel has been disconnected and right now I need to distinguish between waiting for data and reading data.
Calling ClearCommError to check input buffer with ReadFile is not an option, because in such a case InQue is always 0. Therefore I cannot tell whether ReadFile is actually reading or waiting.
//following code runs in separate thread
DWORD dwEventMask1, dwEventMask2, LastError, Status;
OVERLAPPED Overlapped; HANDLE Serial_Port_Handle;
std::string stringBuffer("");
const size_t ReadBufferLength = 256;
char tempBuffer[ReadBufferLength];
GetCommMask(Serial_Port_Handle, &dwEventMask1);
if (dwEventMask1) // Before starting the thread I check the state of Input Buffer with GetCommError().
{ // If Buffer is not empty, CommMask is set to 0 signaling there is no need for waiting.
Overlapped.hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
//wait for comm event
if (!WaitCommEvent(Serial_Port_Handle, &dwEventMask1, &Overlapped))
{
if ((LastError = GetLastError()) == ERROR_IO_PENDING)
{
Waiting = true; //signal bool for synchronization purposes
if ((Status = WaitForSingleObject(Overlapped.hEvent, INFINITE)) == WAIT_OBJECT_0)
{
GetCommMask(Serial_Port_Handle, &dwEventMask2);
Waiting = false;
CloseHandle(Overlapped.hEvent);
// I close handle and set all members of Overlapped struct to 0
}
if (dwEventMask2 !== dwEventMask1) // check if wait have not exited because of SetCommMast()
return;
}
}
}
do // this loop reads from input buffer until empty
{
Overlapped.hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);//set up read overlapped operation
if (ReadFile(Serial_Port_Handle, tempBuffer, ReadBufferLength - 1, &NoBytesRead, &Overlapped)) //start read opperation
{ //Read operation done on 1 go
GetOverlappedResult(Serial_Port_Handle, &Overlapped, &NoBytesRead, FALSE); //get NoBytesRead
CloseHandle(Overlapped.hEvent)
stringBuffer.append(tempBuffer, (size_t) NoBytesRead); // save read data
}
else if ((LastError = GetLastError()) == ERROR_IO_PENDING) //operation not yet complete
{
GetOverlappedResult(Serial_Port_Handle, &Overlapped, &NoBytesRead, TRUE); // wait for completion
stringBuffer.append(tempBuffer, (size_t)NoBytesRead);
}
else
{
CloseHandle(Overlapped.hEvent)
return;
}
} while ((NoBytesRead == (ReadBufferLength - 1)) && NoBytesRead);
// Loop runs while tempBuffer's full capacity is used.
// I realize that since I don't use Total Timeout there is a possibility
// of the loop getting stuck. If you can suggest other solution than using
// Total Timeout or use GetCommError() to get input buffer state, please do so.
return;
This code is somewhat simplified (checking for return values, ets.).
1) Have any of you experienced such a behaviour?
2) I use OVERLAPPED operations in the code. After the operation exits I always use CloseHandle and reinitialize the OVERLAPPED structure before using it for another operation. Is this correct or is resetting the structure sufficient?
It is bad logic as a whole. For example, there are the following issues.
CreateEvent/CloseHandle should not executed at every ReadFile/WaitCommEvent.
The usage of GetCommMask/WaitCommEvent is also wrong.
The read data size specified in ReadFile is fixed regardless of the situation.
It also includes the comment on #RbMm.
You may want to redesign your program with reference to the following articles and source code:
Synchronization and Overlapped Input and Output
Serial Communications
bmo/mttty
In Addition:
I did not notice that a file handle (not an event handle) was specified for WaitForSingleObject, as pointed out by #Rita Han.
The biggest problem is that.
However, the situation that it is better to redesign has not changed.
There is no description of WaitCommEvent and Overlapped for it in the source of #Rita Han's answer. Also, the read data size is fixed in ReadFile.
On the other hand:
Although it does not occur in the source code of the question, it is possible for WaitCommEvent/WaitForSingleObject to generate an event that is not specified in SetCommMask.
While the WaitCommEvent is waiting for completion, change the event mask with SetCommMask.
Remarks - WaitCommEvent function
If a process attempts to change the device handle's event mask by using the SetCommMask function while an overlapped WaitCommEvent operation is in progress, WaitCommEvent returns immediately. The variable pointed to by the lpEvtMask parameter is set to zero.
While WaitCommEvent is waiting for completion, call WaitCommEvent multiple times using the same Overlapped structure.
Synchronization and Overlapped Input and Output
When performing multiple simultaneous overlapped operations on a single thread, the calling thread must specify an OVERLAPPED structure for each operation. Each OVERLAPPED structure must specify a handle to a different manual-reset event object.
A thread should not reuse an event with the assumption that the event will be signaled only by that thread's overlapped operation. An event is signaled on the same thread as the overlapped operation that is completing. Using the same event on multiple threads can lead to a race condition in which the event is signaled correctly for the thread whose operation completes first and prematurely for other threads using that event.
The document is described as above, but depending on the device driver/vendor, the WaitCommEvent that is called later ends with an error, and the WaitCommEvent waiting for completion is lpEvtMask return with zero (as in SetCommMask).
For multiple Overlapped structure variables:
A common programming know-how is that using a single variable for multiple purposes is prone to bugs.
If you are designing in asynchronous and/or multi-threading, it is better to prepare at least three Overlapped structure variables for ReadFile, WriteFile, WaitCommEvent.
About starting ReadFile regardless of the state of the input buffer:
This is about calling ReadFile with a fixed length of 256 bytes without acquiring the size of the received data in the input buffer of the device driver.
In fact, even if all the data arrives, if it is less than 256 bytes, it will always be delayed until a 256 byte receive timeout occurs.
For example, the loop reads one byte at a time until a timeout error occurs, which means the end of the received data (1-byte read timeout would have no impact).
Or, as answered in the previous article, use ClearCommError to obtain the size of the data stored in the device driver's input buffer, and call ReadFile specifying that size.
There will be no problem with the application-side buffer handling you are explaining.
About the behavior of WaiCommEvent when calling SetCommMask:
It may depend on the device driver you are using.
However the WaitForSingleObject tends to exit without actually
receiving any chars.
Wait for the event handle instead of serial device handle.
I got it working. The following is my code example you can have a try:
DWORD errCode = 0;
BOOL result = false;
HANDLE serialDeviceHdl = CreateFile(L"COM8", GENERIC_READ | GENERIC_WRITE, 0, NULL, OPEN_EXISTING, FILE_FLAG_OVERLAPPED, NULL);
if (!serialDeviceHdl)
{
errCode = GetLastError();
cout << "Open device failed. Error code: " << errCode << endl;
return 0;
}
OVERLAPPED overlappedForWrite = {};
overlappedForWrite.hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
DWORD writenSize = 0;
result = WriteFile(serialDeviceHdl, "hello", 5, &writenSize, &overlappedForWrite);
if (FALSE == result)
{
errCode = GetLastError();
if (ERROR_IO_PENDING == errCode)
{
cout << "Overlapped I/O operation is in progress." << endl;
}
else
{
cout << "Write to device failed. Error code: " << errCode << endl;
}
}
DWORD returnValue = WaitForSingleObject(overlappedForWrite.hEvent, INFINITE);
if (WAIT_OBJECT_0 == returnValue)
{
cout << "The state of the specified object is signaled." << endl;
}
else
{
cout << "Wait for single object failed. Error code: " << returnValue << endl;
}
CHAR readBuf[5];
DWORD readSize = 0;
OVERLAPPED overlappedForRead = {};
overlappedForRead.hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
result = ReadFile(serialDeviceHdl, readBuf, 5, &readSize, &overlappedForRead);
if (FALSE == result)
{
errCode = GetLastError();
if (ERROR_IO_PENDING == errCode)
{
cout << "Overlapped I/O operation is in progress." << endl;
}
else
{
cout << "Write to device failed. Error code: " << errCode << endl;
}
}
returnValue = WaitForSingleObject(overlappedForRead.hEvent, INFINITE);
if (WAIT_OBJECT_0 == returnValue)
{
cout << "The state of the specified object is signaled." << endl;
}
else
{
cout << "Wait for single object failed. Error code: " << returnValue << endl;
}

Using IRPs for I/O on device object returned by IoGetDeviceObjectPointer()

Can one use IoCallDriver() with an IRP created by IoBuildAsynchronousFsdRequest() on a device object returned by IoGetDeviceObjectPointer()? What I have currently fails with blue screen (BSOD) 0x7E (unhandled exception), which when caught shows an Access Violation (0xc0000005). Same code worked when the device was stacked (using the device object returned by IoAttachDeviceToDeviceStack()).
So what I have is about the following:
status = IoGetDeviceObjectPointer(&device_name, FILE_ALL_ACCESS, &FileObject, &windows_device);
if (!NT_SUCCESS(status)) {
return -1;
}
offset.QuadPart = 0;
newIrp = IoBuildAsynchronousFsdRequest(io, windows_device, buffer, 4096, &offset, &io_stat);
if (newIrp == NULL) {
return -1;
}
IoSetCompletionRoutine(newIrp, DrbdIoCompletion, bio, TRUE, TRUE, TRUE);
status = ObReferenceObjectByPointer(newIrp->Tail.Overlay.Thread, THREAD_ALL_ACCESS, NULL, KernelMode);
if (!NT_SUCCESS(status)) {
return -1;
}
status = IoCallDriver(bio->bi_bdev->windows_device, newIrp);
if (!NT_SUCCESS(status)) {
return -1;
}
return 0;
device_name is \Device\HarddiskVolume7 which exists according to WinObj.exe .
buffer has enough space and is read/writable. offset and io_stat are on stack (also tried with heap, didn't help). When catching the exception (SEH exception) it doesn't blue screen but shows an access violation as reason for the exception. io is IRP_MJ_READ.
Do I miss something obvious? Is it in general better to use IRPs than the ZwCreateFile / ZwReadFile / ZwWriteFile API (which would be an option, but isn't that slower?)? I also tried a ZwCreateFile to have an extra reference, but this also didn't help.
Thanks for any insights.
you make in this code how minimum 2 critical errors.
can I ask - from which file you try read (or write) data ? from
FileObject you say ? but how file system driver, which will handle
this request know this ? you not pass any file object to newIrp.
look for IoBuildAsynchronousFsdRequest - it have no file object
parameter (and impossible get file object from device object - only
visa versa - because on device can be multiple files open). so it
and can not be filled by this api in newIrp. you must setup it
yourself:
PIO_STACK_LOCATION irpSp = IoGetNextIrpStackLocation( newIrp );
irpSp->FileObject = FileObject;
I guess bug was exactly when file system try access FileObject
from irp which is 0 in your case. also read docs for
IRP_MJ_READ - IrpSp->FileObject -
Pointer to the file object that is associated with DeviceObject
you pass I guess local variables io_stat (and offset) to
IoBuildAsynchronousFsdRequest. as result io_stat must be valid
until newIrp is completed - I/O subsystem write final result to it
when operation completed. but you not wait in function until request
will be completed (in case STATUS_PENDING returned) but just exit
from function. as result later I/O subsystem, if operation completed
asynchronous, write data to arbitrary address &io_stat (it became
arbitrary just after you exit from function). so you need or check
for STATUS_PENDING returned and wait in this case (have actually
synchronous io request). but more logical use
IoBuildSynchronousFsdRequest in this case. or allocate io_stat
not from stack, but say in your object which correspond to file. in
this case you can not have more than single io request with this
object at time. or if you want exactly asynchronous I/O - you can do
next trick - newIrp->UserIosb = &newIrp->IoStatus. as result you
iosb always will be valid for newIrp. and actual operation status
you check/use in DrbdIoCompletion
also can you explain (not for me - for self) next code line ?:
status = ObReferenceObjectByPointer(newIrp->Tail.Overlay.Thread, THREAD_ALL_ACCESS, NULL, KernelMode);
who and where dereference thread and what sense in this ?
Can one use ...
we can use all, but with condition - we understand what we doing and deep understand system internally.
Is it in general better to use IRPs than the ZwCreateFile / ZwReadFile
/ ZwWriteFile API
for performance - yes, better. but this require more code and more complex code compare api calls. and require more knowledge. also if you know that previous mode is kernel mode - you can use NtCreateFile, NtWriteFile, NtReadFile - this of course will be bit slow (need every time reference file object by handle) but more faster compare Zw version
Just wanted to add that the ObReferenceObjectByPointer is needed
because the IRP references the current thread which may exit before
the request is completed. It is dereferenced in the Completion
Routine. Also as a hint the completion routine must return
STATUS_MORE_PROCESSING_REQUIRED if it frees the IRP (took me several
days to figure that out).
here you make again several mistakes. how i understand you in completion routine do next:
IoFreeIrp(Irp);
return StopCompletion;
but call simply call IoFreeIrp here is error - resource leak. i advice you check (DbgPrint) Irp->MdlAddress at this point. if you read data from file system object and request completed asynchronous - file system always allocate Mdl for access user buffer in arbitrary context. now question - who free this Mdl ? IoFreeIrp - simply free Irp memory - nothing more. you do this yourself ? doubt. but Irp is complex object, which internally hold many resources. as result need not only free it memory but call "destructor" for it. this "destructor" is IofCompleteRequest. when you return StopCompletion (=STATUS_MORE_PROCESSING_REQUIRED) you break this destructor at very begin. but you must latter again call IofCompleteRequest for continue Irp (and it resources) correct destroy.
about referencing Tail.Overlay.Thread - what you doing - have no sense:
It is dereferenced in the Completion Routine.
but IofCompleteRequest access Tail.Overlay.Thread after it
call your completion routine (and if you not return
StopCompletion). as result your reference/dereference thread lost
sense - because you deference it too early, before system
actually access it.
also if you return StopCompletion and not more call
IofCompleteRequest for this Irp - system not access
Tail.Overlay.Thread at all. and you not need reference it in this
case.
and exist else one reason, why reference thread is senseless. system
access Tail.Overlay.Thread only for insert Apc to him - for call
final part (IopCompleteRequest) of Irp destruction in original
thread context. really this need only for user mode Irp's requests,
where buffers and iosb located in user mode and valid only in
context of process (original thread ). but if thread is terminated -
call of KeInsertQueueApc fail - system not let insert apc to
died thread. as result IopCompleteRequest will be not called and
resources not freed.
so you or dereference Tail.Overlay.Thread too early or you not need do this at all. and reference for died thread anyway not help. in all case what you doing is error.
you can try do next here:
PETHREAD Thread = Irp->Tail.Overlay.Thread;
IofCompleteRequest(Irp, IO_NO_INCREMENT);// here Thread will be referenced
ObfDereferenceObject(Thread);
return StopCompletion;
A second call to IofCompleteRequest causes the I/O manager to resume calling the IRP's completion. here io manager and access Tail.Overlay.Thread insert Apc to him. and finally you call ObfDereferenceObject(Thread); already after system access it and return StopCompletion for break first call to IofCompleteRequest. look like correct but.. if thread already terminated, how i explain in 3 this will be error, because KeInsertQueueApc fail. for extended test - call IofCallDriver from separate thread and just exit from it. and in completion run next code:
PETHREAD Thread = Irp->Tail.Overlay.Thread;
if (PsIsThreadTerminating(Thread))
{
DbgPrint("ThreadTerminating\n");
if (PKAPC Apc = (PKAPC)ExAllocatePool(NonPagedPool, sizeof(KAPC)))
{
KeInitializeApc(Apc, Thread, 0, KernelRoutine, 0, 0, KernelMode, 0);
if (!KeInsertQueueApc(Apc, 0, 0, IO_NO_INCREMENT))
{
DbgPrint("!KeInsertQueueApc\n");
ExFreePool(Apc);
}
}
}
PMDL MdlAddress = Irp->MdlAddress;
IofCompleteRequest(Irp, IO_NO_INCREMENT);
ObfDereferenceObject(Thread);
if (MdlAddress == Irp->MdlAddress)
{
// IopCompleteRequest not called due KeInsertQueueApc fail
DbgPrint("!!!!!!!!!!!\n");
IoFreeMdl(MdlAddress);
IoFreeIrp(Irp);
}
return StopCompletion;
//---------------
VOID KernelRoutine (PKAPC Apc,PKNORMAL_ROUTINE *,PVOID *,PVOID *,PVOID *)
{
DbgPrint("KernelRoutine(%p)\n", Apc);
ExFreePool(Apc);
}
and you must got next debug output:
ThreadTerminating
!KeInsertQueueApc
!!!!!!!!!!!
and KernelRoutine will be not called (like and IopCompleteRequest) - no print from it.
so what is correct solution ? this of course not documented anywhere, but based on deep internal understand. you not need reference original thread. you need do next:
Irp->Tail.Overlay.Thread = KeGetCurrentThread();
return ContinueCompletion;
you can safe change Tail.Overlay.Thread - if you have no any pointers valid only in original process context. this is true for kernel mode requests - all your buffers in kernel mode and valid in any context. and of course you not need break Irp destruction but continue it. for correct free mdl and all irp resources. and finally system call IoFreeIrp for you.
and again for iosb pointer. how i say pass local variable address, if you exit from function before irp completed (and this iosb accessed) is error. if you break Irp destruction, iosb will be not accessed of course, but in this case much better pass 0 pointer as iosb. (if you latter something change and iosb pointer will be accessed - will be the worst error - arbitrary memory corrupted - with unpredictable effect. and research crash of this will be very-very hard). but if you completion routine - you not need separate iosb at all - you have irp in completion and can direct access it internal iosb - for what you need else one ? so the best solution will be do next:
Irp->UserIosb = &Irp->IoStatus;
full correct example how read file asynchronous:
NTSTATUS DemoCompletion (PDEVICE_OBJECT /*DeviceObject*/, PIRP Irp, BIO* bio)
{
DbgPrint("DemoCompletion(p=%x mdl=%p)\n", Irp->PendingReturned, Irp->MdlAddress);
bio->CheckResult(Irp->IoStatus.Status, Irp->IoStatus.Information);
bio->Release();
Irp->Tail.Overlay.Thread = KeGetCurrentThread();
return ContinueCompletion;
}
VOID DoTest (PVOID buf)
{
PFILE_OBJECT FileObject;
NTSTATUS status;
UNICODE_STRING ObjectName = RTL_CONSTANT_STRING(L"\\Device\\HarddiskVolume2");
OBJECT_ATTRIBUTES oa = { sizeof(oa), 0, &ObjectName, OBJ_CASE_INSENSITIVE };
if (0 <= (status = GetDeviceObjectPointer(&oa, &FileObject)))
{
status = STATUS_INSUFFICIENT_RESOURCES;
if (BIO* bio = new BIO(FileObject))
{
if (buf = bio->AllocBuffer(PAGE_SIZE))
{
LARGE_INTEGER ByteOffset = {};
PDEVICE_OBJECT DeviceObject = IoGetRelatedDeviceObject(FileObject);
if (PIRP Irp = IoBuildAsynchronousFsdRequest(IRP_MJ_READ, DeviceObject, buf, PAGE_SIZE, &ByteOffset, 0))
{
Irp->UserIosb = &Irp->IoStatus;
Irp->Tail.Overlay.Thread = 0;
PIO_STACK_LOCATION IrpSp = IoGetNextIrpStackLocation(Irp);
IrpSp->FileObject = FileObject;
bio->AddRef();
IrpSp->CompletionRoutine = (PIO_COMPLETION_ROUTINE)DemoCompletion;
IrpSp->Context = bio;
IrpSp->Control = SL_INVOKE_ON_CANCEL|SL_INVOKE_ON_ERROR|SL_INVOKE_ON_SUCCESS;
status = IofCallDriver(DeviceObject, Irp);
}
}
bio->Release();
}
ObfDereferenceObject(FileObject);
}
DbgPrint("DoTest=%x\n", status);
}
struct BIO
{
PVOID Buffer;
PFILE_OBJECT FileObject;
LONG dwRef;
void AddRef()
{
InterlockedIncrement(&dwRef);
}
void Release()
{
if (!InterlockedDecrement(&dwRef))
{
delete this;
}
}
void* operator new(size_t cb)
{
return ExAllocatePool(PagedPool, cb);
}
void operator delete(void* p)
{
ExFreePool(p);
}
BIO(PFILE_OBJECT FileObject) : FileObject(FileObject), Buffer(0), dwRef(1)
{
DbgPrint("%s<%p>(%p)\n", __FUNCTION__, this, FileObject);
ObfReferenceObject(FileObject);
}
~BIO()
{
if (Buffer)
{
ExFreePool(Buffer);
}
ObfDereferenceObject(FileObject);
DbgPrint("%s<%p>(%p)\n", __FUNCTION__, this, FileObject);
}
PVOID AllocBuffer(ULONG NumberOfBytes)
{
return Buffer = ExAllocatePool(PagedPool, NumberOfBytes);
}
void CheckResult(NTSTATUS status, ULONG_PTR Information)
{
DbgPrint("CheckResult:status = %x, info = %p\n", status, Information);
if (0 <= status)
{
if (ULONG_PTR cb = min(16, Information))
{
char buf[64], *sz = buf;
PBYTE pb = (PBYTE)Buffer;
do sz += sprintf(sz, "%02x ", *pb++); while (--cb); sz[-1]= '\n';
DbgPrint(buf);
}
}
}
};
NTSTATUS GetDeviceObjectPointer(POBJECT_ATTRIBUTES poa, PFILE_OBJECT *FileObject )
{
HANDLE hFile;
IO_STATUS_BLOCK iosb;
NTSTATUS status = IoCreateFile(&hFile, FILE_READ_DATA, poa, &iosb, 0, 0,
FILE_SHARE_VALID_FLAGS, FILE_OPEN, FILE_NO_INTERMEDIATE_BUFFERING, 0, 0, CreateFileTypeNone, 0, 0);
if (0 <= (status))
{
status = ObReferenceObjectByHandle(hFile, 0, *IoFileObjectType, KernelMode, (void**)FileObject, 0);
NtClose(hFile);
}
return status;
}
and output:
BIO::BIO<FFFFC000024D4870>(FFFFE00001BAAB70)
DoTest=103
DemoCompletion(p=1 mdl=FFFFE0000200EE70)
CheckResult:status = 0, info = 0000000000001000
eb 52 90 4e 54 46 53 20 20 20 20 00 02 08 00 00
BIO::~BIO<FFFFC000024D4870>(FFFFE00001BAAB70)
the eb 52 90 4e 54 46 53 read ok

win32 using while loop with ReadFile/WriteFile in C to write from one program to a new one

I've written a program that reads text from one file and copies it to a new file. Using a while loop and the ReadFile/Writefile functions, my program works...but my program won't stop running unless I force stop it. I'm guessing that I'm not closing my handles properly or that my while loop may be set up wrong. Once I force stop my program, the file is successfully copied over to the new location with a new name.
int n = 0;
while(n=ReadFile(hFileSource, buffer, 23, &dwBytesRead, NULL)){
WriteFile(hFileNew, buffer, dwBytesRead, &dwBytesWritten, NULL);
}
CloseHandle(hFileSource);
CloseHandle(hFileNew);
return 0;
You're not correctly testing for the end-of-file. ReadFile doesn't return failure for EOF, it returns success but with 0 bytes read. To correctly check for EOF:
while (ReadFile(hFileSource, buffer, 23, &dwBytesRead, NULL))
{
if (dwBytesRead == 0)
break;
// write data etc
}
Is there any reason you're only reading/writing 23 bytes at a time? This will be rather inefficient.

Duplex named pipe hangs on a certain write

I have a C++ pipe server app and a C# pipe client app communicating via Windows named pipe (duplex, message mode, wait/blocking in separate read thread).
It all works fine (both sending and receiving data via the pipe) until I try and write to the pipe from the client in response to a forms 'textchanged' event. When I do this, the client hangs on the pipe write call (or flush call if autoflush is off). Breaking into the server app reveals it's also waiting on the pipe ReadFile call and not returning.
I tried running the client write on another thread -- same result.
Suspect some sort of deadlock or race condition but can't see where... don't think I'm writing to the pipe simultaneously.
Update1: tried pipes in byte mode instead of message mode - same lockup.
Update2: Strangely, if (and only if) I pump lots of data from the server to the client, it cures the lockup!?
Server code:
DWORD ReadMsg(char* aBuff, int aBuffLen, int& aBytesRead)
{
DWORD byteCount;
if (ReadFile(mPipe, aBuff, aBuffLen, &byteCount, NULL))
{
aBytesRead = (int)byteCount;
aBuff[byteCount] = 0;
return ERROR_SUCCESS;
}
return GetLastError();
}
DWORD SendMsg(const char* aBuff, unsigned int aBuffLen)
{
DWORD byteCount;
if (WriteFile(mPipe, aBuff, aBuffLen, &byteCount, NULL))
{
return ERROR_SUCCESS;
}
mClientConnected = false;
return GetLastError();
}
DWORD CommsThread()
{
while (1)
{
std::string fullPipeName = std::string("\\\\.\\pipe\\") + mPipeName;
mPipe = CreateNamedPipeA(fullPipeName.c_str(),
PIPE_ACCESS_DUPLEX,
PIPE_TYPE_MESSAGE | PIPE_READMODE_MESSAGE | PIPE_WAIT,
PIPE_UNLIMITED_INSTANCES,
KTxBuffSize, // output buffer size
KRxBuffSize, // input buffer size
5000, // client time-out ms
NULL); // no security attribute
if (mPipe == INVALID_HANDLE_VALUE)
return 1;
mClientConnected = ConnectNamedPipe(mPipe, NULL) ? TRUE : (GetLastError() == ERROR_PIPE_CONNECTED);
if (!mClientConnected)
return 1;
char rxBuff[KRxBuffSize+1];
DWORD error=0;
while (mClientConnected)
{
Sleep(1);
int bytesRead = 0;
error = ReadMsg(rxBuff, KRxBuffSize, bytesRead);
if (error == ERROR_SUCCESS)
{
rxBuff[bytesRead] = 0; // terminate string.
if (mMsgCallback && bytesRead>0)
mMsgCallback(rxBuff, bytesRead, mCallbackContext);
}
else
{
mClientConnected = false;
}
}
Close();
Sleep(1000);
}
return 0;
}
client code:
public void Start(string aPipeName)
{
mPipeName = aPipeName;
mPipeStream = new NamedPipeClientStream(".", mPipeName, PipeDirection.InOut, PipeOptions.None);
Console.Write("Attempting to connect to pipe...");
mPipeStream.Connect();
Console.WriteLine("Connected to pipe '{0}' ({1} server instances open)", mPipeName, mPipeStream.NumberOfServerInstances);
mPipeStream.ReadMode = PipeTransmissionMode.Message;
mPipeWriter = new StreamWriter(mPipeStream);
mPipeWriter.AutoFlush = true;
mReadThread = new Thread(new ThreadStart(ReadThread));
mReadThread.IsBackground = true;
mReadThread.Start();
if (mConnectionEventCallback != null)
{
mConnectionEventCallback(true);
}
}
private void ReadThread()
{
byte[] buffer = new byte[1024 * 400];
while (true)
{
int len = 0;
do
{
len += mPipeStream.Read(buffer, len, buffer.Length);
} while (len>0 && !mPipeStream.IsMessageComplete);
if (len==0)
{
OnPipeBroken();
return;
}
if (mMessageCallback != null)
{
mMessageCallback(buffer, len);
}
Thread.Sleep(1);
}
}
public void Write(string aMsg)
{
try
{
mPipeWriter.Write(aMsg);
mPipeWriter.Flush();
}
catch (Exception)
{
OnPipeBroken();
}
}
If you are using separate threads you will be unable to read from the pipe at the same time you write to it. For example, if you are doing a blocking read from the pipe then a subsequent blocking write (from a different thread) then the write call will wait/block until the read call has completed and in many cases if this is unexpected behavior your program will become deadlocked.
I have not tested overlapped I/O, but it MAY be able to resolve this issue. However, if you are determined to use synchronous calls then the following models below may help you to solve the problem.
Master/Slave
You could implement a master/slave model in which the client or the server is the master and the other end only responds which is generally what you will find the MSDN examples to be.
In some cases you may find this problematic in the event the slave periodically needs to send data to the master. You must either use an external signaling mechanism (outside of the pipe) or have the master periodically query/poll the slave or you can swap the roles where the client is the master and the server is the slave.
Writer/Reader
You could use a writer/reader model where you use two different pipes. However, you must associate those two pipes somehow if you have multiple clients since each pipe will have a different handle. You could do this by having the client send a unique identifier value on connection to each pipe which would then let the server associate the two pipes. This number could be the current system time or even a unique identifier that is global or local.
Threads
If you are determined to use the synchronous API you can use threads with the master/slave model if you do not want to be blocked while waiting for a message on the slave side. You will however want to lock the reader after it reads a message (or encounters the end of a series of message) then write the response (as the slave should) and finally unlock the reader. You can lock and unlock the reader using locking mechanisms that put the thread to sleep as these would be most efficient.
Security Problem With TCP
The loss going with TCP instead of named pipes is also the biggest possible problem. A TCP stream does not contain any security natively. So if security is a concern you will have to implement that and you have the possibility of creating a security hole since you would have to handle authentication yourself. The named pipe can provide security if you properly set the parameters. Also, to note again more clearly: security is no simple matter and generally you will want to use existing facilities that have been designed to provide it.
I think you may be running into problems with named pipes message mode. In this mode, each write to the kernel pipe handle constitutes a message. This doesn't necessarily correspond with what your application regards a Message to be, and a message may be bigger than your read buffer.
This means that your pipe reading code needs two loops, the inner reading until the current [named pipe] message has been completely received, and the outer looping until your [application level] message has been received.
Your C# client code does have a correct inner loop, reading again if IsMessageComplete is false:
do
{
len += mPipeStream.Read(buffer, len, buffer.Length);
} while (len>0 && !mPipeStream.IsMessageComplete);
Your C++ server code doesn't have such a loop - the equivalent at the Win32 API level is testing for the return code ERROR_MORE_DATA.
My guess is that somehow this is leading to the client waiting for the server to read on one pipe instance, whilst the server is waiting for the client to write on another pipe instance.
It seems to me that what you are trying to do will rather not work as expected.
Some time ago I was trying to do something that looked like your code and got similar results, the pipe just hanged
and it was difficult to establish what had gone wrong.
I would rather suggest to use client in very simple way:
CreateFile
Write request
Read answer
Close pipe.
If you want to have two way communication with clients which are also able to receive unrequested data from server you should
rather implement two servers. This was the workaround I used: here you can find sources.

Resources