Can not read from child process pipe - winapi

I'm desperately trying to create a child process and redirect its output to new pipes and read from those pipes, but I just can't get it to work. I am very new the Win32API, please be nice to me. :)
After having failed on using the Win32API "normally", I created wrappers to focus on finding an error in the logic and/or order of API calls. You can find the interface for the wrappers below. Since most of the methods directly translate to Win32API calls, it should (hopefully) not be an obstacle to answering this question.
I get the same behaviour with using the wrapper classes as I have experienced originally.
I've read a lot of online resources about this topic and one says something different than the other. The one that has been most useful until now was https://msdn.microsoft.com/en-us/library/windows/desktop/ms682499(v=vs.85).aspx, especially this information (emphasis mine):
The parent process uses the opposite ends of these two pipes to write to the child process's input and read from the child process's output. As specified in the STARTUPINFO structure, these handles are also inheritable. However, these handles must not be inherited. Therefore, before creating the child process, the parent process uses the SetHandleInformation function to ensure that the write handle for the child process's standard input and the read handle for the child process's standard input cannot be inherited. For more information, see Pipes.
Before I found this topic and closed the ends that I'm not using from the parent process side, I head ReadFile() blocking forever on the standard output read handle of the child process. Now, it always immediately returns that the pipe is broken.
This is how I create the Pipes and Process:
Popen(const String& command, const String& args,
Bool use_current_pipes = false, Bool merge_stderr = true)
{
Bool ok = true;
_error = 0;
ZeroMemory(&_pi, sizeof(_pi));
STARTUPINFO si;
ZeroMemory(&si, sizeof(si));
si.cb = sizeof(si);
if (!use_current_pipes) {
// Create pipes for standard input, output and error.
_stdin = Pipe(true);
_stdout = Pipe(true);
if (_stdout && merge_stderr)
_stderr = _stdout.Duplicate();
else
_stderr = Pipe(true);
if (_stdin && _stdout && _stderr) {
_stdin.w.SetInheritable(false);
_stderr.r.SetInheritable(false);
_stdout.r.SetInheritable(false);
si.hStdInput = _stdin.r.Get();
si.hStdOutput = _stdout.w.Get();
si.hStdError = _stderr.w.Get();
si.dwFlags |= STARTF_USESTDHANDLES;
}
else {
ok = false;
}
}
else {
si.hStdInput = GetStdHandle(STD_INPUT_HANDLE);
si.hStdOutput = GetStdHandle(STD_OUTPUT_HANDLE);
si.hStdError = GetStdHandle(STD_ERROR_HANDLE);
si.dwFlags |= STARTF_USESTDHANDLES;
}
// Create the process. Enclose the actual command in quotes.
ok = ok && CreateProcess(
nullptr, // command might contain whitespace, pass it quoted in arg 2 instead.
AutoString("\"" + command + "\" " + args),
nullptr, // Process handle not inheritable
nullptr, // Thread handle not inheritable
true, // handles are inherited
0, // No creation flags
nullptr, // Use parent's environment block
nullptr, // Use parent's starting directory
&si, // Pointer to STARTUPINFO
&_pi); // Pointer to PROCESS_INFORMATION
// Something went wrong? Well, bad.
if (!ok) {
_error = GetLastError();
}
// Close the handles that have been inherited by the child process
// and to which we don't need access to, otherwise they will not
// close when the child exits.
_stdin.r.Close();
_stdout.w.Close();
_stderr.w.Close();
}
And this is how I read from the standard output (_stdout.r):
UInt Read(UInt num_bytes, char* buffer) {
if (!_stdout.r) return 0;
DWORD bytes_read = 0;
if (!ReadFile(_stdout.r.Get(), buffer, num_bytes - 1, &bytes_read, nullptr)) {
_error = GetLastError();
ConsoleOut("[ERROR]: ReadFile() : " + String::IntToString((Int32) _error));
if (_error == ERROR_BROKEN_PIPE) {
ConsoleOut("No Wait, the Pipe is just broken.");
_error = 0; // that's fine
}
return 0;
}
buffer[bytes_read] = '\0';
return bytes_read;
}
When I comment out the last lines of the Popen constructor (closing the pipe handles that are not used from the parent process) ReadFile() blocks forever. With these lines enabled, the Pipe is always immediately broken (the child process exits pretty quickly).
Question
Can someone see what is wrong in my code/logic?
If not, I would already appreciate if there is a complete working example of opening a child process and reading its output
Wrapper Interface
struct Handle {
HANDLE h;
explicit Handle();
explicit Handle(HANDLE h);
Handle(Handle&& other);
Handle& operator = (Handle&& other);
~Handle();
void Close();
HANDLE Get();
HANDLE Release();
Handle Duplicate(DWORD options = DUPLICATE_SAME_ACCESS, HANDLE src_proc = nullptr, HANDLE dst_proc = nullptr) const;
DWORD GetInfo() const; // uses GetHandleInformation
void SetInheritable(bool inheritable) const; // uses SetHandleInformation
bool GetInheritable() const;
operator bool() const;
explicit Handle(const Handle&) = delete;
Handle* operator = (const Handle&) = delete;
};
struct Pipe {
Handle r, w;
DWORD error;
explicit Pipe();
explicit Pipe(bool inheritable);
Pipe(Pipe&& other);
Pipe& operator = (Pipe&& other);
~Pipe();
void Close();
Pipe Duplicate(DWORD options = DUPLICATE_SAME_ACCESS, HANDLE src_proc = nullptr, HANDLE dst_proc = nullptr) const;
operator bool() const;
explicit Pipe(const Pipe&) = delete;
Pipe* operator = (const Pipe&) = delete;
};

Without using either threads or overlapped I/O, you risk deadlock. The child process could be trying to read from its stdin or waiting for space in its stdout buffer so it can write, you cannot tell which, and when you choose wrong, you get the observed behavior. The blocking read on the child's output means you guessed wrong, and it is actually waiting for input.
Read Raymond Chen's blog article Be careful when redirecting both a process's stdin and stdout to pipes, for you can easily deadlock which I also linked in your earlier question today. It specifically calls out the horrible brokenness in the very same sample you linked in your question.

Related

Do I have to specify handles to be inherited with PROC_THREAD_ATTRIBUTE_LIST when using CreateProcess in threads?

I'm writing a program that spawns multiple child processes simultaneously and communicates to each of them by redirecting their stdin/stdout to anonymous pipes (as described here).
This is my actual code adapted from above example:
HANDLE h_child_stdout_r = NULL;
HANDLE h_child_stdout_w = NULL;
HANDLE h_child_stdin_r = NULL;
HANDLE h_child_stdin_w = NULL;
SECURITY_ATTRIBUTES saAttr;
BOOL bSuccess = TRUE;
// Set the bInheritHandle flag so pipe handles are inherited.
saAttr.nLength = sizeof(SECURITY_ATTRIBUTES);
saAttr.bInheritHandle = TRUE;
saAttr.lpSecurityDescriptor = NULL;
// Create a pipe for the child process's STDOUT.
bSuccess &= CreatePipe(&h_child_stdout_r, &h_child_stdout_w, &saAttr, 0);
// Ensure the read handle to the pipe for STDOUT is not inherited.
bSuccess &= SetHandleInformation(h_child_stdout_r, HANDLE_FLAG_INHERIT, 0);
// Create a pipe for the child process's STDIN.
bSuccess &= CreatePipe(&h_child_stdin_r, &h_child_stdin_w, &saAttr, 0);
// Ensure the write handle to the pipe for STDIN is not inherited.
bSuccess &= SetHandleInformation(h_child_stdin_w, HANDLE_FLAG_INHERIT, 0);
if(!bSuccess)
{
throw dlg_exception("Failed to create child process handles");
}
PROCESS_INFORMATION piProcInfo;
STARTUPINFO siStartInfo;
ZeroMemory( &piProcInfo, sizeof(PROCESS_INFORMATION) );
ZeroMemory( &siStartInfo, sizeof(STARTUPINFO) );
// Set up members of the STARTUPINFO structure.
// This structure specifies the STDIN and STDOUT handles for redirection.
siStartInfo.cb = sizeof(STARTUPINFO);
siStartInfo.dwFlags |= STARTF_USESTDHANDLES;
siStartInfo.hStdError = h_child_stdout_w;
siStartInfo.hStdOutput = h_child_stdout_w;
siStartInfo.hStdInput = h_child_stdin_r;
DWORD processFlags = CREATE_NO_WINDOW | CREATE_UNICODE_ENVIRONMENT;
// Create the child process.
bSuccess = CreateProcess(
const_cast<wchar_t *>(utf8::widen(exeName).c_str()), // application name
const_cast<wchar_t *>(utf8::widen(cmd).c_str()), // command line
NULL, // process security attributes
NULL, // primary thread security attributes
TRUE, // handles are inherited
processFlags, // creation flags
NULL, // use parent's environment
NULL, // use parent's current directory
&siStartInfo, // STARTUPINFO pointer
&piProcInfo); // receives PROCESS_INFORMATION
// If an error occurs, exit the application.
if (!bSuccess)
{
string msg = "Create process failed with error code ";
msg.append( std::to_string(GetLastError()) );
throw dlg_exception(msg.c_str());
}
// Close handles to the child process and its primary thread.
// Some applications might keep these handles to monitor the status
// of the child process, for example.
CloseHandle(piProcInfo.hProcess);
CloseHandle(piProcInfo.hThread);
// Close handles to the stdin and stdout pipes no longer needed by the child process.
// If they are not explicitly closed, there is no way to recognize that the child process has ended.
CloseHandle(h_child_stdout_w);
CloseHandle(h_child_stdin_r);
The above function will be called from multiple threads that are responsible for creating child processes and reading their output.
Everything works as expected but just to make sure I started searching about thread-safety of CreateProcess and I came across this article which in short says there will be an issue if multiple calls to CreateProcess are made from different threads because then each child process will inherit ALL handles from the parent process.
Now I have two questions:
1- Will this really be an issue for me? Since I am only redirecting the child process's stdin and stdout to the pipe handles, so I'm not sure if there will be a problem even if all the handles are inherited by the child. The child is merely writing to its own stdout and doesn't care about what handles it has inherited right?
2- I tried inspecting the child processes with Sysinternal's Process Explorer but as far as I could tell only two handles were inherited by each child not all of them.
Here's a screenshot of Process Explorer. As you can see the parent process has created 4 child processes. For each process I created, a new pair of handles would appear in parent process's list of handles (I have highlighted the first two):
As you can see there is four pairs of these, one for each child process.
And for each child process the handle list looks like this:
After reading the article what I was expecting was that the fourth child process would have all the handles listed because it's supposed to inherit all inheritable handles. But it only has two handles like all other childs.
So I'm wondering if I can get away with my current code without doing the PROC_THREAD_ATTRIBUTE_LIST which looks complicated and unfun.

Why is my file with redirected stderr empty? [duplicate]

In the following program I print to the console using two different functions
#include <windows.h>
int main() {
HANDLE h = GetStdHandle(STD_OUTPUT_HANDLE);
DWORD byteswritten;
WriteConsole(h, "WriteConsole", 12, &byteswritten, NULL);
WriteFile(h, "WriteFile", 9, &byteswritten, NULL);
}
If when I execute this program and redirect it's output using a > out.txt or a 1> out.txt nothing gets printed to the console (as expected) but the contents of out.txt are only
WriteFile
What is different between the two that allows calls to WriteFile to be redirected to the file and calls to WriteConsole to go to ... nowhere
Tested with gcc and msvc on windows 10
WriteConsole only works with console screen handles, not files nor pipes.
If you are only writing ASCII content you can use WriteFile for everything.
If you need to write Unicode characters you can use GetConsoleMode to detect the handle type, it fails for everything that is not a console handle.
When doing raw output like this you also have to deal with the BOM if the handle is redirected to a file.
This blog post is a good starting point for dealing with Unicode in the Windows console...
Edit 2021:
Windows 10 now has the ConPTY API (aka pseudo-console), which basically allows any program to act like the console for another program, thus enables capturing output that is directly written to the console.
This renders my original answer obsolete for Windows versions that support ConPTY.
Original answer:
From the reference:
WriteConsole fails if it is used with a standard handle that is
redirected to a file. If an application processes multilingual output
that can be redirected, determine whether the output handle is a
console handle (one method is to call the GetConsoleMode function and
check whether it succeeds). If the handle is a console handle, call
WriteConsole. If the handle is not a console handle, the output is
redirected and you should call WriteFile to perform the I/O.
This is only applicable if you control the source code of the application that you want to redirect. I recently had to redirect output from a closed-source application that unconditionally called WriteConsole() so it could not be redirected normally.
Reading the console screen buffer (as suggested by this answer) prooved to be unreliable, so I used Microsoft Detours library to hook the WriteConsole() API in the target process and call WriteFile() if necessary. Otherwise call the original WriteConsole() function.
I created a hook DLL based on the example of Using Detours:
#include <windows.h>
#include <detours.h>
// Target pointer for the uninstrumented WriteConsoleW API.
//
auto WriteConsoleW_orig = &WriteConsoleW;
// Detour function that replaces the WriteConsoleW API.
//
BOOL WINAPI WriteConsoleW_hooked(
_In_ HANDLE hConsoleOutput,
_In_ const VOID *lpBuffer,
_In_ DWORD nNumberOfCharsToWrite,
_Out_ LPDWORD lpNumberOfCharsWritten,
_Reserved_ LPVOID lpReserved
)
{
// Check if this actually is a console screen buffer handle.
DWORD mode;
if( GetConsoleMode( hConsoleOutput, &mode ) )
{
// Forward to the original WriteConsoleW() function.
return WriteConsoleW_orig( hConsoleOutput, lpBuffer, nNumberOfCharsToWrite, lpNumberOfCharsWritten, lpReserved );
}
else
{
// This is a redirected handle (e. g. a file or a pipe). We multiply with sizeof(WCHAR), because WriteFile()
// expects the number of bytes, but WriteConsoleW() gets passed the number of characters.
BOOL result = WriteFile( hConsoleOutput, lpBuffer, nNumberOfCharsToWrite * sizeof(WCHAR), lpNumberOfCharsWritten, nullptr );
// WriteFile() returns number of bytes written, but WriteConsoleW() has to return the number of characters written.
if( lpNumberOfCharsWritten )
*lpNumberOfCharsWritten /= sizeof(WCHAR);
return result;
}
}
// DllMain function attaches and detaches the WriteConsoleW_hooked detour to the
// WriteConsoleW target function. The WriteConsoleW target function is referred to
// through the WriteConsoleW_orig target pointer.
//
BOOL WINAPI DllMain(HINSTANCE hinst, DWORD dwReason, LPVOID reserved)
{
if (DetourIsHelperProcess()) {
return TRUE;
}
if (dwReason == DLL_PROCESS_ATTACH) {
DetourRestoreAfterWith();
DetourTransactionBegin();
DetourUpdateThread(GetCurrentThread());
DetourAttach(&(PVOID&)WriteConsoleW_orig, WriteConsoleW_hooked);
DetourTransactionCommit();
}
else if (dwReason == DLL_PROCESS_DETACH) {
DetourTransactionBegin();
DetourUpdateThread(GetCurrentThread());
DetourDetach(&(PVOID&)WriteConsoleW_orig, WriteConsoleW_hooked);
DetourTransactionCommit();
}
return TRUE;
}
Note: In the WriteFile() branch I don't write a BOM (byte order mark), because it is not always wanted (e. g. when redirecting to a pipe instead of a file or when appending to an existing file). An application that is using the DLL to redirect process output to a file can simply write the UTF-16 LE BOM on its own before launching the redirected process.
The target process is created using DetourCreateProcessWithDllExW(), specifying the name of our hook DLL as argument for the lpDllName parameter. The other arguments are identical to how you create a redirected process via the CreateProcessW() API. I won't go into detail, because these are all well documented.
The code below can be used to redirect console output if the other party uses WriteConsole. The code reads the output via a hidden console screen buffer. I've written this code to intercept debug output some directshow drivers write to the console. Directshow drivers have the habit of doing things drivers should not do, like writing unwanted logfiles, writing to console and crashing.
// info to redirected console output
typedef struct tagRedConInfo
{
// hidden console
HANDLE hCon;
// old console handles
HANDLE hOldConOut;
HANDLE hOldConErr;
// buffer to read screen content
CHAR_INFO *BufData;
INT BufSize;
//
} TRedConInfo;
//------------------------------------------------------------------------------
// GLOBALS
//------------------------------------------------------------------------------
// initial handles
HANDLE gv_hOldConOut;
HANDLE gv_hOldConErr;
//------------------------------------------------------------------------------
// PROTOTYPES
//------------------------------------------------------------------------------
/* init redirecting the console output */
BOOL Shell_InitRedirectConsole(BOOL,TRedConInfo*);
/* done redirecting the console output */
BOOL Shell_DoneRedirectConsole(TRedConInfo*);
/* read string from hidden console, then clear */
BOOL Shell_ReadRedirectConsole(TRedConInfo*,TCHAR*,INT);
/* clear buffer of hidden console */
BOOL Shell_ClearRedirectConsole(TRedConInfo*);
//------------------------------------------------------------------------------
// IMPLEMENTATIONS
//------------------------------------------------------------------------------
/***************************************/
/* init redirecting the console output */
/***************************************/
BOOL Shell_InitRedirectConsole(BOOL in_SetStdHandles, TRedConInfo *out_RcInfo)
{
/* locals */
HANDLE lv_hCon;
SECURITY_ATTRIBUTES lv_SecAttr;
// preclear structure
memset(out_RcInfo, 0, sizeof(TRedConInfo));
// prepare inheritable handle just in case an api spans an external process
memset(&lv_SecAttr, 0, sizeof(SECURITY_ATTRIBUTES));
lv_SecAttr.nLength = sizeof(SECURITY_ATTRIBUTES);
lv_SecAttr.bInheritHandle = TRUE;
// create hidden console buffer
lv_hCon = CreateConsoleScreenBuffer(
GENERIC_READ|GENERIC_WRITE, FILE_SHARE_READ|FILE_SHARE_WRITE,
&lv_SecAttr, CONSOLE_TEXTMODE_BUFFER, 0);
// failed to create console buffer?
if (lv_hCon == INVALID_HANDLE_VALUE)
return FALSE;
// store
out_RcInfo->hCon = lv_hCon;
// set as standard handles for own process?
if (in_SetStdHandles)
{
// mutex the globals
WaitForGlobalVarMutex();
// remember the old handles
out_RcInfo->hOldConOut = GetStdHandle(STD_OUTPUT_HANDLE);
out_RcInfo->hOldConErr = GetStdHandle(STD_ERROR_HANDLE);
// set hidden console as std output
SetStdHandle(STD_OUTPUT_HANDLE, lv_hCon);
SetStdHandle(STD_ERROR_HANDLE, lv_hCon);
// is this the first instance?
if (!gv_hOldConOut)
{
// inform our own console output code about the old handles so our own
// console will be writing to the real console, only console output from
// other parties will write to the hidden console
gv_hOldConOut = out_RcInfo->hOldConOut;
gv_hOldConErr = out_RcInfo->hOldConErr;
}
// release mutex
ReleaseGlobalVarMutex();
}
// done
return TRUE;
}
/***************************************/
/* done redirecting the console output */
/***************************************/
BOOL Shell_DoneRedirectConsole(TRedConInfo *in_RcInfo)
{
// validate
if (!in_RcInfo->hCon)
return FALSE;
// restore original handles?
if (in_RcInfo->hOldConOut)
{
// mutex the globals
WaitForGlobalVarMutex();
// restore original handles
SetStdHandle(STD_OUTPUT_HANDLE, in_RcInfo->hOldConOut);
SetStdHandle(STD_ERROR_HANDLE, in_RcInfo->hOldConErr);
// was this the first instance?
if (in_RcInfo->hOldConOut == gv_hOldConOut)
{
// clear
gv_hOldConOut = NULL;
gv_hOldConErr = NULL;
}
// release mutex
ReleaseGlobalVarMutex();
}
// close the console handle
CloseHandle(in_RcInfo->hCon);
// free read buffer
if (in_RcInfo->BufData)
MemFree(in_RcInfo->BufData);
// clear structure
memset(in_RcInfo, 0, sizeof(TRedConInfo));
// done
return TRUE;
}
/***********************************************/
/* read string from hidden console, then clear */
/***********************************************/
BOOL Shell_ReadRedirectConsole(TRedConInfo *in_RcInfo, TCHAR *out_Str, INT in_MaxLen)
{
/* locals */
TCHAR lv_C;
INT lv_X;
INT lv_Y;
INT lv_W;
INT lv_H;
INT lv_N;
INT lv_Len;
INT lv_Size;
INT lv_PrvLen;
COORD lv_DstSze;
COORD lv_DstOfs;
DWORD lv_Written;
SMALL_RECT lv_SrcRect;
CHAR_INFO *lv_BufData;
CONSOLE_SCREEN_BUFFER_INFO lv_Info;
// preclear output
out_Str[0] = 0;
// validate
if (!in_RcInfo->hCon)
return FALSE;
// reserve character for eos
--in_MaxLen;
// get current buffer info
if (!GetConsoleScreenBufferInfo(in_RcInfo->hCon, &lv_Info))
return FALSE;
// check whether there is something at all
if (!lv_Info.dwSize.X || !lv_Info.dwSize.Y)
return FALSE;
// limit the buffer passed onto read call otherwise it
// will fail with out-of-resources error
lv_DstSze.X = (INT16)(lv_Info.dwSize.X);
lv_DstSze.Y = (INT16)(lv_Info.dwSize.Y < 8 ? lv_Info.dwSize.Y : 8);
// size of buffer needed
lv_Size = lv_DstSze.X * lv_DstSze.Y * sizeof(CHAR_INFO);
// is previous buffer too small?
if (!in_RcInfo->BufData || in_RcInfo->BufSize < lv_Size)
{
// free old buffer
if (in_RcInfo->BufData)
MemFree(in_RcInfo->BufData);
// allocate read buffer
if ((in_RcInfo->BufData = (CHAR_INFO*)MemAlloc(lv_Size)) == NULL)
return FALSE;
// store new size
in_RcInfo->BufSize = lv_Size;
}
// always write to (0,0) in buffer
lv_DstOfs.X = 0;
lv_DstOfs.Y = 0;
// init src rectangle
lv_SrcRect.Left = 0;
lv_SrcRect.Top = 0;
lv_SrcRect.Right = lv_DstSze.X;
lv_SrcRect.Bottom = lv_DstSze.Y;
// buffer to local
lv_BufData = in_RcInfo->BufData;
// start at first string position in output
lv_Len = 0;
// loop until no more rows to read
do
{
// read buffer load
if (!ReadConsoleOutput(in_RcInfo->hCon, lv_BufData, lv_DstSze, lv_DstOfs, &lv_SrcRect))
return FALSE;
// w/h of actually read content
lv_W = lv_SrcRect.Right - lv_SrcRect.Left + 1;
lv_H = lv_SrcRect.Bottom - lv_SrcRect.Top + 1;
// remember previous position
lv_PrvLen = lv_Len;
// loop through rows of buffer
for (lv_Y = 0; lv_Y < lv_H; ++lv_Y)
{
// reset output position of current row
lv_N = 0;
// loop through columns
for (lv_X = 0; lv_X < lv_W; ++lv_X)
{
// is output full?
if (lv_Len + lv_N > in_MaxLen)
break;
// get character from screen buffer, ignore attributes
lv_C = lv_BufData[lv_Y * lv_DstSze.X + lv_X].Char.UnicodeChar;
// append character
out_Str[lv_Len + lv_N++] = lv_C;
}
// remove spaces at the end of the line
while (lv_N > 0 && out_Str[lv_Len+lv_N-1] == ' ')
--lv_N;
// if row was not blank
if (lv_N > 0)
{
// update output position
lv_Len += lv_N;
// is output not full?
if (lv_Len + 2 < in_MaxLen)
{
// append cr/lf
out_Str[lv_Len++] = '\r';
out_Str[lv_Len++] = '\n';
}
}
}
// update screen position
lv_SrcRect.Top = (INT16)(lv_SrcRect.Top + lv_H);
lv_SrcRect.Bottom = (INT16)(lv_SrcRect.Bottom + lv_H);
// until nothing is added or no more screen rows
} while (lv_PrvLen != lv_Len && lv_SrcRect.Bottom < lv_Info.dwSize.Y);
// remove last cr/lf
if (lv_Len > 2)
lv_Len -= 2;
// append eos
out_Str[lv_Len] = 0;
// total screen buffer size in characters
lv_Size = lv_Info.dwSize.X * lv_Info.dwSize.Y;
// clear the buffer with spaces
FillConsoleOutputCharacter(in_RcInfo->hCon, ' ', lv_Size, lv_DstOfs, &lv_Written);
// reset cursor position to (0,0)
SetConsoleCursorPosition(in_RcInfo->hCon, lv_DstOfs);
// done
return TRUE;
}
/**********************************/
/* clear buffer of hidden console */
/**********************************/
BOOL Shell_ClearRedirectConsole(TRedConInfo *in_RcInfo)
{
/* locals */
INT lv_Size;
COORD lv_ClrOfs;
DWORD lv_Written;
CONSOLE_SCREEN_BUFFER_INFO lv_Info;
// validate
if (!in_RcInfo->hCon)
return FALSE;
// get current buffer info
if (!GetConsoleScreenBufferInfo(in_RcInfo->hCon, &lv_Info))
return FALSE;
// clear from (0,0) onward
lv_ClrOfs.X = 0;
lv_ClrOfs.Y = 0;
// total screen buffer size in characters
lv_Size = lv_Info.dwSize.X * lv_Info.dwSize.Y;
// clear the buffer with spaces
FillConsoleOutputCharacter(in_RcInfo->hCon, ' ', lv_Size, lv_ClrOfs, &lv_Written);
// reset cursor position to (0,0)
SetConsoleCursorPosition(in_RcInfo->hCon, lv_ClrOfs);
// done
return TRUE;
}

ReadFile does not return while reading stdout from a child process after it ends

I am working on my library which needs to capture and process the standard output (and err) of a child process as it runs. The problem arises when ReadFile is used to read the output, it does not return once the process ends (gets killed or exits).
It looks like ReadFile is not able to detect that the other end of the pipe (the write handle) is closed. According to the documentation it should return FALSE and set the last error to ERROR_BROKEN_PIPE:
If an anonymous pipe is being used and the write handle has been closed, when ReadFile attempts to read using the pipe's corresponding read handle, the function returns FALSE and GetLastError returns ERROR_BROKEN_PIPE.
Here is my code, I have stripped out the irrelevant bits: (NOTE: I have updated the allium_start to follow the suggested changes, I am keeping the original for reference, please use the newer function code to find flaws)
bool allium_start(struct TorInstance *instance, char *config, allium_pipe *output_pipes) {
// Prepare startup info with appropriate information
SecureZeroMemory(&instance->startup_info, sizeof instance->startup_info);
instance->startup_info.dwFlags = STARTF_USESTDHANDLES;
SECURITY_ATTRIBUTES pipe_secu_attribs = {sizeof(SECURITY_ATTRIBUTES), NULL, true};
HANDLE pipes[2];
if (output_pipes == NULL) {
CreatePipe(&pipes[0], &pipes[1], &pipe_secu_attribs, 0);
output_pipes = pipes;
}
instance->startup_info.hStdOutput = output_pipes[1];
instance->startup_info.hStdError = output_pipes[1];
instance->stdout_pipe = output_pipes[0]; // Stored for internal reference
// Create the process
bool success = CreateProcessA(
NULL,
cmd,
NULL,
NULL,
config ? true : false,
0,
NULL,
NULL,
&instance->startup_info,
SecureZeroMemory(&instance->process, sizeof instance->process)
);
// Return on failure
if (!success) return false;
}
char *allium_read_stdout_line(struct TorInstance *instance) {
char *buffer = instance->buffer.data;
// Process the input
unsigned int read_len = 0;
while (true) {
// Read data
unsigned long bytes_read;
if (ReadFile(instance->stdout_pipe, buffer, 1, &bytes_read, NULL) == false || bytes_read == 0) return NULL;
// Check if we have reached end of line
if (buffer[0] == '\n') break;
// Proceed to the next character
++buffer; ++read_len;
}
// Terminate the new line with null character and return
// Special handling for Windows, terminate at CR if present
buffer[read_len >= 2 && buffer[-1] == '\r' ? -1 : 0] = '\0';
return instance->buffer.data;
}
The allium_start creates the pipe for output redirection (it uses the same pipe for both stdout and stderr to get merged streams) and then creates the child process. The other allium_read_stdout_line function is responsible for reading the output from the pipe and returning it when it encounters a new line.
The issue occurs at the ReadFile function call, it never returns if there is nothing to read after the process exits, from my understanding all the handles of a process are closed by Windows when it ends, so it looks like ReadFile is not able to detect the fact that the pipe (write handle) at the other end has been closed.
How do I fix this? I have been searching for a solution but I have found none so far, one potential option is to use multi-threading and put ReadFile in a separate thread so that it doesn't block the whole program, by using that method I can check if the process still exists periodically while I wait for the reading to finish... or kill/stop the thread if the process is gone.
I do prefer fixing the issue instead of opting for a workaround, but I am open to any other solutions to make it work. Thanks in advance!
Edit: After reading #RemyLebeau's answer and #RbMm's comments in that answer, it is pretty clear that my understand of how handle inheritance works is fundamentally flawed. So I incorporated their suggestions (SetHandleInformation to disable inheritance of read handle and closing it after creating the child process) into my allium_start function:
bool allium_start(struct TorInstance *instance, char *config, allium_pipe *output_pipes) {
// Prepare startup info with appropriate information
SecureZeroMemory(&instance->startup_info, sizeof instance->startup_info);
instance->startup_info.dwFlags = STARTF_USESTDHANDLES;
SECURITY_ATTRIBUTES pipe_secu_attribs = {sizeof(SECURITY_ATTRIBUTES), NULL, true};
HANDLE pipes[2];
if (output_pipes == NULL) {
CreatePipe(&pipes[0], &pipes[1], &pipe_secu_attribs, 0);
output_pipes = pipes;
}
SetHandleInformation(output_pipes[0], HANDLE_FLAG_INHERIT, 0);
instance->startup_info.hStdOutput = output_pipes[1];
instance->startup_info.hStdError = output_pipes[1];
instance->stdout_pipe = output_pipes[0]; // Stored for internal reference
// Create the process
bool success = CreateProcessA(
NULL,
cmd,
NULL,
NULL,
config ? true : false,
0,
NULL,
NULL,
&instance->startup_info,
SecureZeroMemory(&instance->process, sizeof instance->process)
);
// Close the write end of our stdout handle
CloseHandle(output_pipes[1]);
// Return on failure
if (!success) return false;
}
(The below text was originally here before edit 2)
But sadly it still doesn't work :(
Edit 2 (after accepting answer): It does work! See my last comment on the accepted answer.
You are not managing your pipes correctly, or more specifically, you are not controlling the inheritance of your pipe handles. DO NOT let the child process inherit the reading handle of your pipe (output_pipes[0]), otherwise the pipe will not break correctly when the child process ends.
Read MSDN for more details:
Creating a Child Process with Redirected Input and Output
The case of the redirected standard handles that won’t close even though the child process has exited
Use SetHandleInformation() or PROC_THREAD_ATTRIBUTE_LIST to prevent CreateProcess() from passing output_pipes[0] to the child process as an inheritable handle. The child process does not need access to that handle, so there is no need to pass it over the process boundary anyway. It only needs access to the writing handle of your pipe (output_pipes[1]).
For anonymous pipelines, the read process and the write process will have the handler of hRead and hWrite, each of process has its own handler(copy after inheritance). So after your child process exit and close the handler in it, anther hWrite still in parent process. We must pay attention to close hRead in the write process, close hWrite in the read process.
I can reproduce this ReadFile issue, and if closing write handler after setting child's hStdOutput and hStdError, the ReadFile will return 0 after the child process exit.
Here is my code sample,
Parent.cpp:
#include <windows.h>
#include <iostream>
#include <stdio.h>
HANDLE childInRead = NULL;
HANDLE W1 = NULL;
HANDLE W2 = NULL;
HANDLE R2 = NULL;
HANDLE R1 = NULL;
#define BUFSIZE 4096
void CreateChildProcess() {
TCHAR applicationName[] = TEXT("kids.exe");
PROCESS_INFORMATION pi;
STARTUPINFO si;
BOOL success = FALSE;
ZeroMemory(&pi, sizeof(PROCESS_INFORMATION));
ZeroMemory(&si, sizeof(STARTUPINFO));
si.cb = sizeof(STARTUPINFO);
si.hStdError = W1;
si.hStdOutput = W1;
si.hStdInput = R2;
si.dwFlags |= STARTF_USESTDHANDLES;
success = CreateProcess(NULL, applicationName, NULL, NULL, TRUE, CREATE_NEW_CONSOLE, NULL, NULL, &si, &pi);
if (!success) {
printf("Error creating child process \n");
}
else {
printf("Child process successfuly created \n");
CloseHandle(pi.hProcess);
CloseHandle(pi.hThread);
}
}
int main()
{
printf("Parent process running.... \n");
DWORD dRead, dWritten;
CHAR chBuf[BUFSIZE] = { 0 };
BOOL bSuccess = FALSE;
SECURITY_ATTRIBUTES secAttr;
secAttr.nLength = sizeof(SECURITY_ATTRIBUTES);
secAttr.bInheritHandle = TRUE;
secAttr.lpSecurityDescriptor = NULL;
printf("Creating first pipe \n");
if (!CreatePipe(&R1, &W1, &secAttr, 0)) {
printf("\n error creating first pipe \n");
}
printf("Creating second pipe \n");
if (!CreatePipe(&R2, &W2, &secAttr, 0)) {
printf("\n error creating second pipe \n");
}
if (!SetHandleInformation(R1, HANDLE_FLAG_INHERIT, 0)) {
printf("\n R1 SetHandleInformation \n");
}
if (!SetHandleInformation(W2, HANDLE_FLAG_INHERIT, 0)) {
printf("\n W1 SetHandleInformation \n");
}
printf("\n Creating child process..... \n");
HANDLE hStdOut = GetStdHandle(STD_OUTPUT_HANDLE);
HANDLE hStdIn = GetStdHandle(STD_INPUT_HANDLE);
CreateChildProcess();
CloseHandle(W1);
CloseHandle(R2);
for (;;) {
printf("Inside for loop \n");
//1. read from stdin
printf("read from stdin:\n");
bSuccess = ReadFile(hStdIn, chBuf, BUFSIZE, &dRead, NULL);
if (!bSuccess) {
printf("error reading \n");
break;
}
//2. write to Pipe2
printf("write to Pipe2...\n");
bSuccess = WriteFile(W2, chBuf, 100, &dWritten, NULL);
if (!bSuccess) {
printf("error reading \n");
break;
}
//3. read from Pipe1
printf("read from Pipe1...\n");
bSuccess = ReadFile(R1, chBuf, BUFSIZE, &dRead, NULL);
if (!bSuccess)
{
printf("error reading :%d \n", GetLastError());
break;
}
//4. write to stdout
printf("write to stdout:\n");
bSuccess = WriteFile(hStdOut, chBuf, 100, &dWritten, NULL);
if (!bSuccess) {
printf("error reading \n");
break;
}
}
getchar();
return 0;
}
Kids.cpp:
#include <windows.h>
#include <stdio.h>
#define BUFSIZE 4096
int main()
{
DWORD dRead, dWritten;
CHAR chBuf[BUFSIZE];
BOOL success = FALSE;
HANDLE stdIn = GetStdHandle(STD_INPUT_HANDLE);
HANDLE stdOut = GetStdHandle(STD_OUTPUT_HANDLE);
printf("Child process running....");
if (stdIn == INVALID_HANDLE_VALUE || stdOut == INVALID_HANDLE_VALUE) {
ExitProcess(1);
}
//for (;;) {
success = ReadFile(stdIn, chBuf, BUFSIZE, &dRead, NULL);
//if (!success || dRead == 0) break;
success = WriteFile(stdOut, chBuf, dRead, &dWritten, NULL);
//if (!success) break;
//}
return 0;
}

Binary Pipe in Windows

I'm able to create a child process and pipe its stdin and stdout. It works fine when it is in text mode.
However, when I try to set the I/O in the child process to be in the binary format (i.e. no 0x0A to 0x0D 0x0A translation) the child process fails. _setmode returns -1 which has been documented to indicate failure. Why is that and how can it be fixed?
Parent code resembles the following:
const std::string path; // = "path_to.exe"
PROCESS_INFORMATION info;
SECURITY_ATTRIBUTES sec_attr;
//in-out from the CHILD process' perspective
HANDLE out_r = nullptr;
HANDLE out_w = nullptr;
HANDLE in_r = nullptr;
HANDLE in_w = nullptr;
sec_attr.nLength = sizeof(SECURITY_ATTRIBUTES);
sec_attr.lpSecurityDescriptor = NULL;
sec_attr.bInheritHandle = TRUE; //inherit by child processes
if (!CreatePipe(&out_r, &out_w, &sec_attr, 0))
throw std::exception();
if (!SetHandleInformation(out_r, HANDLE_FLAG_INHERIT, 0))
throw std::exception();
if (!CreatePipe(&in_r, &in_w, &sec_attr, 0))
throw std::exception();
if (!SetHandleInformation(in_r, HANDLE_FLAG_INHERIT, 0))
throw std::exception();
if (out_r && out_w && in_r && in_w)
{
startup_info.hStdError = out_w;
startup_info.hStdOutput = out_w;
startup_info.hStdInput = in_r;
startup_info.dwFlags = STARTF_USESTDHANDLES;
}
if (CreateProcessA(path.c_str(), (char*)cmd, NULL, NULL, TRUE, 0, NULL, NULL, &startup_info, &info)
!= TRUE)
{
DWORD error = GetLastError();
//error handling
}
// ... read data using ReadFile
Child code resembles the following:
int status = _setmode(_fileno(stdin), _O_BINARY);
if (status == -1)
throw std::exception();
status = _setmode(_fileno(stdout), _O_BINARY);
if (status == -1)
throw std::exception();
puts("hello from the child process");
I'm looking at the MSDN article Creating a Child Process with Redirected Input and Output that provides a reference implementation for nearly what you're trying to do here.
One subtle difference I see is, where in your parent code is written (the second call to SetHandleInformation()):
if (!SetHandleInformation(in_r, HANDLE_FLAG_INHERIT, 0))
the MSDN example (translated to use your variable names) would be written:
if (!SetHandleInformation(in_w, HANDLE_FLAG_INHERIT, 0))
Here are a couple more SO pages that would be worth looking at as well, if this small change doesn't help:
What is the simplest way to write to stdout in binary mode?
Win32 changing to binary mode child's Stdout (pipe)

Overlapped I/O on anonymous pipe

Is it possible to use overlapped I/O with an anonymous pipe? CreatePipe() does not have any way of specifying FILE_FLAG_OVERLAPPED, so I assume ReadFile() will block, even if I supply an OVERLAPPED-structure.
Here is an implementation for an anonymous pipe function with the possibility to specify FILE_FLAG_OVERLAPPED:
/******************************************************************************\
* This is a part of the Microsoft Source Code Samples.
* Copyright 1995 - 1997 Microsoft Corporation.
* All rights reserved.
* This source code is only intended as a supplement to
* Microsoft Development Tools and/or WinHelp documentation.
* See these sources for detailed information regarding the
* Microsoft samples programs.
\******************************************************************************/
/*++
Copyright (c) 1997 Microsoft Corporation
Module Name:
pipeex.c
Abstract:
CreatePipe-like function that lets one or both handles be overlapped
Author:
Dave Hart Summer 1997
Revision History:
--*/
#include <windows.h>
#include <stdio.h>
static volatile long PipeSerialNumber;
BOOL
APIENTRY
MyCreatePipeEx(
OUT LPHANDLE lpReadPipe,
OUT LPHANDLE lpWritePipe,
IN LPSECURITY_ATTRIBUTES lpPipeAttributes,
IN DWORD nSize,
DWORD dwReadMode,
DWORD dwWriteMode
)
/*++
Routine Description:
The CreatePipeEx API is used to create an anonymous pipe I/O device.
Unlike CreatePipe FILE_FLAG_OVERLAPPED may be specified for one or
both handles.
Two handles to the device are created. One handle is opened for
reading and the other is opened for writing. These handles may be
used in subsequent calls to ReadFile and WriteFile to transmit data
through the pipe.
Arguments:
lpReadPipe - Returns a handle to the read side of the pipe. Data
may be read from the pipe by specifying this handle value in a
subsequent call to ReadFile.
lpWritePipe - Returns a handle to the write side of the pipe. Data
may be written to the pipe by specifying this handle value in a
subsequent call to WriteFile.
lpPipeAttributes - An optional parameter that may be used to specify
the attributes of the new pipe. If the parameter is not
specified, then the pipe is created without a security
descriptor, and the resulting handles are not inherited on
process creation. Otherwise, the optional security attributes
are used on the pipe, and the inherit handles flag effects both
pipe handles.
nSize - Supplies the requested buffer size for the pipe. This is
only a suggestion and is used by the operating system to
calculate an appropriate buffering mechanism. A value of zero
indicates that the system is to choose the default buffering
scheme.
Return Value:
TRUE - The operation was successful.
FALSE/NULL - The operation failed. Extended error status is available
using GetLastError.
--*/
{
HANDLE ReadPipeHandle, WritePipeHandle;
DWORD dwError;
UCHAR PipeNameBuffer[ MAX_PATH ];
//
// Only one valid OpenMode flag - FILE_FLAG_OVERLAPPED
//
if ((dwReadMode | dwWriteMode) & (~FILE_FLAG_OVERLAPPED)) {
SetLastError(ERROR_INVALID_PARAMETER);
return FALSE;
}
//
// Set the default timeout to 120 seconds
//
if (nSize == 0) {
nSize = 4096;
}
sprintf( PipeNameBuffer,
"\\\\.\\Pipe\\RemoteExeAnon.%08x.%08x",
GetCurrentProcessId(),
InterlockedIncrement(&PipeSerialNumber)
);
ReadPipeHandle = CreateNamedPipeA(
PipeNameBuffer,
PIPE_ACCESS_INBOUND | dwReadMode,
PIPE_TYPE_BYTE | PIPE_WAIT,
1, // Number of pipes
nSize, // Out buffer size
nSize, // In buffer size
120 * 1000, // Timeout in ms
lpPipeAttributes
);
if (! ReadPipeHandle) {
return FALSE;
}
WritePipeHandle = CreateFileA(
PipeNameBuffer,
GENERIC_WRITE,
0, // No sharing
lpPipeAttributes,
OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL | dwWriteMode,
NULL // Template file
);
if (INVALID_HANDLE_VALUE == WritePipeHandle) {
dwError = GetLastError();
CloseHandle( ReadPipeHandle );
SetLastError(dwError);
return FALSE;
}
*lpReadPipe = ReadPipeHandle;
*lpWritePipe = WritePipeHandle;
return( TRUE );
}
No. As explained here, anonymous pipes do not support asynchronous I/O. You need to use a named pipe. There's example code to do this on MSDN here and here.
first of all need understand - what is Anonymous Pipes and what, are exist difference between anonymous and Named Pipes at all.
really exist only single pipe type (implemented by npfs.sys). no any difference, except name, between named and anonymous pipes at all. both is only pipes.
so called anonymous pipes - this is special/random named pipes before win7 and true unnamed pipes begin from win7.
when msdn write that "anonymous pipe is one-way pipe" - this is lie. as any pipe it can be one-way or duplex. when msdn write that "Asynchronous (overlapped) read and write operations are not supported by anonymous pipes." - this is lie. of course pipes support asynchronous io. the name of pipe not affect this.
before win7 really unnamed pipes even not exist at all. CreatePipe function use Win32Pipes.%08x.%08x format for create name of "Anonymous Pipe".
static LONG PipeSerialNumber;
WCHAR name[64];
swprintf(name, L"\\Device\\NamedPipe\\Win32Pipes.%08x.%08x",
GetCurrentProcessId(), InterlockedIncrement(&PipeSerialNumber));
begin from win7 CreatePipe use another technique (relative file open) for create pipe pair - now it really anonymous.
for example code witch create pipe pair where one pipe is asynchronous and not inheritable. and another pipe is synchronous and inheritable. both pipes is duplex (support both read and write)
ULONG CreatePipeAnonymousPair7(PHANDLE phServerPipe, PHANDLE phClientPipe)
{
HANDLE hNamedPipe;
IO_STATUS_BLOCK iosb;
static UNICODE_STRING NamedPipe = RTL_CONSTANT_STRING(L"\\Device\\NamedPipe\\");
OBJECT_ATTRIBUTES oa = { sizeof(oa), 0, const_cast<PUNICODE_STRING>(&NamedPipe), OBJ_CASE_INSENSITIVE };
NTSTATUS status;
if (0 <= (status = NtOpenFile(&hNamedPipe, SYNCHRONIZE, &oa, &iosb, FILE_SHARE_VALID_FLAGS, 0)))
{
oa.RootDirectory = hNamedPipe;
static LARGE_INTEGER timeout = { 0, MINLONG };
static UNICODE_STRING empty = {};
oa.ObjectName = ∅
if (0 <= (status = ZwCreateNamedPipeFile(phServerPipe,
FILE_READ_ATTRIBUTES|FILE_READ_DATA|
FILE_WRITE_ATTRIBUTES|FILE_WRITE_DATA|
FILE_CREATE_PIPE_INSTANCE,
&oa, &iosb, FILE_SHARE_READ|FILE_SHARE_WRITE,
FILE_CREATE, 0, FILE_PIPE_BYTE_STREAM_TYPE, FILE_PIPE_BYTE_STREAM_MODE,
FILE_PIPE_QUEUE_OPERATION, 1, 0, 0, &timeout)))
{
oa.RootDirectory = *phServerPipe;
oa.Attributes = OBJ_CASE_INSENSITIVE|OBJ_INHERIT;
if (0 > (status = NtOpenFile(phClientPipe, SYNCHRONIZE|FILE_READ_ATTRIBUTES|FILE_READ_DATA|
FILE_WRITE_ATTRIBUTES|FILE_WRITE_DATA, &oa, &iosb,
FILE_SHARE_VALID_FLAGS, FILE_SYNCHRONOUS_IO_NONALERT)))
{
NtClose(oa.RootDirectory);
}
}
NtClose(hNamedPipe);
}
return RtlNtStatusToDosError(status);
}
ULONG CreatePipeAnonymousPair(PHANDLE phServerPipe, PHANDLE phClientPipe)
{
static char flag_supported = -1;
if (flag_supported < 0)
{
ULONG dwMajorVersion, dwMinorVersion;
RtlGetNtVersionNumbers(&dwMajorVersion, &dwMinorVersion, 0);
flag_supported = _WIN32_WINNT_WIN7 <= ((dwMajorVersion << 8)| dwMinorVersion);
}
if (flag_supported)
{
return CreatePipeAnonymousPair7(phServerPipe, phClientPipe);
}
static LONG PipeSerialNumber;
WCHAR name[64];
swprintf(name, L"\\\\?\\pipe\\Win32Pipes.%08x.%08x", GetCurrentProcessId(), InterlockedIncrement(&PipeSerialNumber));
HANDLE hClient, hServer = CreateNamedPipeW(name,
PIPE_ACCESS_DUPLEX|FILE_READ_DATA|FILE_WRITE_DATA|FILE_FLAG_OVERLAPPED,
PIPE_TYPE_BYTE|PIPE_READMODE_BYTE, 1, 0, 0, 0, 0);
if (hServer != INVALID_HANDLE_VALUE)
{
static SECURITY_ATTRIBUTES sa = { sizeof(sa), 0, TRUE };
hClient = CreateFileW(name, FILE_GENERIC_READ|FILE_GENERIC_WRITE,
FILE_SHARE_READ|FILE_SHARE_WRITE, &sa, OPEN_EXISTING, 0, 0);
if (hClient != INVALID_HANDLE_VALUE)
{
*phServerPipe = hServer, *phClientPipe = hClient;
return NOERROR;
}
CloseHandle(hServer);
}
return GetLastError();
}

Resources