How to break (set a breakpoint) on Console output on Windows? - windows

Background:
I'm running a large nUnit test suite on my developer PC (via nunit-console) and I'm seeing some garbage output in the console window.
The units under test do involve .NET code as well as underlying C++ and C libraries and I haven't yet found out who is producing the garbage output.
Question:
Is there a single Windows API function where all Console output goes through? (regardless where it comes from.)
I have tried setting a breakpoint inside WriteConsole but that doesn't even catch printf output from the CRT. Is there any "central" location to set a breakpoint to catch all console output in a Windows application? (Some Nt... function?)

Console I/O makes calls to ReadFile and WriteFile on Windows. WriteFile is the routine for console output. It would be best to hook or breakpoint WriteFile and filter the handle specified to it. Don't mistake actual File I/O for operations on a console handle.
To acquire the correct handle used for console output on a Windows process:
GetStdHandle(STD_OUTPUT_HANDLE);
Alternatively, opening the console output handle using CreateFile and specifying CONOUT$ works and is also recommended:
HANDLE console_output = CreateFileA("CONOUT$", GENERIC_WRITE, FILE_SHARE_WRITE, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
This will return the handle you need to watch for in WriteFile calls. Just like with a hook though, you can specify the conditions for your breakpoint and check for the correct console handle in the call.
There is more about operations on console handles using ReadFile/WriteFile in the Microsoft documentation

Related

Using a VB6 program to check another VB program's Error Codes

I need to develop a "Watchdog" program to detect when a program has crashed, kill the process and then restart it. Unfortunately, the original developer of this program is no longer here, and his computer is no longer able to compile it without buying another expensive license. The program in question crashes with a VB error code ("Runtime error 5 ... etc). I need to detect when the program enters this state. Alternatively, I need to be able to see the first program's error handler if possible, so I can check if it is in a "trappable" error, or at the very least, be able to check the current status of the program in question.
Programs are isolated from each other. The CPU makes it so every program thinks it's the only program running and, in 32 bit, has 4 GB of memory all to it's self. So a program cannot affect or monitor another.
So debugging is an exception to the above because computers wouldn't be usable without it and that is built into the CPU/OS.
So run the VB6 program under a debugger, most biz computers have NTSD installed. Note this is not a basic debugger but a machine code debugger.
If it's a window then just wait for the window title. You can use API calls like FindWindow to monitor window titles.
HWND FindWindow( LPCTSTR lpClassName,
LPCTSTR lpWindowName
);

Does "Image File Execution Options" intercept CreateProcess commands?

I want to know how the mechanism of debugger injection works. Why is "Image File Execution Options" so special?
I have two guesses.
CreateProcess will call an internal function that checks against the list of registry keys. If it is found, then it manipulates the arguments and calls the debugger exe instead.
There is some other service listening for CreateProcess calls and intercepts them. It kills the original call or message (if createprocess is a message or message-like), then it runs the new process as if the original caller called it.
My desire is to verify and update components before an application starts. I like the IFEO "feature" but i need to run the original process after the verification step so I need a way to run it without recursing into the updater. I hope that by learning more about the injection system I can get this system working.
This article explains how it works.
In Windows XP and 2003 the user-mode CreateProcess code reads the registry and, if required, launches the debugger instead.
In more recent versions of Windows this functionality has moved into kernel mode.
But neither case seems to involve a general interception mechanism for CreateProcess.

On Windows, how does console window ownership work?

When a console application is started from another console application, how does console ownership work?
I see four possibilities:
The second application inherits the console from the first application for its lifetime, with the console returning to the original owner on exit.
Each application has its own console. Windows then somehow merges the content of the two into what the "console" visible to the user
The second application get a handle to the console that belongs to the first application.
The console is placed into shared memory and both applications have equal "ownership"
It's quite possible that I missed something and none of these four options adequately describe what Windows does with its consoles.
If the answer is close to option 4. My follow-up question is which of the two processes is responsible for managing the window? (Handling graphical updates when the screen needs to be refreshed / redrawn, etc)
A concrete example: Run CMD. Then, using CMD, run [console application]. The [console application] will write to what appears to be the same console window that CMD was using.
None of your four possibilities is actually the case, and the answer to your follow-on question, "Which of the two processes is responsible for managing the window?", is that neither process is responsible. TUI programs don't have to know anything about windows at all, and, under the covers, aren't necessarily even plumbed in to the GUI.
Consoles are objects, accessed via handles just like files, directories, pipes, processes, and threads. A single process doesn't "own" a console via its handle to it any more than a process "owns" any file that it has an open handle to. Handles to consoles are inherited by child processes from their parents in the same way that all other (inheritable) handles are. Your TUI application, spawned by CMD, simply inherits the standard handles that CMD said that it should inherit, when it called CreateProcess() — which are usually going to be CMD's standard input, output, and error (unless the command-line told CMD to use some other handles as the child's standard input, output, and error).
Consoles aren't dependent upon CMD. They exist as long as there are (a) any open handles to the console's input or output buffers or (b) any processes otherwise "attached" to the console. So in your example you could kill CMD, but only when you terminated the child process too would the console actually be destroyed.
The process that is in charge of displaying the GUI windows in which consoles are presented is, in Windows NT prior to version 6.1, CSRSS, the Client-Server Runtime SubSystem. The window handling code is in WINSRV.DLL, which contains the "console server" that — under the covers — Win32 programs performing console I/O make LPC calls to. In Windows NT 6.1, this functionality, for reasons covered by Raymond Chen, moved out of CSRSS into a less-privileged process that CSRSS spawns.
My guess is somewhere between 3 and 4. The console is a self-standing object, which has standard input, output and error streams. These streams are attached to the first process that uses the console. Subsequent processes can also inherit these streams if not redirected (e.g. running a command with redirect to a file.)
Normally there is no contention, since parent processes usually wait for their child process to complete, and asynchronous processes typically start their own console (e.g. try "start cmd" in a command prompt) or redirect standard output.
However, there is nothing to stop both processes writing to the output stream at the same time - the streams are shared. This can be a problem when using some runtime libraries since writes to standard output/error may not be immediately flushed, leading to mixed garbled output. In general, having to processes actively writing to the same output stream is usually not a good idea, unless you take measures to co-ordinate their output through concurrency primitives like Mutexes, Events and the like.
The way the SDK talks about it strongly resembles 1. It is an option with CreateProcess, described as follows:
CREATE_NEW_CONSOLE
The new process has a new console, instead of inheriting its parent's console (the default). For more information, see Creation of a Console.
Output however happens through handles, you'd get one with GetStdHandle(). Passing STD_OUTPUT_HANDLE returns the console handle, assuming output isn't redirected. Actual output is done through WriteFile() or WriteConsole/Output(). If both processes keep writing output to the handle then their output will be randomly intermingled. This is otherwise indistinguishable from what would happen when two programs write to the same file handle.
Logically, there's a screen buffer associated with a console. You can tinker with it with SetConsoleScreenBufferXxx(). From that point of view you could call it shared memory. The actual implementation is undiscoverable, handles abstract them away, like any Win32 API. It is sure to have changed considerably in Vista with the new conhost.exe process.
CMD 'owns' the console. When it creates a process for an app, that app inherits handles to the console. It can read and write those. When the process goes away, CMD continues ownership.
Note: I'm not entirely sure that 'ownership' is the right word here. Windows will close the Console when CMD exits, but that may be a simple setting.
Each application will run in it's own AppDomain. Each AppDomain should be running it's own console.
Ah, you're right. I was thinking about running executables within a process and forgot they start their own process - I didn't drill down far enough.
I think it's spelled out fairly well in the documentation.

How do you get console input and output in a Windowed C++ project?

I would like to make a debug system in C++ where I can get input from a console and output. Output is the most important, but I would also like to get input to be able to change settings during program execution.
Does anyone have any good methods on solving this dilemma?
look at http://msdn.microsoft.com/en-us/library/ms681944(VS.85).aspx
AllocConsole
AttachConsole
GetStdHandle
WriteFile
ReadFile

How to write to the console in a GUI application

Background: We develop win32 applications, and use the "Thompson Toolkit" on windows to give us a unix-like shell that we use as our command-line.
We have a GUI program (with a WinMain and message loop) that we want to write to the console, but printf and so on don't work, even when we launch the program from the console. How can we write to the console from a GUI program? We need to print text there so that an automated build system can display error messages and so on.
Thanks.
In short, you need to attach a console. For details and ready to use code, see http://www.codeproject.com/KB/dialog/ConsoleAdapter.aspx.
Basicly you have to create a console by your self with AllocConsole, AttachConsole. After that you have to get standard handles with GetStdHandle and "associates a C run-time file descriptor with an existing operating-system file handle" with help of _open_osfhandle.
The returned handle can be used to overwrite crt stdin and stdout. After that all crt methods like printf should work.
Instead of logging to the console, log to a file and then track the file with a separate gui application. This keeps the console uncluttered and gives you a more persistent record of your log, which occasionally is extremely useful. There are various libraries which will do most of this for you, or you can keep it simple and just do it yourself.
somewhere in the Visual Studio Project Settings you can switch on having a console, assuming you are using VS. (Can't say where because I currently don't have it)

Resources