How do debuggers bypass Image File Execution Options when launching their debugee? - debugging

Question

I'm doing some poking around in Windows internals for my general edification, and I'm trying to understand the mechanism behind Image File Execution Options. Specifically, I've set a Debugger entry for calc.exe, with "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -NoLogo -NoProfile -NoExit -Command "& { start-process -filepath $args[0] -argumentlist $args[1..($args.Length - 1)] -nonewwindow -wait}" as the payload. This results in recursion, with many powershell instances being launched, which makes sense given that I'm intercepting their calls to calc.exe.
That begs the question, though: how do normal debuggers launch a program under test without causing this sort of recursive behavior?

Answer 1

It is anyway a good question about Windows internals, but the reason it has my interest right now is that it has become a practical question for me. At somewhere that I do paid work are three computers, each with a different Windows version and even different debuggers, for which using this IFEO trick results in the debugger debugging itself, apparently trapped in this very same circularity that troubles the OP.
How do debuggers usually avoid this circularity? Well, they themselves don't. Windows avoids it for them.
But let's look first at the circularity. Simple demonstrations are hardly ever helped by PowerShell concoctions and calc.exe is not what it used to be. Let's instead set the Debugger value for notepad.exe to c:\windows\system32\cmd.exe /k. Windows will interpret this as meaning that attempting to run notepad.exe should ordinarily run c:\windows\system32\cmd.exe /k notepad.exe instead. CMD will interpret this as meaning to run notepad.exe and hang about. But this execution too of notepad.exe will be turned into c:\windows\system32\cmd.exe /k notepad.exe, and so on. The Task Manager will soon show you many hundreds of cmd.exe instances. (The good news it that they're all on the one console and can be killed together.)
The OP's question is then why does CMD and its /k (or /c) switch for running a child go circular in a Debugger value but WinDbg, for instance, does not.
In one sense, the answer is down to one bit in an undocumented structure, the PS_CREATE_INFO, that's exchanged between user and kernel modes for the NtCreateUserProcess function. This structure has become fairly well known in some circles, not that they ever seem to say how. I think the structure dates from Windows Vista, but it's not known from Microsoft's public symbol files until Windows 8 and even then not from the kernel but from such things as the Internet Explorer component URLMON.DLL.
Anyway, in the modern form of the PS_CREATE_INFO structure, the 0x04 bit at offset 0x08 (32-bit) or 0x10 (64-bit) controls whether the kernel checks for the Debugger value. Symbol files tell us this bit is known to Microsoft as IFEOSkipDebugger. If this bit is clear and there's a Debugger value, then NtCreateUserProcess fails. Other feedback through the PS_CREATE_INFO structure tells KERNELBASE, for its handling of CreateProcessInternalW, to have its own look at the Debugger value and call NtCreateUserProcess again but for (presumably) some other executable and command line.
When instead the bit is set, the kernel doesn't care about the Debugger value and NtCreateUserProcess can succeed. How the bit ordinarily gets set is by KERNELBASE because the caller is asking not just to create a process but is specifically asking to be the debugger of the new process, i.e., has set either DEBUG_PROCESS or DEBUG_ONLY_THIS_PROCESS in the process creation flags. This is what I mean by saying that debuggers don't do anything themselves to avoid the circularity. Windows does it for them just for their wanting to debug the executable.
One way to look at the Debugger value as an Image File Execution Option for an executable X is that the value's presence means X cannot execute except under a debugger and the value's content may tell how to do that. As hackers have long noticed, and the kernel's programmers will have noticed well before, the content need not specify a debugger and the value can be adapted so that attempts to run X instead run Y. Less noticed is that Y won't be able to run X unless Y debugs X (or disables the Debugger value). Also less noticed is that not all attempts to run X will instead run Y: a debugger's attempt to run X as a debuggee will not be diverted.

Answer 2

TLDR of Geoff's great answer - use DEBUG_PROCESS or DEBUG_ONLY_THIS_PROCESS to bypass the Debugger / Image File Execution Options (IFEO) global flag and avoid the recursion.
In C#, using the excellent Vanara.PInvoke.Kernel32 NuGet:
var startupInfo = new STARTUPINFO();
var creationFlags = Kernel32.CREATE_PROCESS.DEBUG_ONLY_THIS_PROCESS;
CreateProcess(path, null, null, null, false, creationFlags, null, null, startupInfo, out var pi);
DebugActiveProcessStop(pi.dwProcessId);
Note that DebugActiveProcessStop was key for me (couldn't see a window when opened notepad.exe otherwise) - and makes sense anyway if your program is not really a debugger and you just want the bypass.