Writing a Windows NT subsystem [closed] - windows

Question

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 8 years ago.
Improve this question
I'd like to try writing my own minimal NT subsystem on Windows 7 for purely educational purposes -- something like a bare-bones equivalent of the posix.exe in Microsoft's Subsystem for Unix-based Applications.
But I can't seem to find any public documentation on this topic. What API does a subsystem need to implement? How does it get registered with Windows? How does the subsystem image need to be built (what flags need to be set in the PE header, etc.)?
I'd most like to find a book or web site with an overview of the entire subject, or even the source code for a "hello world" NT subsystem that someone else has written. But anything at all would be appreciated if you can point me in the right direction here...

Answer 1

Here are the major components of a subsystem:
User-mode server. The server creates a (A)LPC port and listens for and handles client requests.
User-mode client DLL. In the DLL_INIT_ROUTINE, you can connect to the port set up by the server. This DLL will expose your subsystem's API, and some functions will require communication with the server.
Kernel-mode support driver (you might not need this).
You will want to store process or thread state in either your server or driver. If you're storing it in the server, you might need something like NtRegisterThreadTerminatePort to ensure you get to clean up when a process or thread exits. If you're using a driver, you need PsSetCreateProcessNotifyRoutine.
And lastly, if you're on XP and below, you can add new system calls. You can do this by calling KeAddSystemServiceTable. To invoke the system calls from user-mode, you need to create stubs like this (for x86):
; XyzCreateFooBar(__out PHANDLE FooBarHandle, __in ACCESS_MASK DesiredAccess, ...)
mov eax, SYSTEM_CALL_NUMBER
mov edx, 0x7ffe0300
call [edx]
retn 4
On Vista and above you can no longer add new system service tables because there is only room for two: the kernel's system calls and win32k's system calls.
After a bit of Googling I found this: http://winntposix.sourceforge.net/. I think it's very similar to what you're looking for, and uses a lot of the things I have mentioned.

Answer 2

I'm also obsessed with the native API. :)
And I'm glad to say that it's nowhere near as dangerous or as undocumented as some people make it seem. :]
There's no source code for "Hello, world" because the native API doesn't interact so easily with the console, since it's part of the Win32 subsystem and requires client/server communication with ports. If you need to write a console application, you need to communicate directly with CSRSS, whose message formats are undocumented (although some of its format can be found in ReactOS's source -- it would do you many benefits if you get familiar with ReactOS).
I'll post an example here soon that you might find interesting; for now, do be aware that your only option ever is to link with NTDLL.dll, and that, for that, you need the Driver Development Kit (since you need the lib file).
Update: Check this out!
(I have a feeling no one else will post something quite as rebellious as this. Showing GUI with the native API?! I must be crazy!)
#include <Windows.h>
typedef DWORD NTSTATUS;
//These are from ReactOS
typedef enum _HARDERROR_RESPONSE_OPTION
{
OptionAbortRetryIgnore,
OptionOk,
OptionOkCancel,
OptionRetryCancel,
OptionYesNo,
OptionYesNoCancel,
OptionShutdownSystem
} HARDERROR_RESPONSE_OPTION, *PHARDERROR_RESPONSE_OPTION;
typedef enum _HARDERROR_RESPONSE
{
ResponseReturnToCaller,
ResponseNotHandled,
ResponseAbort,
ResponseCancel,
ResponseIgnore,
ResponseNo,
ResponseOk,
ResponseRetry,
ResponseYes,
ResponseTryAgain,
ResponseContinue
} HARDERROR_RESPONSE, *PHARDERROR_RESPONSE;
typedef struct _UNICODE_STRING {
USHORT Length;
USHORT MaximumLength;
PWSTR Buffer;
} UNICODE_STRING, *PUNICODE_STRING;
//You'll need to link to NTDLL.lib
//which you can get from the Windows 2003 DDK or any later WDK
NTSYSAPI VOID NTAPI RtlInitUnicodeString(IN OUT PUNICODE_STRING DestinationString,
IN PCWSTR SourceString);
NTSYSAPI NTSTATUS NTAPI NtRaiseHardError(IN NTSTATUS ErrorStatus,
IN ULONG NumberOfParameters, IN ULONG UnicodeStringParameterMask,
IN PULONG_PTR Parameters,
IN HARDERROR_RESPONSE_OPTION ValidResponseOptions,
OUT PHARDERROR_RESPONSE Response);
#define STATUS_SERVICE_NOTIFICATION_2 0x50000018
int main()
{
HARDERROR_RESPONSE response;
ULONG_PTR items[4] = {0};
UNICODE_STRING text, title;
RtlInitUnicodeString(&text,
L"Hello, NT!\r\nDo you like this?\r\n"
L"This is just about as pretty as the GUI will get.\r\n"
L"This message will self-destruct in 5 seconds...");
RtlInitUnicodeString(&title, L"Native Message Box!");
items[0] = (ULONG_PTR)&text;
items[1] = (ULONG_PTR)&title;
items[2] = (ULONG_PTR)OptionYesNo;
items[3] = (ULONG_PTR)5000;
NtRaiseHardError(STATUS_SERVICE_NOTIFICATION_2, ARRAYSIZE(items),
0x1 | 0x2 /*First two parameters are UNICODE_STRINGs*/, items,
OptionOk /*This is ignored, since we have a custom message box.*/,
&response);
return 0;
}
If you have any questions, feel free to ask! I'm not scared of the native API! :)
Edit 2:
If you're trying to make your own DLL version of Kernel32 and have it load like Kernel32 does with every process (hence a new subsystem), I just wanted to let you know that I don't think it's possible. It's rather similar to this question that I asked a couple of days ago, and it seems that you can't extend the NT PE Loader to know about new subsystems, so I don't think it'll be possible.