I have several questions about the PE "subsystem" field, which may overlap to some extent. To avoid spamming this place with each question separately, I thought I'd ask them together and then re-ask separately anything that doesn't get addressed. Hope this is OK...
I know that IMAGE_SUBSYSTEM_WINDOWS_CUI makes the OS "pre-attach" the process to a console, either of its parent process or creating a new console if necessary. IMAGE_SUBSYSTEM_WINDOWS_GUI doesn't do that.
Are there any other differences between these two in a modern version of Windows? Have there been more in the past?
What about the other values, are they used by Windows merely to reject an EXE, or do they cause Windows to emulate a different API? Is this "emulation" process extensible by end-users or is this hard-baked into the OS?
Yes, pre-attaching a console seems to be the only current difference. If memory serves, that wasn't so much the case with the 16-bit based versions of Windows though (95/98/SE/Me).
Older versions of NT accepted other values for the POSIX and OS/2 subsystems.
In theory, you might be able to write your own subsystem using the NT Native API. Although there's a fair amount of documentation around for that if you look, I'm not at all sure it's sufficient to this task. The Win32 subsystem has always had rather a "special" status anyway, and now that the other subsystems are gone, I'm not at all sure they've even tried to assure that other subsystems can be integrated. On NT 4 (for one example) I'd have said "difficult but almost certainly possible". On a current version of Windows, I'd say there's about even odds that some change to the kernel would prevent it from working -- and much lower than even odds that the currently-available documentation would be up to the task of anybody but MS implementing a subsystem.
Related
I am trying to understand how code, regardless of the language, works. Specifically thinking about software that runs on Windows.
Is my understanding correct that every built in function of a particular language maps to an exposed function in the Windows API when writing software for the Windows platform?
I guess my question can be even more generally, can a language do anything outside of what the OS provides? If so, how? What is an example of this?
There is a theoretical and a practical answer to this.
Practical: yes.
The WinAPI is the API everything uses to do things on Windows. It’s stable and compatible between versions so you can write whatever you want with it and it will work on practically any version of Windows given you don’t use any APIs that aren’t present in an old version. There isn’t any other interface to talk to the operating system properly.
Any language or platform that wants to work on top of this will call WinAPI. C libraries, Python, etc all are written so that they work on top of it (often other languages use C or C++ libraries which use WinAPI).
Theoretical: no.
Windows itself includes a Native API which is the actual OS interface and WinAPI is built on top of this. It is not really used if it’s not necessary since it’s not really documented. It’s used in a couple of Windows components that need to run before the other parts of the system are running and you can build applications linking to this API. But since it’s undocumented it’s not really reasonable and may change whenever.
There is also the syscall level. Several Windows components provide the lower level services for operations done on the WinAPI level. You can write, for example, an assembly program and use the syscalls directly if you want to. Mostly you don’t want to so this is more of a theoretical rather than practical answer to different platforms communicating with the OS. These also may change based on the OS.
WinAPI is basically only one subsystem that runs on the NT kernel. For example, Windows Subsystem for Linux is another one which implements its own syscalls which are then translated to Windows ones. There has also been a POSIX subsystem previously.
So all in all it depends on which level you look at it, but the practical answer is yes. Everything practically runs on WinAPI.
Yes. Even if an application exits immediately, it uses a windows call.
So it is not only theoretical. In theory, as in practice, every Windows application uses the API, because there is nothing else to use.
Even if you try to rewrite each functionality you are about to use, you would eventually have to install a driver, and this also mean you would use the API.
How are operating systems typically debugged? They cannot be stepped through with a debugger like simple console programs, and the build times are too large to repeatedly make small changes and recompile the whole thing.
They aren't debugged as a multi-gigabyte programs! :)
If you mean the individual user-mode components, they can mainly be debugged just like normal programs and libraries (because they are normal programs/libraries!).
For kernel-mode components, though, each OS has its own mechanism; here is some information regarding the way that we do kernel debugging in Windows. It can be done using the help of another machine connected to the machine you're debugging, via a serial port or something. I'm not familiar with the process itself, but that's the gist of how they work. (You need to set some boot loader options so that the system is ready for the debugger to be connected as early as possible.)
It depends on which part of the operating system you're talking about. When I worked at MSFT, I worked on the IE team. We debugged IE and the shell (Windows Explorer) in Visual Studio and stepped through them line by line all day long. Though, sometimes, it's easier to debug using a command line tool such as NTSD.
If, however, you want to debug anything in Kernel land such as the OS kernel or device drivers, which I suspect is really what you're asking, then you must use the Kernel debugger. For Windows that is a command line tool called kd, and generally you run the debugger on one machine and remotely debug the target.
There are a whole set of techniques throughout history from flashing lights on the console, to the use of hardware devices like an ICE, to more modern techniques utilizing fairly standard debuggers. One technique that is more common among OS developers then application developers is the analysis of a core dump. Look at something like mdb on solaris for ideas about how Solaris kernel developers do some of their debugging. Also tracing technologies are used. Anywhere from fairly straightforward logging packages to more modern techniques like dtrace.
Also note that the techniques used depend on the layer of software. Initial boot tends to be a fairly hard place to get your fingers into. But after that the environment of modern operation systems looks more and more like the application setting you are use to. In the end, it is all code :)
I am interested whether I can write an application which will be able to call some code without emulation layer and some code inside of 32 bit emulation layer.
The main reason for that is that I will need to use API SetWindowHook and I want to set hook both for 64 bit and 32 bit applications.
Sure, I can create 2 application (one for 32 bit and another for 64 bit) and launch them simultaneously. However, I will need more code to manage them (start, stop, upgrade and etc).
So, I am looking whether it's possible at all to have one application.
The only idea which I have is to have one application and 2 COM DLL's (32bit and 64bit) and use a surrogate process to run code of 32 bit. However, it will require some additional COM wrappers and so on.
I cannot suggest a better way of doing it but what I can do is give you the source to a simple hook-based tool which does exactly the same kind of thing. Feel free to the bits that are useful to you:
http://www.pretentiousname.com/NoBarTab/NoBarTab_poc3.zip
(If this URL breaks in the future, just go up a level; it'd probably because I've finished it and put a real page up for the tool and its source.)
It's a VS2010 C++ project but should be easy to compile in older IDEs. (Writing this actually put me off using VS2010 any further for now, heh.)
Obviously, if you use it, please rename any window classes and binary names to avoid conflicts with my tool. (Anything with "NoBarTab" in the name.)
FWIW, this is a tool I started writing a few weeks ago but haven't got around to finishing. The hooking part is finished, though. It hooks window creation so that it can, for specific processes, remove tabs from the Windows 7 taskbar. (I hate the way that feature is used by VMware, in particular.) I was going to release the source code anyway when I finished it...
The 32/64-bit hooking part is all done. The only thing I haven't got around to is adding a config UI so you can specify which processes it should care about, but that's not important for what you are doing.
(I should say that the way I remove tabs from the Win7 taskbar is a complete hack and might break with future versions of Windows. There's no documented way to do that so I had to settle on a nasty kludge. The actual hooking code that you'd be interested in is all "proper", though.)
Also, I made it so that almost all of the real logic is within the main 64-bit exe. The 32-bit EXE just exists to install the 32-bit hook DLL and both the 32-bit and 64-bit hook DLLs just post a message to the main 64-bit exe's hidden window. Whether that is suitable for what you're doing I leave to you to decide, but I figure it probably fits with your desire to have everything in one place as much as possible.
Hope it's useful!
SetWinEventHook is a higher-level hooking API which handles the 32-bit/64-bit stuff for you. This came up in the answers to another question today and I thought it'd be worth mentioning here in case that solved your problem. Full credit to #atzz for his answer over there.
Whether or not SetWinEventHook is as suitable to you as the lower-level SetWindowsHookEx will depend on exactly what you're doing. (In my case, I could probably re-write the NoBarTab code in my other answer to use the more simple API. Haven't looked in detail yet, though.)
The biggest reason for the emulation layer is that your 64-bit code is going to have a larger address space. Say, for example, you have data addressed at 0x12345678aa000000. If you pass that pointer straight to 32-bit code, it will get truncated to 32-bit address space. That is, to say, that the high-order 32 bits get removed. So the address to your data looks like 0x00000000aa000000 to your 32-bit code. This is clearly a completely different area and not what you intended.
Yes, it's possible. No you shouldn't do it unless you're extremely experience with x86 and x64 assembly, and have access to the 32-bit source code to ensure that it knows that its actually running in 64-bit space and also have access to 64-bit code to ensure that all of the data it passes to the 32-bit code is only in the 32-bit address range.
No, executing 32-bit code without the emulation layer is undesirable unless you are trying to make an UBER UGLY hack.
I think your only hope is to do it through something like out-of-process COM because each process has to be all 32 bit or all 64 bit.
I was watching the WWDC 2009 Keynote and something someone said about Windows 7/Vista got me curious..
The speaker claimed that 7 was still a poor operating system because it still used the same technologies such as DLLs and the registry. How accurate are his claims and how different is OS X doing it? Even os x has dynamically loaded libraries right? I guess the Registry thing might have some weight..
Can anyone explain to me the differences in each OS' strategy?
I'm not trying to incite fanboys here or anything, I just want to know how both operating systems tackle problems in general..
Thanks,
kreb
Of course both operating systems have facilities for using DLLs (they're called dylibs or Frameworks on OS X depending on how they're packaged).dylibs are very much like DLLs--they are a dynamically linked library and as such there may be multiple versions of them floating around. Frameworks, on the other hand, are really a directory structure. They contain dynamically linked libraries (potentially multiple versions of them), resources, headers, documentation, etc. The dynamic linker on OS X automatically handles choosing the correct library version from the framework for each executable. The system appears to work better than Windows' DLL management which is, well, quite a mess still (of course, Windows' system is tied by legacy issues that Apple dropped when they moved to OS X). To be fair, Unix has had a solution to this problem for a long time, as well using symbolic links to link dylibs to their correct versioned implementation, allowing multiple installed versions.
There is no OS X equivalent of the Windows registry. This is good and bad. The good side is that it's much harder to corrupt an entire OS X system with a registry screw up. OS X instead stores configuration in many separate files, usually one or more per application, user, whatever. These files are generally a plist (an XML schema representing dictionaries, arrays, and primitive types) formatted file. The bad side is that, by retaining this Unix-y heritage, OS X doesn't have the same über-admin tools that can churn through the registry and do all sorts of crazy things.
DLLs
The major difference between OS X and Windows is that Windows historically tried to save space/memory by having everyone share code (i.e. you install one DLL, everyone can use it). Apple statically compiles (well, not really, but it may as well be) all of the non-system libraries into every application. Wastes disk space/memory, but makes app deployment way easier and no versioning issues.
Registry
OS X does have a registry, they're just flat files called plists, instead of a magic component that's mostly like a filesystem except where it's not. Apple's approach makes it easy to migrate settings from one machine to another, whereas Windows' approach is faster in-memory, and allows apps to easily "watch" a key without taking a big perf hit (i.e. one app changes a key and the other instantly knows about it).
In conclusion
The keynote presenter's full of it, 10.6 is mostly the same code as 10.5, which was mostly the same code as 10.4 et al, just like Win7 is mostly Vista, which is mostly Server '03, etc. There's far too much tested code in an operating system to throw it away every release, especially if you actually want your customers' apps to work.
DLL's are bad variations of libaries since they are unable to operate on their own, to use them a further wrapper executable is called(automatically), which adds unrequired overhead and makes it much harder to tell which libraries are actually in use. Another less important flaw, is the inability for systems to truely share a library.
*nix systems avoid this by having libraries exist on the top level running on their own or under a larger wrapper(like kde-init ), the libraries may be shared by any applications, meaning only a single copy of each library is required, and you may at any time kill a single library with ease as required.
The registry is a great idea, except for the fact that it is used for so much, almost anything you install will use the registry, and a corrupt registry and render your operating system almost completely useless until it's fixed.
This is avoided in *nix systems by having multiple different files for different content, drivers are refered to via Xorg's config file, installed applications will be written to their own database, and keys or identification will often be written into a directory, rather than a single all purpose file. This reduces the likely hood of a serious failure and means that at any time you can probably still repair the system. If Xorg becomes corrupt you just reconfigure it, if the installed applications database becomes corrupt you can repair or rebuild it, and should an applications individual settings directory become corrupt you need only reinstall one application(and most good commercial aps should have a way to repair this anyway)
I was curious as to how does one go about finding undocumented APIs in Windows.
I know the risks involved in using them but this question is focused towards finding them and not whether to use them or not.
Use a tool to dump the export table from a shared library (for example, a .dll such as kernel32.dll). You'll see the named entry points and/or the ordinal entry points. Generally for windows the named entry points are unmangled (extern "C"). You will most likely need to do some peeking at the assembly code and derive the parameters (types, number, order, calling convention, etc) from the stack frame (if there is one) and register usage. If there is no stack frame it is a bit more difficult, but still doable. See the following links for references:
http://www.sf.org.cn/symbian/Tools/symbian_18245.html
http://msdn.microsoft.com/en-us/library/31d242h4.aspx
Check out tools such as dumpbin for investigating export sections.
There are also sites and books out there that try to keep an updated list of undocumented windows APIs:
The Undocumented Functions
A Primer of the Windows Architecture
How To Find Undocumented Constants Used by Windows API Functions
Undocumented Windows
Windows API
Edit:
These same principles work on a multitude of operating systems however, you will need to replace the tool you're using to dump the export table. For example, on Linux you could use nm to dump an object file and list its exports section (among other things). You could also use gdb to set breakpoints and step through the assembly code of an entry point to determine what the arguments should be.
IDA Pro is your best bet here, but please please double please don't actually use them for anything ever.
They're internal because they change; they can (and do) even change as a result of a Hotfix, so you're not even guaranteed your undocumented API will work for the specific OS version and Service Pack level you wrote it for. If you ship a product like that, you're living on borrowed time.
Everybody here so far is missing some substantial functionality that comprises hugely un-documented portions of the Windows OS RPC . RPC (think rpcrt4.dll, lsass.exe, csrss.exe, etc...) operations occur very frequently across all subsystems, via LPC ports or other interfaces, their functionality is buried in the mysticism incantations of various type/sub-type/struct-typedef's etc... which are substantially more difficult to debug, due to the asynchronous nature or the fact that they are destine for process's which if you were to debug via single stepping or what have you, you would find the entire system lockup due to blocking keyboard or other I/O from being passed ;)
ReactOS is probably the most expedient way to investigate undocumented API. They have a fairly mature kernel and other executive's built up. IDA is fairly time-intensive and it's unlikely you will find anything the ReactOS people have not already.
Here's a blurb from the linked page;
ReactOS® is a free, modern operating
system based on the design of Windows®
XP/2003. Written completely from
scratch, it aims to follow the
Windows® architecture designed by
Microsoft from the hardware level
right through to the application
level. This is not a Linux based
system, and shares none of the unix
architecture.
The main goal of the
ReactOS project is to provide an
operating system which is binary
compatible with Windows. This will
allow your Windows applications and
drivers to run as they would on your
Windows system. Additionally, the look
and feel of the Windows operating
system is used, such that people
accustomed to the familiar user
interface of Windows® would find using
ReactOS straightforward. The ultimate
goal of ReactOS is to allow you to
remove Windows® and install ReactOS
without the end user noticing the
change.
When I am investigating some rarely seen Windows construct, ReactOS is often the only credible reference.
Look at the system dlls and what functions they export. Every API function, whether documented or not, is exported in one of them (user, kernel, ...).
For user mode APIs you can open Kernel32.dll User32.dll Gdi32.dll, specially ntdll.dll in dependancy walker and find all the exported APIs. But you will not have the documentation offcourse.
Just found a good article on Native APIS by Mark Russinovich