Related
I am trying to understand how code, regardless of the language, works. Specifically thinking about software that runs on Windows.
Is my understanding correct that every built in function of a particular language maps to an exposed function in the Windows API when writing software for the Windows platform?
I guess my question can be even more generally, can a language do anything outside of what the OS provides? If so, how? What is an example of this?
There is a theoretical and a practical answer to this.
Practical: yes.
The WinAPI is the API everything uses to do things on Windows. It’s stable and compatible between versions so you can write whatever you want with it and it will work on practically any version of Windows given you don’t use any APIs that aren’t present in an old version. There isn’t any other interface to talk to the operating system properly.
Any language or platform that wants to work on top of this will call WinAPI. C libraries, Python, etc all are written so that they work on top of it (often other languages use C or C++ libraries which use WinAPI).
Theoretical: no.
Windows itself includes a Native API which is the actual OS interface and WinAPI is built on top of this. It is not really used if it’s not necessary since it’s not really documented. It’s used in a couple of Windows components that need to run before the other parts of the system are running and you can build applications linking to this API. But since it’s undocumented it’s not really reasonable and may change whenever.
There is also the syscall level. Several Windows components provide the lower level services for operations done on the WinAPI level. You can write, for example, an assembly program and use the syscalls directly if you want to. Mostly you don’t want to so this is more of a theoretical rather than practical answer to different platforms communicating with the OS. These also may change based on the OS.
WinAPI is basically only one subsystem that runs on the NT kernel. For example, Windows Subsystem for Linux is another one which implements its own syscalls which are then translated to Windows ones. There has also been a POSIX subsystem previously.
So all in all it depends on which level you look at it, but the practical answer is yes. Everything practically runs on WinAPI.
Yes. Even if an application exits immediately, it uses a windows call.
So it is not only theoretical. In theory, as in practice, every Windows application uses the API, because there is nothing else to use.
Even if you try to rewrite each functionality you are about to use, you would eventually have to install a driver, and this also mean you would use the API.
It seems that the Win32 API (the C API for native Windows applications) is becoming more and more overtaken by more modern frameworks and toolkits, including Microsoft's own WPF and Qt.
If the programming language is not a concern -- if you're not set on a managed environment, or a functional programming style, etc. -- does Win32 API bring anything to the table? Is there any functionality that one can implement with Win32 API that's not available with WPF or other frameworks?
I know it's possible to mix Win32 code into WPF/managed software, so one doesn't have to choose one or the other. But what are some examples of needing to break out Win32 API when developing a program in a higher-level language/framework?
Another more specific example is "windows hooks".
I needed to hook some socket programs at some point and the only possible way was windows api.
To elaborate i wanted to receive all communication received on some listening socket on a different one. Doing this requires hooks
Certainly.
All the frameworks are written in terms of the Win32 API. The frameworks cover 80-95% of what programmers need to do, but if you need really low-level control over something, you'll need to drop to the underlying Win32 API. Some examples would be:
precise control over text rendering (via DirectWrite),
detailed control over speech recognition using SAPI (there are literally dozens of interfaces not exposed through System.Speech),
low-level networking code (i.e., anything not HTTP related),
Practically anything audio related, if you're interested in performance.
... and don't forget about direct hardware access like "WinUSB" and debugging functionality (writing programs that act as debuggers).
The Win32 API is nowhere close to be taken over by any framework at the moment.
If that was the case, most of the API would not be updated by Microsoft. Instead, lots of new interfaces are added and updated.
There are framekorks like Qt, but, unless what you need to create is trivial, you will eventually use the API, especially for new graphics libraries, audio, video, usb, ribbon, sockets, network, COM automation, biometrics, encryption, digital signatures, security, scripting, etc.
Actually, most libraries are quite outdated and, while you can create an application that relies mostly on worker threads and not on the interface, building a nice, modern and useful application today certainly requires the API. So investing on a framework that would only cover a minimum part of your application is not worth the learning curve, these frameworks mostly target really new and unexperienced windows developers.
If a particular piece of software is made to be run on one platform and the programmer/company/whatever wants to port it to the other, what exactly is done? I mean, do they just rewrite linux or windows-specific references to the equivalent in the other? Or is an entire rewrite necessary?
Just trying to understand what makes it so cost-prohibitive that so many major vendors don't port their software to Linux (specifically thinking about Adobe)
Thanks
this is the point of a cross-platform toolkit like qt or gtk, they provide a platform-agnostic API, which delegates to whichever platform the program is compiled for.
some companies don't use such a toolkit, and write their own (for whatever reason - could well be optimisation-related), meaning they can't just recompile their code for another os.
There are also libraries available that ease, at least on a specific area, the port of Windows API calls to Linux. See the Windows to Linux porting library.
In my experience, there are three main reasons why it's cost-prohibitive to take a large existing program on one platform and port it to another:
it has (not necessarily purposely) extensively used some library or API (often GUI, but there are also plenty of other things) that turns out not to exist on the other platform
it has unknowingly become riddled with dependency on nonstandard features or oddities of the compiler or other tools
it was written by somebody who didn't know that you had to use some oddball feature to get things to work on the other platform (like a Linux library that isn't sprinkled with the right __declspec directives you need for a good Windows DLL).
It's much easier to write a cross-platform app if you consider that a design goal from the start, and I have three specific recommendations:
Use Boost—oodles of handy things you might ordinarily get from platform-specific APIs and libraries, but by using Boost you get it cross-platform.
Do all your GUI programming using a cross-platform library. My favorite these days is Qt, but there are other worthy ones as well.
Build and test every day on both platforms, never provide an opportunity for the code to develop a dependency on only one platform and discover it only too late.
There are many reasons why it may be very difficult to port an application to a different platform, most often it is because some interfaces the application uses to communicate with the system are not available, and one either has to implement them on their own, port a library your application depends on, or rewrite the application, so that it uses alternative functions. Most languages today are very portable across hardware architectures and operating systems, but the problem is with libraries, system calls and potentially other interfaces the OS (or platform) provides. To be more specific:
Compilers may differ in their configuration and the standard functions they provide. On Windows the most popular compiler for C/C++ is Visual Studio, while on unix it is gcc and llvm (in combination with the standard library glibc or BSD libc). They expect different flags, different forms of declaration, produce different file format of executables and shared libraries. Even though C and C++ have standard libraries, they are implemented differently across platforms. There are some systems whose aim is to make compilation portable, such as Autotools, CMake and SCons.
On top of standard libraries there are additional functions OS provides. On Windows they are covered by win32 API, on unix systems these are part of the POSIX standard, with various GNU, BSD and Linux specific extensions, and there are also plain system calls (the lowest-level interface between applications and the operating system). POSIX functions are available on Windows via systems such as cygwin and mingw, win32 API function are available on unix via Wine. The problem is that these implementations are not complete, and often there are minor (but important) differences.
Communication with the desktop system (in order to make a GUI interface) is done differently. On Linux this might be the X Window System (together with freedesktop libraries) or Wayland, while Windows has its own systems. There are GUI libraries which try to provide an abstract interface for these, such as Qt, GTK, wxWidgets, EFL.
Other services the OS provides, such as network communication may be implemented differently. On Windows many applications use .NET libraries, for which there is only limited support on unix systems. Some unix applications rely on Linux-specific features such as systemd, /proc, KMS, cgroups, namespaces. This limits portability even among unix systems (Linux, BSD systems, Mac OS X, ...). Even .NET libraries are not very compatible across different versions, and they might not be available on an older version of Windows or on embedded systems. Android and iOS have different interfaces entirely.
Web applications are usually the most portable, but HTML5 is a live standard, and many interfaces may not be available yet in some browsers/web engines. This requires the use of polyfills, but it is usually much less painful than the situation with "native" applications.
Because of all of these limitations, porting can be a pretty hard work and sometimes it is easier to create a new application from scratch, either specifically for the other platform, using cross-platform abstraction libraries/platforms (such as Qt or Java), or as a web application (potentially bundled in something like Electron). It is a good idea to use these from the beginning, but many programmers choose not to because the applications tend to look and behave differently from "native" applications on the platform, and they might also be slower and more restricted in the way they interact with the OS.
Porting a piece of software that has not been made platform-independant upfront can be an enormous task. Often, the code is deeply ingrained with non-portable APIs, whether 3rd party or just OS libraries. If the 3rd party vendor does not provide the API for the platform you are porting on, you are pretty much forced into a full rewrite of that functionality, or finding another 3rd party that is portable. This only can be awfully costly.
Finally, porting software also means supporting it on another platform, which means hiring some specialists, and training support to answer more complex queries.
In the end, such a process can be very costly, for very little additional sales. Sadly, the decision is easy: concentrate on new functionality on your current platform that you know your customers are going to pay for.
If the software was written for a single OS, a major rework is likely. The first step is to move absolutely all platform-specific code into a single area of the code base; this area should have little or no app-specific stuff. Then rewrite this isolated portion of the code for the new target OS.
Of course, this glosses over some extremely major implications. For instance, if your first version targeted the Win32 API, then any GUI code will be heavily tied to Windows, and to maintain any hope of preserving your sanity, you will need to move all that code to a cross-platform GUI framework like Qt or GTK.
Under Mono, you can write a C# Winforms program that works on both platforms. But to make that possible, the Mono team had to write their own Winforms library that essentially duplicates all of the functions of Winforms. So there is still no free lunch.
Most software is portable to some extent. In the case of a C app - there will be a lot of #ifdefs in the area, apart from path changes, etc.
Rarely windows/linux version of the same software don't share a common codebase - this would actually mean that they only share a common name. It's always harder to maintain more codebases, but I think that the actual problem with porting applications has little to do with the technical side and a lot with business side. Linux has much fewer users that Windows/OSX, most of them expect everything to be free as in beer or simply hate commercial software on some religious grounds.
When you come to think about it - most open source software is multiplatform, no matter what language was used to implement it. This speaks for itself...
P.S. Disclaimer - I'm an avid supporter of Free and Open source software, I don't want to insult anybody - I just share my perspective on the topic.
Is there any fundamental differences in the WinAPI/Win32? Is there any additional knowledge required to take advantage of new OS features?
Are there any pitfalls which someone who's coded Win32 apps in the past might fall in?
I'm not talking about Silverlight, that's a whole different ball of wax. (I don't have the VS that supports that at work yet.)
Edit:
Drew has a pretty good answer so far, but what's critical for a programmer to know?
i.e. What should be in an appendix to Charles Petzold's book? (Theoretically)
There are of course lots of new APIs that you should be aware of to make sure you have the tools you need. Beyond that, there are some changes to note.
Philosophical changes
Large parts of the old win32 APIs focused on C-style APIs where handles were passed around. Nowadays, many of the new APIs being developed are COM-based, so boning up on COM and ATL would be worthwhile.
You might also want to take note of the new API style if you're writing your own libraries, which is a bit more consistent and avoids things like hungarian notation.
Replacements
Generally, don't assume that the methods you knew about 10 years ago are still state-of-the-art; they all still exist, so you won't necessarily be told you're doing it wrong. Check MSDN to see if it refers you to something better, and use the latest SDK so that you'll get deprecation warnings for some functions. Especially, make sure string functions you're using are secure.
Specifically, one 'replacement' API is Direct 2d, which is a DirectX-style API for UIs. If you're writing graphics code for Windows 7, you should consider Direct2d over GDI, which has a programming model that is compatible with, but very different than, GDI's. Direct 2d may be ported back to Vista.
Also, instead of using win32-style menuing, consider using the Ribbon, which will be available for Vista as well as Win7.
If you're using the common controls library, make sure to use v6, not the default of v5.
Finally, make sure you're not unnecessarily calling things that require admin privileges, as that will prompt UAC.
All I can think of for now.
There are new API's for each.
There is additional knowledge, though it may not be required, you should be familiar with 64-bit and multi-threaded application development to name a few. The higher level constructs such as Direct2D, .NET etc., etc., are what require the adjustment in knowledge, not necessarily the lower level APIs.
You have the choice: traditional C/C++ or use the newer .Net framework languages (C# / VB.net / Python.net and more). For the latter, knowing the framework is more important than the implementation. You're isolated (in general) from pointers, threading, buffers and memory management and apart from a few differences in syntax once you know the framework it's portable between languages (ie, you could easily pick up VB.net programming if you're a C# guy as most of what your apps will do is call parts of the framework). You can create a class in C#, use it in a VB.net program and reference the same class out of a Powershell cmdlet for example.
The old-style C interfaces are still for Win32 there but unless you've got a specific need to use them (legacy code, Direct X, device drivers for example) I'd look at the newer stuff. As for things like WPF, there isn't even a direct route in via unmanaged code - you have to jump through all sorts of ugly interop hoops.
Nothing special. The old stuff pretty much works as it did.
There are some new APIs, but nothing earth shattering (and following the old Win32 conventions).
So everything you know from Vista is still true for Win7.
Now, there are some new guidelines regarding user experience (touch screen, libraries (user experience stuff, not programmer stuff)), but the API style is the same.
Integrity levels are also a good thing to learn about. Depending on the nature of your application, if it attempts to do anything involving other processes that are running on the OS, it is important to know about this. This technology prevents processes at a lower integrity level from interacting with processes that are running at a higher integrity level. This includes messaging, hooks, DLL injection, opening handles, and many other techniques.
I was curious as to how does one go about finding undocumented APIs in Windows.
I know the risks involved in using them but this question is focused towards finding them and not whether to use them or not.
Use a tool to dump the export table from a shared library (for example, a .dll such as kernel32.dll). You'll see the named entry points and/or the ordinal entry points. Generally for windows the named entry points are unmangled (extern "C"). You will most likely need to do some peeking at the assembly code and derive the parameters (types, number, order, calling convention, etc) from the stack frame (if there is one) and register usage. If there is no stack frame it is a bit more difficult, but still doable. See the following links for references:
http://www.sf.org.cn/symbian/Tools/symbian_18245.html
http://msdn.microsoft.com/en-us/library/31d242h4.aspx
Check out tools such as dumpbin for investigating export sections.
There are also sites and books out there that try to keep an updated list of undocumented windows APIs:
The Undocumented Functions
A Primer of the Windows Architecture
How To Find Undocumented Constants Used by Windows API Functions
Undocumented Windows
Windows API
Edit:
These same principles work on a multitude of operating systems however, you will need to replace the tool you're using to dump the export table. For example, on Linux you could use nm to dump an object file and list its exports section (among other things). You could also use gdb to set breakpoints and step through the assembly code of an entry point to determine what the arguments should be.
IDA Pro is your best bet here, but please please double please don't actually use them for anything ever.
They're internal because they change; they can (and do) even change as a result of a Hotfix, so you're not even guaranteed your undocumented API will work for the specific OS version and Service Pack level you wrote it for. If you ship a product like that, you're living on borrowed time.
Everybody here so far is missing some substantial functionality that comprises hugely un-documented portions of the Windows OS RPC . RPC (think rpcrt4.dll, lsass.exe, csrss.exe, etc...) operations occur very frequently across all subsystems, via LPC ports or other interfaces, their functionality is buried in the mysticism incantations of various type/sub-type/struct-typedef's etc... which are substantially more difficult to debug, due to the asynchronous nature or the fact that they are destine for process's which if you were to debug via single stepping or what have you, you would find the entire system lockup due to blocking keyboard or other I/O from being passed ;)
ReactOS is probably the most expedient way to investigate undocumented API. They have a fairly mature kernel and other executive's built up. IDA is fairly time-intensive and it's unlikely you will find anything the ReactOS people have not already.
Here's a blurb from the linked page;
ReactOS® is a free, modern operating
system based on the design of Windows®
XP/2003. Written completely from
scratch, it aims to follow the
Windows® architecture designed by
Microsoft from the hardware level
right through to the application
level. This is not a Linux based
system, and shares none of the unix
architecture.
The main goal of the
ReactOS project is to provide an
operating system which is binary
compatible with Windows. This will
allow your Windows applications and
drivers to run as they would on your
Windows system. Additionally, the look
and feel of the Windows operating
system is used, such that people
accustomed to the familiar user
interface of Windows® would find using
ReactOS straightforward. The ultimate
goal of ReactOS is to allow you to
remove Windows® and install ReactOS
without the end user noticing the
change.
When I am investigating some rarely seen Windows construct, ReactOS is often the only credible reference.
Look at the system dlls and what functions they export. Every API function, whether documented or not, is exported in one of them (user, kernel, ...).
For user mode APIs you can open Kernel32.dll User32.dll Gdi32.dll, specially ntdll.dll in dependancy walker and find all the exported APIs. But you will not have the documentation offcourse.
Just found a good article on Native APIS by Mark Russinovich