I am trying to understand how code, regardless of the language, works. Specifically thinking about software that runs on Windows.
Is my understanding correct that every built in function of a particular language maps to an exposed function in the Windows API when writing software for the Windows platform?
I guess my question can be even more generally, can a language do anything outside of what the OS provides? If so, how? What is an example of this?
There is a theoretical and a practical answer to this.
Practical: yes.
The WinAPI is the API everything uses to do things on Windows. It’s stable and compatible between versions so you can write whatever you want with it and it will work on practically any version of Windows given you don’t use any APIs that aren’t present in an old version. There isn’t any other interface to talk to the operating system properly.
Any language or platform that wants to work on top of this will call WinAPI. C libraries, Python, etc all are written so that they work on top of it (often other languages use C or C++ libraries which use WinAPI).
Theoretical: no.
Windows itself includes a Native API which is the actual OS interface and WinAPI is built on top of this. It is not really used if it’s not necessary since it’s not really documented. It’s used in a couple of Windows components that need to run before the other parts of the system are running and you can build applications linking to this API. But since it’s undocumented it’s not really reasonable and may change whenever.
There is also the syscall level. Several Windows components provide the lower level services for operations done on the WinAPI level. You can write, for example, an assembly program and use the syscalls directly if you want to. Mostly you don’t want to so this is more of a theoretical rather than practical answer to different platforms communicating with the OS. These also may change based on the OS.
WinAPI is basically only one subsystem that runs on the NT kernel. For example, Windows Subsystem for Linux is another one which implements its own syscalls which are then translated to Windows ones. There has also been a POSIX subsystem previously.
So all in all it depends on which level you look at it, but the practical answer is yes. Everything practically runs on WinAPI.
Yes. Even if an application exits immediately, it uses a windows call.
So it is not only theoretical. In theory, as in practice, every Windows application uses the API, because there is nothing else to use.
Even if you try to rewrite each functionality you are about to use, you would eventually have to install a driver, and this also mean you would use the API.
Related
Programming languages are platform independent, so why is it that we can't write a program that will run on both a PC and a Mac?
I want to develop a software and I'm on a mac, but I want it to run on a PC also, is it possible to develop such a software without having to require the user to download a special program that will make my program compatible with their computer?
The problem with this is that most software is dependent on the OS to handle some tasks. Yes, most programming languages are compatible with many platforms, but the OS provides a lot of support. When software uses the OS, it is sometimes called making a system call. If you want here is some more information.
Theoretically if you write your program in a 'high level language' it should be portable between two operating systems.
Practically however, the differences start from the very beginning - the API of choice, which works on one and does not on another(Such as, Mac's BSD API is incompatible with Win32 API) and boils down to the very last, which is, executable format, linker and loader. Each operating system has its own quirks.
Then comes the difference between the underlying architecture. Previously Macs ran on PowerPC architecture and Motorola architectures, while PCs used Intel. Since Macs have switched to Intel, there have been attempts at making cross platform executables inside Apple. Most attempts have failed.
There is however a way around your problem. You can use a very high level language such as Python to code and then distribute your python code to your PC friends.(But remember remember, you need a Python interpreter in your PC friends' computers for your program to run). I have successfully ported Python programs from Mac to PC with 0 code changes, and sometimes requiring only 2-4% code changes.
Simple answer: because language per se is not enough to make an application cross-platform. Also the framework it uses must be cross-platform too, frameworks are required for everything: handling data, displaying things, communicate with the hardware, multi threading, etc
This can usually be done:
by choosing a complete solution like Java, which will actually run on both platforms seamlessly and even with the same binary
by using C/C++ and cross platform libraries so that the same program can be compiled for both platforms (keep in mind that you can't distribute the real same binary, you need to compile two in any case)
by writing the logic of your program just using standard libraries and a standard language and then attach whatever you need for a specific platform just to build two different libraries. Of course you will have to wrap as much as you need so that the cross-platform part of your program doesn't know it
Mind that developing cross-platform applications which are not trivial examples like a game (for which there are plenty of cross-platform APIs) without using a complete solution like Java is not an easy task at all. Especially because most of the GUI you can build are strictly platform specific and relies on their own frameworks.
If you want an application to run "anywhere" your best option is a JIT type language which means that it compiles as it runs (Just In Time) for the platform that it's on. Really the language that stands out in my mind is Java (there's others and personally I don't like java). However, it's not quite that simple. For example a Window on a Mac computer has pieces and functions that a Window on a PC doesn't have and vice versa. And other operating systems don't even have windows or anything equivilant yet still run Java like Android or iOS for example or countless Linux Distros. And that's just a very basic example it gets MUCH MUCH harrier. Really the best way to build an application that can be used by anyone on just about any device is going web based.
The lesson is that if it was that simple a lot of people wouldn't have jobs and it never will be that simple, things will always progress and change and not everyone is going to want to do the same thing with their OS as someone else. There's a million ways to skin a cat and there's many more ways to implement something in an OS.
Yes, it is possible. But it is quite tricky. You need to:
Use a cross platform language (this is the easy part, many languages run on different plaftforms)
Avoid using any platform-specific features (usually not too hard, but needs testing)
Ensure you have cross platform libraries for all your dependencies (hard!)
Because of the library issue in particular, there are very few options that work across platforms. Your best options are probably:
A JVM language (like Java, Scala or Clojure) - because the JVM abstracts away from platform specific features, pure Java applications and libraries will run on any platform. Java probably has the best ecosystem of cross platform libraries and tools as a result.
JavaScript - quite a good option if you don't mind running in a browser. There are lots of quirks to deal with, but JavaScript is one of the best cross-platform options because of it's ubiquity.
I am into new project, which should use microcontroller. The easiest way to program it is using parallel port. But, there are few things I hope you can help me with. Oh, and the preferred language is C and platform Windows.
So, I studied LPT ports and Windows a bit, and from what I learned the most important is: Since Windows NT based systems, you cannot use instructions for direct port manipulation. This should be, because now programs are run in different privilege mode, which doesn't support the kind of instructions that are used by outport() function.
But at this point, I don´t understand a few things. First, I thought that Windows actually used privilege levels since first protected mode version, but that's the wrong assumption.
But more importantly, I thought that Windows has included functions for just about any hardware communication. I mean, anything you do in Windows these days, you just call windows functions which further call kernel services. I assumed that outport() doesn´t use any Windows function, and just makes the communication itself, which is prohibited now. But I am literally shocked that there is no system function to control parallel ports in modern Windows systems. At least that's what I read.
But even if I could get the control of parallel port, there comes my second problem.
For programming the controller, I need to follow special protocol, especially timing. But since Windows is multitasked, I worry about what if Scheduler switches to another app, and therefore when is the right time to switch signals on LPT, my program just will not be able to run.
Oh, by the way, I know I could use any 3rd-party apps, but I just like to be able to do it myself, or at least before I use some 3rd-party app, I want to know how it works. And yes, you can program some microcontrollers just by parallel port with some resistors, I know this for sure.
Thanks.
For windows you need to install a DLL which contains a driver to run at elevated privileges to get access to the HW ports.
You can find such a library at :
http://logix4u.net/Legacy_Ports/Parallel_Port/Inpout32.dll_for_Windows_98/2000/NT/XP.html
There are also some links to sample code.
I do not know which uController you are using, but I programmed in the past a variety of them and never had issues with timings, well for programming at least. The programming protocols are usually robust enough to deal with the jitter caused by multitasking. Just keep your clock edges and signa edges well separated and it should go fine.
If a particular piece of software is made to be run on one platform and the programmer/company/whatever wants to port it to the other, what exactly is done? I mean, do they just rewrite linux or windows-specific references to the equivalent in the other? Or is an entire rewrite necessary?
Just trying to understand what makes it so cost-prohibitive that so many major vendors don't port their software to Linux (specifically thinking about Adobe)
Thanks
this is the point of a cross-platform toolkit like qt or gtk, they provide a platform-agnostic API, which delegates to whichever platform the program is compiled for.
some companies don't use such a toolkit, and write their own (for whatever reason - could well be optimisation-related), meaning they can't just recompile their code for another os.
There are also libraries available that ease, at least on a specific area, the port of Windows API calls to Linux. See the Windows to Linux porting library.
In my experience, there are three main reasons why it's cost-prohibitive to take a large existing program on one platform and port it to another:
it has (not necessarily purposely) extensively used some library or API (often GUI, but there are also plenty of other things) that turns out not to exist on the other platform
it has unknowingly become riddled with dependency on nonstandard features or oddities of the compiler or other tools
it was written by somebody who didn't know that you had to use some oddball feature to get things to work on the other platform (like a Linux library that isn't sprinkled with the right __declspec directives you need for a good Windows DLL).
It's much easier to write a cross-platform app if you consider that a design goal from the start, and I have three specific recommendations:
Use Boost—oodles of handy things you might ordinarily get from platform-specific APIs and libraries, but by using Boost you get it cross-platform.
Do all your GUI programming using a cross-platform library. My favorite these days is Qt, but there are other worthy ones as well.
Build and test every day on both platforms, never provide an opportunity for the code to develop a dependency on only one platform and discover it only too late.
There are many reasons why it may be very difficult to port an application to a different platform, most often it is because some interfaces the application uses to communicate with the system are not available, and one either has to implement them on their own, port a library your application depends on, or rewrite the application, so that it uses alternative functions. Most languages today are very portable across hardware architectures and operating systems, but the problem is with libraries, system calls and potentially other interfaces the OS (or platform) provides. To be more specific:
Compilers may differ in their configuration and the standard functions they provide. On Windows the most popular compiler for C/C++ is Visual Studio, while on unix it is gcc and llvm (in combination with the standard library glibc or BSD libc). They expect different flags, different forms of declaration, produce different file format of executables and shared libraries. Even though C and C++ have standard libraries, they are implemented differently across platforms. There are some systems whose aim is to make compilation portable, such as Autotools, CMake and SCons.
On top of standard libraries there are additional functions OS provides. On Windows they are covered by win32 API, on unix systems these are part of the POSIX standard, with various GNU, BSD and Linux specific extensions, and there are also plain system calls (the lowest-level interface between applications and the operating system). POSIX functions are available on Windows via systems such as cygwin and mingw, win32 API function are available on unix via Wine. The problem is that these implementations are not complete, and often there are minor (but important) differences.
Communication with the desktop system (in order to make a GUI interface) is done differently. On Linux this might be the X Window System (together with freedesktop libraries) or Wayland, while Windows has its own systems. There are GUI libraries which try to provide an abstract interface for these, such as Qt, GTK, wxWidgets, EFL.
Other services the OS provides, such as network communication may be implemented differently. On Windows many applications use .NET libraries, for which there is only limited support on unix systems. Some unix applications rely on Linux-specific features such as systemd, /proc, KMS, cgroups, namespaces. This limits portability even among unix systems (Linux, BSD systems, Mac OS X, ...). Even .NET libraries are not very compatible across different versions, and they might not be available on an older version of Windows or on embedded systems. Android and iOS have different interfaces entirely.
Web applications are usually the most portable, but HTML5 is a live standard, and many interfaces may not be available yet in some browsers/web engines. This requires the use of polyfills, but it is usually much less painful than the situation with "native" applications.
Because of all of these limitations, porting can be a pretty hard work and sometimes it is easier to create a new application from scratch, either specifically for the other platform, using cross-platform abstraction libraries/platforms (such as Qt or Java), or as a web application (potentially bundled in something like Electron). It is a good idea to use these from the beginning, but many programmers choose not to because the applications tend to look and behave differently from "native" applications on the platform, and they might also be slower and more restricted in the way they interact with the OS.
Porting a piece of software that has not been made platform-independant upfront can be an enormous task. Often, the code is deeply ingrained with non-portable APIs, whether 3rd party or just OS libraries. If the 3rd party vendor does not provide the API for the platform you are porting on, you are pretty much forced into a full rewrite of that functionality, or finding another 3rd party that is portable. This only can be awfully costly.
Finally, porting software also means supporting it on another platform, which means hiring some specialists, and training support to answer more complex queries.
In the end, such a process can be very costly, for very little additional sales. Sadly, the decision is easy: concentrate on new functionality on your current platform that you know your customers are going to pay for.
If the software was written for a single OS, a major rework is likely. The first step is to move absolutely all platform-specific code into a single area of the code base; this area should have little or no app-specific stuff. Then rewrite this isolated portion of the code for the new target OS.
Of course, this glosses over some extremely major implications. For instance, if your first version targeted the Win32 API, then any GUI code will be heavily tied to Windows, and to maintain any hope of preserving your sanity, you will need to move all that code to a cross-platform GUI framework like Qt or GTK.
Under Mono, you can write a C# Winforms program that works on both platforms. But to make that possible, the Mono team had to write their own Winforms library that essentially duplicates all of the functions of Winforms. So there is still no free lunch.
Most software is portable to some extent. In the case of a C app - there will be a lot of #ifdefs in the area, apart from path changes, etc.
Rarely windows/linux version of the same software don't share a common codebase - this would actually mean that they only share a common name. It's always harder to maintain more codebases, but I think that the actual problem with porting applications has little to do with the technical side and a lot with business side. Linux has much fewer users that Windows/OSX, most of them expect everything to be free as in beer or simply hate commercial software on some religious grounds.
When you come to think about it - most open source software is multiplatform, no matter what language was used to implement it. This speaks for itself...
P.S. Disclaimer - I'm an avid supporter of Free and Open source software, I don't want to insult anybody - I just share my perspective on the topic.
Is there any fundamental differences in the WinAPI/Win32? Is there any additional knowledge required to take advantage of new OS features?
Are there any pitfalls which someone who's coded Win32 apps in the past might fall in?
I'm not talking about Silverlight, that's a whole different ball of wax. (I don't have the VS that supports that at work yet.)
Edit:
Drew has a pretty good answer so far, but what's critical for a programmer to know?
i.e. What should be in an appendix to Charles Petzold's book? (Theoretically)
There are of course lots of new APIs that you should be aware of to make sure you have the tools you need. Beyond that, there are some changes to note.
Philosophical changes
Large parts of the old win32 APIs focused on C-style APIs where handles were passed around. Nowadays, many of the new APIs being developed are COM-based, so boning up on COM and ATL would be worthwhile.
You might also want to take note of the new API style if you're writing your own libraries, which is a bit more consistent and avoids things like hungarian notation.
Replacements
Generally, don't assume that the methods you knew about 10 years ago are still state-of-the-art; they all still exist, so you won't necessarily be told you're doing it wrong. Check MSDN to see if it refers you to something better, and use the latest SDK so that you'll get deprecation warnings for some functions. Especially, make sure string functions you're using are secure.
Specifically, one 'replacement' API is Direct 2d, which is a DirectX-style API for UIs. If you're writing graphics code for Windows 7, you should consider Direct2d over GDI, which has a programming model that is compatible with, but very different than, GDI's. Direct 2d may be ported back to Vista.
Also, instead of using win32-style menuing, consider using the Ribbon, which will be available for Vista as well as Win7.
If you're using the common controls library, make sure to use v6, not the default of v5.
Finally, make sure you're not unnecessarily calling things that require admin privileges, as that will prompt UAC.
All I can think of for now.
There are new API's for each.
There is additional knowledge, though it may not be required, you should be familiar with 64-bit and multi-threaded application development to name a few. The higher level constructs such as Direct2D, .NET etc., etc., are what require the adjustment in knowledge, not necessarily the lower level APIs.
You have the choice: traditional C/C++ or use the newer .Net framework languages (C# / VB.net / Python.net and more). For the latter, knowing the framework is more important than the implementation. You're isolated (in general) from pointers, threading, buffers and memory management and apart from a few differences in syntax once you know the framework it's portable between languages (ie, you could easily pick up VB.net programming if you're a C# guy as most of what your apps will do is call parts of the framework). You can create a class in C#, use it in a VB.net program and reference the same class out of a Powershell cmdlet for example.
The old-style C interfaces are still for Win32 there but unless you've got a specific need to use them (legacy code, Direct X, device drivers for example) I'd look at the newer stuff. As for things like WPF, there isn't even a direct route in via unmanaged code - you have to jump through all sorts of ugly interop hoops.
Nothing special. The old stuff pretty much works as it did.
There are some new APIs, but nothing earth shattering (and following the old Win32 conventions).
So everything you know from Vista is still true for Win7.
Now, there are some new guidelines regarding user experience (touch screen, libraries (user experience stuff, not programmer stuff)), but the API style is the same.
Integrity levels are also a good thing to learn about. Depending on the nature of your application, if it attempts to do anything involving other processes that are running on the OS, it is important to know about this. This technology prevents processes at a lower integrity level from interacting with processes that are running at a higher integrity level. This includes messaging, hooks, DLL injection, opening handles, and many other techniques.
I was curious as to how does one go about finding undocumented APIs in Windows.
I know the risks involved in using them but this question is focused towards finding them and not whether to use them or not.
Use a tool to dump the export table from a shared library (for example, a .dll such as kernel32.dll). You'll see the named entry points and/or the ordinal entry points. Generally for windows the named entry points are unmangled (extern "C"). You will most likely need to do some peeking at the assembly code and derive the parameters (types, number, order, calling convention, etc) from the stack frame (if there is one) and register usage. If there is no stack frame it is a bit more difficult, but still doable. See the following links for references:
http://www.sf.org.cn/symbian/Tools/symbian_18245.html
http://msdn.microsoft.com/en-us/library/31d242h4.aspx
Check out tools such as dumpbin for investigating export sections.
There are also sites and books out there that try to keep an updated list of undocumented windows APIs:
The Undocumented Functions
A Primer of the Windows Architecture
How To Find Undocumented Constants Used by Windows API Functions
Undocumented Windows
Windows API
Edit:
These same principles work on a multitude of operating systems however, you will need to replace the tool you're using to dump the export table. For example, on Linux you could use nm to dump an object file and list its exports section (among other things). You could also use gdb to set breakpoints and step through the assembly code of an entry point to determine what the arguments should be.
IDA Pro is your best bet here, but please please double please don't actually use them for anything ever.
They're internal because they change; they can (and do) even change as a result of a Hotfix, so you're not even guaranteed your undocumented API will work for the specific OS version and Service Pack level you wrote it for. If you ship a product like that, you're living on borrowed time.
Everybody here so far is missing some substantial functionality that comprises hugely un-documented portions of the Windows OS RPC . RPC (think rpcrt4.dll, lsass.exe, csrss.exe, etc...) operations occur very frequently across all subsystems, via LPC ports or other interfaces, their functionality is buried in the mysticism incantations of various type/sub-type/struct-typedef's etc... which are substantially more difficult to debug, due to the asynchronous nature or the fact that they are destine for process's which if you were to debug via single stepping or what have you, you would find the entire system lockup due to blocking keyboard or other I/O from being passed ;)
ReactOS is probably the most expedient way to investigate undocumented API. They have a fairly mature kernel and other executive's built up. IDA is fairly time-intensive and it's unlikely you will find anything the ReactOS people have not already.
Here's a blurb from the linked page;
ReactOS® is a free, modern operating
system based on the design of Windows®
XP/2003. Written completely from
scratch, it aims to follow the
Windows® architecture designed by
Microsoft from the hardware level
right through to the application
level. This is not a Linux based
system, and shares none of the unix
architecture.
The main goal of the
ReactOS project is to provide an
operating system which is binary
compatible with Windows. This will
allow your Windows applications and
drivers to run as they would on your
Windows system. Additionally, the look
and feel of the Windows operating
system is used, such that people
accustomed to the familiar user
interface of Windows® would find using
ReactOS straightforward. The ultimate
goal of ReactOS is to allow you to
remove Windows® and install ReactOS
without the end user noticing the
change.
When I am investigating some rarely seen Windows construct, ReactOS is often the only credible reference.
Look at the system dlls and what functions they export. Every API function, whether documented or not, is exported in one of them (user, kernel, ...).
For user mode APIs you can open Kernel32.dll User32.dll Gdi32.dll, specially ntdll.dll in dependancy walker and find all the exported APIs. But you will not have the documentation offcourse.
Just found a good article on Native APIS by Mark Russinovich