Related
It seems that the Win32 API (the C API for native Windows applications) is becoming more and more overtaken by more modern frameworks and toolkits, including Microsoft's own WPF and Qt.
If the programming language is not a concern -- if you're not set on a managed environment, or a functional programming style, etc. -- does Win32 API bring anything to the table? Is there any functionality that one can implement with Win32 API that's not available with WPF or other frameworks?
I know it's possible to mix Win32 code into WPF/managed software, so one doesn't have to choose one or the other. But what are some examples of needing to break out Win32 API when developing a program in a higher-level language/framework?
Another more specific example is "windows hooks".
I needed to hook some socket programs at some point and the only possible way was windows api.
To elaborate i wanted to receive all communication received on some listening socket on a different one. Doing this requires hooks
Certainly.
All the frameworks are written in terms of the Win32 API. The frameworks cover 80-95% of what programmers need to do, but if you need really low-level control over something, you'll need to drop to the underlying Win32 API. Some examples would be:
precise control over text rendering (via DirectWrite),
detailed control over speech recognition using SAPI (there are literally dozens of interfaces not exposed through System.Speech),
low-level networking code (i.e., anything not HTTP related),
Practically anything audio related, if you're interested in performance.
... and don't forget about direct hardware access like "WinUSB" and debugging functionality (writing programs that act as debuggers).
The Win32 API is nowhere close to be taken over by any framework at the moment.
If that was the case, most of the API would not be updated by Microsoft. Instead, lots of new interfaces are added and updated.
There are framekorks like Qt, but, unless what you need to create is trivial, you will eventually use the API, especially for new graphics libraries, audio, video, usb, ribbon, sockets, network, COM automation, biometrics, encryption, digital signatures, security, scripting, etc.
Actually, most libraries are quite outdated and, while you can create an application that relies mostly on worker threads and not on the interface, building a nice, modern and useful application today certainly requires the API. So investing on a framework that would only cover a minimum part of your application is not worth the learning curve, these frameworks mostly target really new and unexperienced windows developers.
I'm new to programming Linux kernel modules, and many getting started guides on the topic include little information about how to build a kernel module which will run on many versions and CPU platforms of Linux. Most of the guides I've seen simply state things like, "Linux doesn't ensure any ABI/API compatibility between versions." However, other OSes do provide these guarantees for major versions, and the guides are mostly targeting 2.7 (which is a bit old now).
I was wondering if there is any kind of ABI/API compatibility now, or if there are any standard ways to deal with versioning other than isolating the kernel-dependent bits of my code into files with a ton of preprocessor directives. (Also, are there any standard preprocessor symbols I should be using in the second case?)
There isn't a stable ABI for the kernel and most likely never will be because it'd make Linux suck. The reasons for not having one are all pretty much documented in that link.
The best way to deal with this is to get your driver merged upstream where it'll be maintained by other kernel developers.
As to being cross-platform, that pretty much comes free with the Linux kernel as long as you only use the standard, platform-independent functions provided in the API.
Linux, the ying and the yang. Tangrs answer is good; it answers your question. However, there is the linux compat projects. See the backports wiki. Basically, there are libraries that provide shim functionality for newer Linux ABI's which you can use to link your code. The KERNEL_VERSION macro that Eugene notes is inspected in a compat.h, and appropriate compat-2.6.38.h, etc are included where each version has either macros and/or library functions to provide a forward API.
This lets the Linux Wifi group write code for the bleeding edge kernel, while still making it possible to compile on older kernel versions.
I guess this answers the question,
if there are any standard ways to deal with versioning?
The compat library is not a panacea, but at least it is there and under development.
Open source - There are many mutations. They all have a different plan.
If a particular piece of software is made to be run on one platform and the programmer/company/whatever wants to port it to the other, what exactly is done? I mean, do they just rewrite linux or windows-specific references to the equivalent in the other? Or is an entire rewrite necessary?
Just trying to understand what makes it so cost-prohibitive that so many major vendors don't port their software to Linux (specifically thinking about Adobe)
Thanks
this is the point of a cross-platform toolkit like qt or gtk, they provide a platform-agnostic API, which delegates to whichever platform the program is compiled for.
some companies don't use such a toolkit, and write their own (for whatever reason - could well be optimisation-related), meaning they can't just recompile their code for another os.
There are also libraries available that ease, at least on a specific area, the port of Windows API calls to Linux. See the Windows to Linux porting library.
In my experience, there are three main reasons why it's cost-prohibitive to take a large existing program on one platform and port it to another:
it has (not necessarily purposely) extensively used some library or API (often GUI, but there are also plenty of other things) that turns out not to exist on the other platform
it has unknowingly become riddled with dependency on nonstandard features or oddities of the compiler or other tools
it was written by somebody who didn't know that you had to use some oddball feature to get things to work on the other platform (like a Linux library that isn't sprinkled with the right __declspec directives you need for a good Windows DLL).
It's much easier to write a cross-platform app if you consider that a design goal from the start, and I have three specific recommendations:
Use Boost—oodles of handy things you might ordinarily get from platform-specific APIs and libraries, but by using Boost you get it cross-platform.
Do all your GUI programming using a cross-platform library. My favorite these days is Qt, but there are other worthy ones as well.
Build and test every day on both platforms, never provide an opportunity for the code to develop a dependency on only one platform and discover it only too late.
There are many reasons why it may be very difficult to port an application to a different platform, most often it is because some interfaces the application uses to communicate with the system are not available, and one either has to implement them on their own, port a library your application depends on, or rewrite the application, so that it uses alternative functions. Most languages today are very portable across hardware architectures and operating systems, but the problem is with libraries, system calls and potentially other interfaces the OS (or platform) provides. To be more specific:
Compilers may differ in their configuration and the standard functions they provide. On Windows the most popular compiler for C/C++ is Visual Studio, while on unix it is gcc and llvm (in combination with the standard library glibc or BSD libc). They expect different flags, different forms of declaration, produce different file format of executables and shared libraries. Even though C and C++ have standard libraries, they are implemented differently across platforms. There are some systems whose aim is to make compilation portable, such as Autotools, CMake and SCons.
On top of standard libraries there are additional functions OS provides. On Windows they are covered by win32 API, on unix systems these are part of the POSIX standard, with various GNU, BSD and Linux specific extensions, and there are also plain system calls (the lowest-level interface between applications and the operating system). POSIX functions are available on Windows via systems such as cygwin and mingw, win32 API function are available on unix via Wine. The problem is that these implementations are not complete, and often there are minor (but important) differences.
Communication with the desktop system (in order to make a GUI interface) is done differently. On Linux this might be the X Window System (together with freedesktop libraries) or Wayland, while Windows has its own systems. There are GUI libraries which try to provide an abstract interface for these, such as Qt, GTK, wxWidgets, EFL.
Other services the OS provides, such as network communication may be implemented differently. On Windows many applications use .NET libraries, for which there is only limited support on unix systems. Some unix applications rely on Linux-specific features such as systemd, /proc, KMS, cgroups, namespaces. This limits portability even among unix systems (Linux, BSD systems, Mac OS X, ...). Even .NET libraries are not very compatible across different versions, and they might not be available on an older version of Windows or on embedded systems. Android and iOS have different interfaces entirely.
Web applications are usually the most portable, but HTML5 is a live standard, and many interfaces may not be available yet in some browsers/web engines. This requires the use of polyfills, but it is usually much less painful than the situation with "native" applications.
Because of all of these limitations, porting can be a pretty hard work and sometimes it is easier to create a new application from scratch, either specifically for the other platform, using cross-platform abstraction libraries/platforms (such as Qt or Java), or as a web application (potentially bundled in something like Electron). It is a good idea to use these from the beginning, but many programmers choose not to because the applications tend to look and behave differently from "native" applications on the platform, and they might also be slower and more restricted in the way they interact with the OS.
Porting a piece of software that has not been made platform-independant upfront can be an enormous task. Often, the code is deeply ingrained with non-portable APIs, whether 3rd party or just OS libraries. If the 3rd party vendor does not provide the API for the platform you are porting on, you are pretty much forced into a full rewrite of that functionality, or finding another 3rd party that is portable. This only can be awfully costly.
Finally, porting software also means supporting it on another platform, which means hiring some specialists, and training support to answer more complex queries.
In the end, such a process can be very costly, for very little additional sales. Sadly, the decision is easy: concentrate on new functionality on your current platform that you know your customers are going to pay for.
If the software was written for a single OS, a major rework is likely. The first step is to move absolutely all platform-specific code into a single area of the code base; this area should have little or no app-specific stuff. Then rewrite this isolated portion of the code for the new target OS.
Of course, this glosses over some extremely major implications. For instance, if your first version targeted the Win32 API, then any GUI code will be heavily tied to Windows, and to maintain any hope of preserving your sanity, you will need to move all that code to a cross-platform GUI framework like Qt or GTK.
Under Mono, you can write a C# Winforms program that works on both platforms. But to make that possible, the Mono team had to write their own Winforms library that essentially duplicates all of the functions of Winforms. So there is still no free lunch.
Most software is portable to some extent. In the case of a C app - there will be a lot of #ifdefs in the area, apart from path changes, etc.
Rarely windows/linux version of the same software don't share a common codebase - this would actually mean that they only share a common name. It's always harder to maintain more codebases, but I think that the actual problem with porting applications has little to do with the technical side and a lot with business side. Linux has much fewer users that Windows/OSX, most of them expect everything to be free as in beer or simply hate commercial software on some religious grounds.
When you come to think about it - most open source software is multiplatform, no matter what language was used to implement it. This speaks for itself...
P.S. Disclaimer - I'm an avid supporter of Free and Open source software, I don't want to insult anybody - I just share my perspective on the topic.
I was curious as to how does one go about finding undocumented APIs in Windows.
I know the risks involved in using them but this question is focused towards finding them and not whether to use them or not.
Use a tool to dump the export table from a shared library (for example, a .dll such as kernel32.dll). You'll see the named entry points and/or the ordinal entry points. Generally for windows the named entry points are unmangled (extern "C"). You will most likely need to do some peeking at the assembly code and derive the parameters (types, number, order, calling convention, etc) from the stack frame (if there is one) and register usage. If there is no stack frame it is a bit more difficult, but still doable. See the following links for references:
http://www.sf.org.cn/symbian/Tools/symbian_18245.html
http://msdn.microsoft.com/en-us/library/31d242h4.aspx
Check out tools such as dumpbin for investigating export sections.
There are also sites and books out there that try to keep an updated list of undocumented windows APIs:
The Undocumented Functions
A Primer of the Windows Architecture
How To Find Undocumented Constants Used by Windows API Functions
Undocumented Windows
Windows API
Edit:
These same principles work on a multitude of operating systems however, you will need to replace the tool you're using to dump the export table. For example, on Linux you could use nm to dump an object file and list its exports section (among other things). You could also use gdb to set breakpoints and step through the assembly code of an entry point to determine what the arguments should be.
IDA Pro is your best bet here, but please please double please don't actually use them for anything ever.
They're internal because they change; they can (and do) even change as a result of a Hotfix, so you're not even guaranteed your undocumented API will work for the specific OS version and Service Pack level you wrote it for. If you ship a product like that, you're living on borrowed time.
Everybody here so far is missing some substantial functionality that comprises hugely un-documented portions of the Windows OS RPC . RPC (think rpcrt4.dll, lsass.exe, csrss.exe, etc...) operations occur very frequently across all subsystems, via LPC ports or other interfaces, their functionality is buried in the mysticism incantations of various type/sub-type/struct-typedef's etc... which are substantially more difficult to debug, due to the asynchronous nature or the fact that they are destine for process's which if you were to debug via single stepping or what have you, you would find the entire system lockup due to blocking keyboard or other I/O from being passed ;)
ReactOS is probably the most expedient way to investigate undocumented API. They have a fairly mature kernel and other executive's built up. IDA is fairly time-intensive and it's unlikely you will find anything the ReactOS people have not already.
Here's a blurb from the linked page;
ReactOS® is a free, modern operating
system based on the design of Windows®
XP/2003. Written completely from
scratch, it aims to follow the
Windows® architecture designed by
Microsoft from the hardware level
right through to the application
level. This is not a Linux based
system, and shares none of the unix
architecture.
The main goal of the
ReactOS project is to provide an
operating system which is binary
compatible with Windows. This will
allow your Windows applications and
drivers to run as they would on your
Windows system. Additionally, the look
and feel of the Windows operating
system is used, such that people
accustomed to the familiar user
interface of Windows® would find using
ReactOS straightforward. The ultimate
goal of ReactOS is to allow you to
remove Windows® and install ReactOS
without the end user noticing the
change.
When I am investigating some rarely seen Windows construct, ReactOS is often the only credible reference.
Look at the system dlls and what functions they export. Every API function, whether documented or not, is exported in one of them (user, kernel, ...).
For user mode APIs you can open Kernel32.dll User32.dll Gdi32.dll, specially ntdll.dll in dependancy walker and find all the exported APIs. But you will not have the documentation offcourse.
Just found a good article on Native APIS by Mark Russinovich
--edit--
What application development frameworks/component models for Linux and Mac OS X are analogous to Windows Communication Foundation & COM, which provide high level IPC mechanisms for manipulating/communicating with software applications?
By high level communication model I mean RPC, publish/subscribe, etc.
I'm writing software for interaction devices which can be bound for manipulation of computation and data within some software. So far instance, I may have a control panel with physical sliders which are bound to various cells within a spreadsheet loaded in Excel, Calculate or Numbers. I have ways to sense events originating from the device, but the interaction subsystems(X11, Explorer, etc.) aren't extensible to handle new types of interaction events, so I need to capture and transport my own events to and from the software to which they're bound.
Also most of the applications I'd like to control cannot be modified to respond to these new types of events. And simulating hot-keys and shortcuts sequences aren't expressive enough, although it would take advantage of existing interaction subsystems. This is why I'm interested in platform specific frameworks, so I can take advantage of pre-existing hooks to control software in the wild.
Since this is for user interaction, it needs to have low latencies. I'd like to have the option of stateless, asynchronous communication also.
I've thought about this question and I realize I needed to ask two questions. One is about high level IPC, and the other is about object component models which allow you call methods on a software object.
--original question--
What communication/component frameworks provide functionality most similar to Windows Communications Foundation on Linux and Mac OS X?
My research group is beginning to develop some type of application development framework for interactive applications which involve new types of devices to manipulate/interact with software applications. We want use an adapter pattern to provide abstractions for device/device application/application or device/application communication so that software built with this framework will be portable across the 3 major OS platforms.
If the Linux apps are modern enough, you might be able to take advantage of D-Bus. It's the offspring of DCop from KDE primarily, and, to a lesser extent, Bonobo from GNOME, so if it's older, you might look one of those things up. Of course, that assumes that it uses one of those toolkits.
If it's even older than that or uses a toolkit without any high-level IPC (e.g. Motif (I think)), you may end up having to resort to pipes, sockets, fifos, and anything else I might have forgotten. That assumes, of course, that the app has some kind of hooks for that stuff.
Here is a list of guides related to networking on Mac OS X It covers different levels of networking from low level socket communications up http://developer.apple.com/documentation/Cocoa/Networking-date.html#doclist
It covers:
Core Foundation networking functions
to work with low-level BSD sockets.
Cocoa stream classes to read data
from and write data to streams.
Web Services client-side frameworks
available in Mac OS X.
Bonjour in Cocoa or Carbon
applications.
URL Loading System to manipulate URLs and transfer
URL contents
Distributed Objects Programming - communication between Objective-C objects in different
threads or different processes
Otherwise Mac OS X is fully Posix compliant unix system so you will find similar IPC system as found on Linux.
Or maybe you are after Mac OS X System Services, system that allows one application to access functionality of another application. Application advertise the services they provide. For more details read linked Apple documentation.
The nearest cousin would be libsoap and equivalent SOAP libraries in your language of choice.