I am developing a cross platform application (OS X, Linux, Windows) in C++11 that should provide a custom debug shell as an external application.
This debug shell is used to invoke methods that are exposed by the public interface of the application. Mainly it should allow to perform the same operations that are available from the app GUI.
Debug shell and application will run on the same machine (remote access would be a plus, so it is not necessary).
I was wondering which approach I could use to assure portability (different OS), performance and code reuse (I would like to share the same model for GUI and debug shell).
My first idea was to use gRPC and Google Protocol Buffer to access to the application interface from both GUI and debug shell, but I am afraid that such solution would add performance and design overhead.
Do you have any design and technical suggestion about?
gRPC should be fine for this use.
On Mac and Linux you could make use of unix domain sockets to do this.
Since you want to be portable to Windows too, you'd be better off binding to localhost, as is done in the helloworld examples in the github repository: https://github.com/grpc/grpc/tree/master/examples/cpp/helloworld.
gRPC uses very little CPU while nothing is happening, and the serialization and network overhead in making a request will be well within interactive usage latencies (expect 100's of microseconds at worst).
Related
I am trying to understand how code, regardless of the language, works. Specifically thinking about software that runs on Windows.
Is my understanding correct that every built in function of a particular language maps to an exposed function in the Windows API when writing software for the Windows platform?
I guess my question can be even more generally, can a language do anything outside of what the OS provides? If so, how? What is an example of this?
There is a theoretical and a practical answer to this.
Practical: yes.
The WinAPI is the API everything uses to do things on Windows. It’s stable and compatible between versions so you can write whatever you want with it and it will work on practically any version of Windows given you don’t use any APIs that aren’t present in an old version. There isn’t any other interface to talk to the operating system properly.
Any language or platform that wants to work on top of this will call WinAPI. C libraries, Python, etc all are written so that they work on top of it (often other languages use C or C++ libraries which use WinAPI).
Theoretical: no.
Windows itself includes a Native API which is the actual OS interface and WinAPI is built on top of this. It is not really used if it’s not necessary since it’s not really documented. It’s used in a couple of Windows components that need to run before the other parts of the system are running and you can build applications linking to this API. But since it’s undocumented it’s not really reasonable and may change whenever.
There is also the syscall level. Several Windows components provide the lower level services for operations done on the WinAPI level. You can write, for example, an assembly program and use the syscalls directly if you want to. Mostly you don’t want to so this is more of a theoretical rather than practical answer to different platforms communicating with the OS. These also may change based on the OS.
WinAPI is basically only one subsystem that runs on the NT kernel. For example, Windows Subsystem for Linux is another one which implements its own syscalls which are then translated to Windows ones. There has also been a POSIX subsystem previously.
So all in all it depends on which level you look at it, but the practical answer is yes. Everything practically runs on WinAPI.
Yes. Even if an application exits immediately, it uses a windows call.
So it is not only theoretical. In theory, as in practice, every Windows application uses the API, because there is nothing else to use.
Even if you try to rewrite each functionality you are about to use, you would eventually have to install a driver, and this also mean you would use the API.
There are a few programs like Proxifier that can force an exe to use a proxy. There are a few others, too. But the sites all look a bit shady. I don't even trust Proxifier tbh... So I'd like to know how these programs work. How do they do it? Is there a WinAPI function that can be used to do that? Or do you have to actually inject code into the processes?
I was only able to find functions to change the global proxy of windows. But some programs don't care what the global proxy says, they always try to connect directly, even if it's not possible...
Proxifier is based on LSP (layered service provider), but personally I never liked this technology because of often stability issues. However, besides LSP there are other possible approaches suitable to achieve the same functionality, you can find a short coverage of network filtering methods (including LSP) here: https://www.ntkernel.com/ndis-hooking-drivers-and-legacy-windows-systems/, however the document is a little bit out-of-date, I have written it in times of early Windows XP dawn and it does not cover WFP (Windows Filtering Platform), which replaced TDI, and NDIS Lightweight Filter, which replaced NDIS Intermediate and NDIS-hooking drivers. Both technologies were introduced by Windows Vista along with NDIS 6.0.
I think Wininet.dll is the canonical method for accessing HTTP from user mode programs on Windows. (The documentation for Windows Networking and Internet support is here. I didn't want to go through all of the doc, but I'm pretty sure Wininet.dll is the right one)
One method for doing a per process proxy, is to write a DLL that acts like Wininet.dll (and sits on top of the Windows' Wininet.dll). Your Wininet would have some sort of mechanism (registry, config file, etc.) to determine whether a particular process is to be proxied or not. If process isn't proxied then all calls go through to original Wininet, but if process is proxied then your Wininet does the redirection.
Another, somewhat similar, injection point is at the winsock layer (ws2_32.dll). (Back in the Windows 3.1, Win95 era, it was fairly common to replace winsock.dll (ws2_32 predecessor) by vendors of TCP/IP stacks.) Here's a case where the same concept is used to capture traffic at winsock layer. The article at the link has a nice diagram that illustrates the concept as well as implementation details of replacing ws2_32.dll.
The modern way to do this is to use Windows Filtering Platform.
https://en.wikipedia.org/wiki/Windows_Filtering_Platform
https://learn.microsoft.com/en-us/windows/win32/fwp/windows-filtering-platform-start-page
Windows Filtering Platform (WFP) is a set of API and system services
that provide a platform for creating network filtering applications.
The WFP API allows developers to write code that interacts with the
packet processing that takes place at several layers in the networking
stack of the operating system. Network data can be filtered and also
modified before it reaches its destination.
How can I compile a C++ program with a command-line interface and use it under Android 2.3.5 on my phone?
No - the model is completely different. Simple C++ programs are single threaded - they do whatever they have to do as quickly as they can in a single thread of execution and if they have to wait or block on something like retrieving data from the network then they just have to wait. They are given timeslices by a multitasking operating system and when they're finished they're finished.
In Android there is always one thread running which handles GUI interactions and passes the results into 'hooks' in your Activity instance. Anything that might block the GUI thread has to be farmed out to another thread, and call back on another method in your Activity. It's event-driven, and you have remarkably little control or certainty about things like object lifetime. So you need to program in a completely different way.
An emulator of some kind running as an Android app could - in principle - run C++ binaries compiled for a specific VM. But as far as I'm aware such an app doesn't exist and neither does the toolchain to produce such binaries. Google have discouraged such an approach too AFAIK. There are fully-fledged computer emulators but for obvious reasons they're mainly old 8-bit nostalgia fests :)
I'm a C++ programmer who recently got involved in Android programming and I'd recommend it. You'll think about programs in a different way from the single-threaded IFTT way you may be used to.
I have a custom CLI debugger for which I'm interested in a GUI. The debugger exposes an API with simple functions such as GetMemory(), SetMemory(), GetRegister(), Run(), Stop(), Address2Line() etc. through a TCP socket using a very simple protocol.
I'm looking for the easiest, fastest way of connecting it to a GUI. It seems there are many very good graphical debuggers, so after some research I think these are my best options:
Write a GDB translator - that will act as a gdbserver on one hand, translating all requests for the debugger, and also translate all events from the debugger to gdb compatible events. Then I can use any of the many gdb front-ends.
Write a Visual Studio Debug Engine
Write a plug-in for Eclipse (or some other open IDE)
Write a fresh GUI myself
So which will take the least effort / time ? Is there another way? Is there maybe a graphical debugger where I can easily define custom functions for debugging?
I would write an adapter so you can interact with something standard like GDB or Eclipse. Writing custom GUI code would be a waste of effort
How are operating systems typically debugged? They cannot be stepped through with a debugger like simple console programs, and the build times are too large to repeatedly make small changes and recompile the whole thing.
They aren't debugged as a multi-gigabyte programs! :)
If you mean the individual user-mode components, they can mainly be debugged just like normal programs and libraries (because they are normal programs/libraries!).
For kernel-mode components, though, each OS has its own mechanism; here is some information regarding the way that we do kernel debugging in Windows. It can be done using the help of another machine connected to the machine you're debugging, via a serial port or something. I'm not familiar with the process itself, but that's the gist of how they work. (You need to set some boot loader options so that the system is ready for the debugger to be connected as early as possible.)
It depends on which part of the operating system you're talking about. When I worked at MSFT, I worked on the IE team. We debugged IE and the shell (Windows Explorer) in Visual Studio and stepped through them line by line all day long. Though, sometimes, it's easier to debug using a command line tool such as NTSD.
If, however, you want to debug anything in Kernel land such as the OS kernel or device drivers, which I suspect is really what you're asking, then you must use the Kernel debugger. For Windows that is a command line tool called kd, and generally you run the debugger on one machine and remotely debug the target.
There are a whole set of techniques throughout history from flashing lights on the console, to the use of hardware devices like an ICE, to more modern techniques utilizing fairly standard debuggers. One technique that is more common among OS developers then application developers is the analysis of a core dump. Look at something like mdb on solaris for ideas about how Solaris kernel developers do some of their debugging. Also tracing technologies are used. Anywhere from fairly straightforward logging packages to more modern techniques like dtrace.
Also note that the techniques used depend on the layer of software. Initial boot tends to be a fairly hard place to get your fingers into. But after that the environment of modern operation systems looks more and more like the application setting you are use to. In the end, it is all code :)