EDIT: through another question on the forum, I learned that DeviceIoControl can be async, so question 4 is now just question 2
The extensive Windows driver documentation says little, that I've found, about how a client user-mode app can communicate directly to a specific device. I understand that normally such operations are managed by the Win32 api, but in the case of specific devices or (what I'm interested in) software drivers, I don't know many ways in which it can be done
The docs say that one can use CreateFile, ReadFile, WriteFile etc. to "open" the driver as a "file" and then r/w from/to it, maybe asynchronously if you want. That sounds good but it feels like that can't be the best option for everything, nor is it the only option. DeviceIoControl can have specific control codes and you can command a driver like that, but I can't see any async capabilities in the docs there.
In the driver docs, it's clear a driver must write its callbacks routines for dispatch calls which are sent to it, but I don't understand where these dispatch calls come from, or how a user-mode client might interact with that directly.
Using Valorant's Vanguard as an example software driver, I highly doubt it just r/w'd from a "file" in operation - it seems too abstract to be fast, or not specific enough for a complex system, as all you can do in fileapi.h is read, and write, without any real parametrisation - right?
My questions are:
Must a software driver write routines for all dispatch calls that the docs recommend even though they have nothing to do with hardware?
Are there other techniques than the R/W file api and the DeviceIoControl function to communicate with a specific (software) driver?
Are there efficient, "lean and mean" solutions, when our software driver is entirely custom to the targeted user app, as Vanguard was?
(ignore) Are the async R/W file operations the only way to get this done in a multi-threaded async manner, where the client submits many possibly overlapping calls, or can DeviceIoControl leverage threading and asynchronicity?
To answer your questions
No. Not all dispatch calls needs to be implemented for a software driver. I think only CREATE/CLOSE/DEVICE CONTROL needs to be implemented. You dont even need to implement unload but then you will not be able to unload the driver, which is required for testing. If there are any other required dispatch methods, you can simply return not implemented from those implementation.
Yes. You can use named pipe between driver and application as another way to communicate.
Not sure how much more lean can you get than just implementing the minimum dispatch methods.
You can use multiple threads and synchronous operations OR you can use single thread and asynchronous operation. Depends on what model is best for you.
Related
I need to do something relatively simple, and I don't really want to install a MOM like RabittMQ etc.
There are several programs that "register" with a central
"service" server through TCP. The only function of the server is to
call back all the registered clients when they all in turn say
"DONE". So it is a kind of "join" (edit: Barrier) for distributed client processes.
When all clients say "DONE" (they can be done at totally different times), the central server messages
them all saying "ALL-COMPLETE". The clients "block" until asynchronously called back.
So this is a kind of distributed asynchronous Observer Pattern. The server has to keep track of where the clients are somehow. It is ok for the client to pass its IP address to the server etc. It is constructable with things like Boost::Signal, BOOST::Asio, BOOST::Dataflow etc, but I don't want to reinvent the wheel if something simple already exists. I got very close with ZeroMQ, but non of their patterns support this use-case very well, AFAIK.
Is there a very simple system that does this? Notice that the server can be written in any language. I just need C++ bindings for the clients.
After much searching, I used this library
https://github.com/actor-framework
It turns out that doing this with this framework is relatively straightforward. The only real "impediment" to using it is that the library seems to have gotten an API transition recently and the documentation .pdf file has not completely caught up with the source. No biggie since the example programs and the source (.hpp) files get you over this hump. However, they need to bring the docs in sync with the source. In addition, IMO they need to provide more interesting examples on how to use c++ Actors for extreme performance. For my case it is not needed, but the idea of actors (shared nothing) in this use-case is one of the reasons people use it instead shared memory communication when using threads.
Also, getting used to the syntax that the library enforces (get used to lambdas!) if one is not used to state of the art c++11 programs it can be a bit of a mind-twister at first. Then, the triviality of remembering all the clients that registered with the server was the only other caveat.
STRONGLY RECOMMENDED.
In its simplest form, COM allows you to instantiate C++-like classes from DLL in your application. Basically it's a glorified wrapper around LoadLibrary and some conventions regarding the interface. This is called using an in-process component.
But COM also supports out-of-process components. If you instantiate a class from such a component, COM starts a new process. Your objects live in said process, and are marshalled transparently over to you, so you don't care too much about where they live. They might even be on a different computer (DCOM). You can also fetch objects from already running applications. A well-known example is controlling MS Office via a script. This is called Automation (formerly OLE Automation, and there is a bit of confusion around what exactly this term encompasses).
There are a couple of nice articles explaining how (in-process) COM works low-level (e.g. COM from scratch. I'd like to know how it works when your component is out-of-process. Especially, what IPC does COM use beneath the hood to communicate between the processes? Window messages, shared memory, sockets, or something else? MSDN lists COM as an IPC method by itself, but I'm guessing it has to use something else underneath. Are different IPC methods used in different cases (instantiating an OOP component from C++, accessing an Excel document from VBScript, embedding a document in another via OLE)? It seems like it is all the same underlying technology. And lastly, how does marshalling fit in the picture? I believe it is neccessary to serialize method parameters for transmitting between processes, correct?
According to this MSDN article, it's RPC.
When you instantiate an OOP component, the COM subsystem generates an in-process proxy. This proxy is responsible for packing parameters and unpacking return values. It also generates a stub in the server process, which, expectably, unpacks parameters and packs return values.
Interestingly enough, the whole marshaling process can be customized, by implementing IMarshal.
DCOM was originally added as an extension to COM, precisely for cross apartment calls. Note cross apartment calls are not always from process to process. A process can have many apartments (0 or 1 MTA and/or 0 to n STAs, etc.) . There is at least one apartment per process, etc.
DCOM, some kind of a "middleware", needed a technology for all this low-level work: data representation, caller/callee convention, memory management, wire marshaling, session handling, security, error handling, etc. so Microsoft naturally used the in-house implementation of DCE/RPC: MSRPC. Note that as Microsoft says on its site,
"With the exception of some of its advanced features, Microsoft RPC is
interoperable with other vendors’ implementations of OSF RPC."
There was some tentative work to have all this implemented by other vendors, but they were basically killed by the rise of the internet and HTTP.
Also, note this RPC uses Windows Messages for STA apartement messages. I suggest you read carefully this document (not available any more on Microsoft site, shame on them :-) for more details:
DCOM Architecture by Markus Horstmann and Mary Kirtland - July 23, 1997 .
See also this interesting case study about a DCOM/RCP issue that should tell you a lot of how RPC over Windows message works under the scene: Troubleshooting a DCOM issue: Case Study
I am trying to port Wine 1.7.13 to modern Cocoa. I am considering running Windows binaries in an XPC service’s process, for security isolation and crash-proofing. However, there is one problem: To the best of my knowledge, XPC services are singletons. Only one XPC service process is allowed to be running at a time. This is a problem because, if I use threads to enable multiple Windows binaries to run at once, a segfault or other hard crash in one Windows binary would cause all the other binaries to crash with it.
As mentioned here, it is generally understood that the above assertion is true. If that is so, it would seem that I cannot implement this sort of isolation within a single XPC service process.
My other alternative is to use sandbox inheritance (having the GUI application fork and using more traditional IPC to have the Windows processes talk to each other) instead of an XPC service. What are the pros and cons of using that instead of an XPC service? I understand that processes that inherit their parent’s sandbox does not get to have its own entitlements. What other drawbacks are there?
I also understand that Apple discourages the use of sandbox inheritance in favor of XPC, but it is still an available design decision. They must have kept it around for a reason. Would a sandboxed Mac App Store app be able to use sandbox inheritance in this fashion?
I am going through the same decision. I had my heart set on XPC services, but upon discovering that there would be a single XPC Service with multiple connections, I cannot use them (my XPC Service will use plug-ins provided by third-parties so I want to keep them apart, and also the XPC Service will use libraries that might not clean-up properly, so I want to be able to dispose of them while keeping the UI stable - well I shouldn't have to justify this - I want one-process-per-job and that's that).
I am considering the normal sub-process model using posix_spawn() (I think this behaves better than fork() WRT to Sandboxing), CocoaAsyncSocket for the comms. I am going to see if I can replace the use of TCP/IP in CocoaAsynSocket with UNIX sockets to speed-up up comms (with the intent of contributing it back to the project if this works out). (UPDATE: this has already been done, some time ago by github user #jdiehl. See his socketUN branch and the discussion in issue #88 of the upstream repo).
For data marshalling I will use Google Protocol Buffers (UPDATE #2: Nope; not worth the hassle when NSKeyedArchiver and NSKeyedUnarchiver provide everything required out-of-the box. They may not provide data as packed as Google Protocol Buffers, but they 1) Don't require writing and maintaining, 2) Allow any class to participate by implementing the NSCoding protocol, and 3) Don't have to solve the issue of cross-platform data exchange.
The only possible disadvantage I can see is I don't know if file bookmarks can be passed to the subprocess and used (i.e. the UI opens a file or has a file dragged to it and wants to give access to the file to the worker process). I will update this answer with whatever I learn. (FINAL UPDATE: Passing the URL bookmark across the UNIX domain socket works fine, and the bookmark doesn't even need to be a security-scoped bookmark for this to work. There are no more impediments to this alternative to XPC).
Your assertion is incorrect about sub-processes not having their own entitlements; they do and are embedded into the executable and it must have "inherits sandbox" set for the sub-process to work correctly.
And the end-of-the-day the one-xpc-service-per-app is a show stopper so you have no choice but to find an alternative.
Microsoft RPC provides an IPC mechanism that can be done in a function-calling manner. This has been extremely helpful for my project where my main service delegates tasks to a child process, and functions in the child process can be called as if they were implemented in the main service. That takes away the burden of having to serialize abstract data and define custom protocols when using other IPC mechanisms such as named pipes, sockets, protobuf, etc. I'm aware that RPC does use them internally.
I've read an article on implementing COM for Mac OS X which is probably the closest thing to what I need. If I find no other no other way of implementing the type of IPC I need, I'm probably going to go with COM, but I thought I'd make sure that I'm not missing anything.
Have a look at "XPC Services". From the documentation:
XPC services are managed by launchd and provide services to a single
application. They are typically used to divide an application into
smaller parts. This can be used to improve reliability by limiting the
impact if a process crashes, and to improve security by limiting the
impact if a process is compromised.
And later in that guide:
The NSXPCConnection API is an Objective-C-based API that provides a
remote procedure call mechanism, allowing the client application to
call methods on proxy objects that transparently relay those calls to
corresponding objects in the service helper and vice-versa.
My company is looking at implementing a new VPN solution, but require that the connection be maintained programatically by our software. The VPN solution consists of a background service that seems to manage the physical connection and a command line/GUI utilty that initiates the request to connect/disconnect. I am looking for a way to "spy" on the API calls between the front-end utilty and back-end service so that our software can make the same calls to the service. Are there any recommended software solutions or methods to do this?
Typically, communications between a front-end application and back-end service are done through some form of IPC (sockets, named pipes, etc.) or through custom messages sent through the Service Control Manager. You'll probably need to find out which method this solution uses, and work from there - though if it's encrypted communication over a socket, this could be difficult.
Like Harper Shelby said, it could be very difficult, but you may start with filemon, which can tell you when certain processes create or write to files, regmon, which can do the same for registry writes and reads, and wireshark to monitor the network traffic. This can get you some data, but even with the data, it may be too difficult to interpret in a manner that would allow you to make the same calls.
I don't understand why you want to replace the utility, instead of simply running the utility from your application.
Anyway, you can run "dumpbin /imports whatevertheutilitynameis.exe" to see the static list of API function names to which the utility is linked; this doesn't show the sequence in which they're called, nor the parameter values.
You can then use a system debugger (e.g. Winice or whatever its more modern equivalent might be) to set breakpoints on these API, so that you break into the debugger (and can then inspect parameter values) when the utility invokes these APIs.
You might be able to glean some information using tools such as Spy++ to look at Windows messages. Debugging/tracing tools (Windbg, or etc.) may allow you to see API calls that are in process. The Sysinternals tools can show you system information to some degree of detail of usage.
Although I would recommend against this for the most part -- is it possible to contact the solution provider and get documentation? One reason for that is fragility -- if a vendor is not expecting users to utilize that aspect of the interface, they are more likely to change it without notice.