Microsoft RPC provides an IPC mechanism that can be done in a function-calling manner. This has been extremely helpful for my project where my main service delegates tasks to a child process, and functions in the child process can be called as if they were implemented in the main service. That takes away the burden of having to serialize abstract data and define custom protocols when using other IPC mechanisms such as named pipes, sockets, protobuf, etc. I'm aware that RPC does use them internally.
I've read an article on implementing COM for Mac OS X which is probably the closest thing to what I need. If I find no other no other way of implementing the type of IPC I need, I'm probably going to go with COM, but I thought I'd make sure that I'm not missing anything.
Have a look at "XPC Services". From the documentation:
XPC services are managed by launchd and provide services to a single
application. They are typically used to divide an application into
smaller parts. This can be used to improve reliability by limiting the
impact if a process crashes, and to improve security by limiting the
impact if a process is compromised.
And later in that guide:
The NSXPCConnection API is an Objective-C-based API that provides a
remote procedure call mechanism, allowing the client application to
call methods on proxy objects that transparently relay those calls to
corresponding objects in the service helper and vice-versa.
Related
Does Marklogic supports backpressure or allow to send data in chunks that is reactive approach ?
'Reactive' is a fairly new term describing a particular incarnation of old concepts common in server and database technologies, but fairly new to modern client and middle-tier programming.
I am assuming the question is prompted by the need/desire to work within an existing 'Reactive' framework (such as vert.x or Rx/Java). For that question, the answer is 'no' - there is not an 'official' API which integrates directly with these frameworks to my knowledge. There are community APIs which I have not personally used, an example is https://github.com/etourdot/vertx-marklogic (reactive, vert.x marklogic API).
MarkLogic is a 'reactive' design internally in that it implements the functionality the modern 'reactive' term is used to describe -- but does not expose any standard 'reactive APIs' for this (there are very few standards in this area). Code running within MarkLogic server (xquery,javascript) implicitly benefits from this - although there is not an explicit backpressure API, a side effect of single threaded blocking IO (from the app perspective) is that the equivalent of 'back pressure' is implemented by implicit flow control of the IO APIS - you cannot over drive a properly configured ML server on a single thread doing blocking IO. Connections to an overloaded server will take longer and eventually time out ('backpressure' :)
Similarly, (most of) the external APIs (REST, XCC) are also blocking, single threaded.
The server core manages rate control via a variety of methods such as actively managing the TCP connection queue size, keep alive times, numbers of active threads etc.
In general the server does a very good job at this without explicit low level programming needed, balancing the latency across all clients. If this needs improving, the administration guides have good direction on how to tune the various parameters so the system behaves well on its own.
If you want to implement a per-connection client aware 'reactive' API you will need to implement it yourself. This can be done using the same techniques used for other blocking IO APis -- i.e. either use multiple threads or non-blocking IO. Some of the ML SDK's have provision for non-blocking IO or control over timeouts which can be used to implement a 'reactive' API.
Similarly, code running in the server itself (XQuery or JavaScript) can implement 'reactive' type behaviour by making use of the task queue -- as exposed by the xdmp:spawn-xxx apis. This is done in many libraries to manage bulk ingest. Care must be taken to carefully control the amount of concurrency as you can easily overload the server by spawning too many concurrent requests. Managing state is a bit tricky as there is a interaction/opposition between the transaction model and task creation -- the former generally presenting an idempotent view of data that can be incongruous with the concept of 'current' wrt asynchronous tasks.
In its simplest form, COM allows you to instantiate C++-like classes from DLL in your application. Basically it's a glorified wrapper around LoadLibrary and some conventions regarding the interface. This is called using an in-process component.
But COM also supports out-of-process components. If you instantiate a class from such a component, COM starts a new process. Your objects live in said process, and are marshalled transparently over to you, so you don't care too much about where they live. They might even be on a different computer (DCOM). You can also fetch objects from already running applications. A well-known example is controlling MS Office via a script. This is called Automation (formerly OLE Automation, and there is a bit of confusion around what exactly this term encompasses).
There are a couple of nice articles explaining how (in-process) COM works low-level (e.g. COM from scratch. I'd like to know how it works when your component is out-of-process. Especially, what IPC does COM use beneath the hood to communicate between the processes? Window messages, shared memory, sockets, or something else? MSDN lists COM as an IPC method by itself, but I'm guessing it has to use something else underneath. Are different IPC methods used in different cases (instantiating an OOP component from C++, accessing an Excel document from VBScript, embedding a document in another via OLE)? It seems like it is all the same underlying technology. And lastly, how does marshalling fit in the picture? I believe it is neccessary to serialize method parameters for transmitting between processes, correct?
According to this MSDN article, it's RPC.
When you instantiate an OOP component, the COM subsystem generates an in-process proxy. This proxy is responsible for packing parameters and unpacking return values. It also generates a stub in the server process, which, expectably, unpacks parameters and packs return values.
Interestingly enough, the whole marshaling process can be customized, by implementing IMarshal.
DCOM was originally added as an extension to COM, precisely for cross apartment calls. Note cross apartment calls are not always from process to process. A process can have many apartments (0 or 1 MTA and/or 0 to n STAs, etc.) . There is at least one apartment per process, etc.
DCOM, some kind of a "middleware", needed a technology for all this low-level work: data representation, caller/callee convention, memory management, wire marshaling, session handling, security, error handling, etc. so Microsoft naturally used the in-house implementation of DCE/RPC: MSRPC. Note that as Microsoft says on its site,
"With the exception of some of its advanced features, Microsoft RPC is
interoperable with other vendors’ implementations of OSF RPC."
There was some tentative work to have all this implemented by other vendors, but they were basically killed by the rise of the internet and HTTP.
Also, note this RPC uses Windows Messages for STA apartement messages. I suggest you read carefully this document (not available any more on Microsoft site, shame on them :-) for more details:
DCOM Architecture by Markus Horstmann and Mary Kirtland - July 23, 1997 .
See also this interesting case study about a DCOM/RCP issue that should tell you a lot of how RPC over Windows message works under the scene: Troubleshooting a DCOM issue: Case Study
I am trying to port Wine 1.7.13 to modern Cocoa. I am considering running Windows binaries in an XPC service’s process, for security isolation and crash-proofing. However, there is one problem: To the best of my knowledge, XPC services are singletons. Only one XPC service process is allowed to be running at a time. This is a problem because, if I use threads to enable multiple Windows binaries to run at once, a segfault or other hard crash in one Windows binary would cause all the other binaries to crash with it.
As mentioned here, it is generally understood that the above assertion is true. If that is so, it would seem that I cannot implement this sort of isolation within a single XPC service process.
My other alternative is to use sandbox inheritance (having the GUI application fork and using more traditional IPC to have the Windows processes talk to each other) instead of an XPC service. What are the pros and cons of using that instead of an XPC service? I understand that processes that inherit their parent’s sandbox does not get to have its own entitlements. What other drawbacks are there?
I also understand that Apple discourages the use of sandbox inheritance in favor of XPC, but it is still an available design decision. They must have kept it around for a reason. Would a sandboxed Mac App Store app be able to use sandbox inheritance in this fashion?
I am going through the same decision. I had my heart set on XPC services, but upon discovering that there would be a single XPC Service with multiple connections, I cannot use them (my XPC Service will use plug-ins provided by third-parties so I want to keep them apart, and also the XPC Service will use libraries that might not clean-up properly, so I want to be able to dispose of them while keeping the UI stable - well I shouldn't have to justify this - I want one-process-per-job and that's that).
I am considering the normal sub-process model using posix_spawn() (I think this behaves better than fork() WRT to Sandboxing), CocoaAsyncSocket for the comms. I am going to see if I can replace the use of TCP/IP in CocoaAsynSocket with UNIX sockets to speed-up up comms (with the intent of contributing it back to the project if this works out). (UPDATE: this has already been done, some time ago by github user #jdiehl. See his socketUN branch and the discussion in issue #88 of the upstream repo).
For data marshalling I will use Google Protocol Buffers (UPDATE #2: Nope; not worth the hassle when NSKeyedArchiver and NSKeyedUnarchiver provide everything required out-of-the box. They may not provide data as packed as Google Protocol Buffers, but they 1) Don't require writing and maintaining, 2) Allow any class to participate by implementing the NSCoding protocol, and 3) Don't have to solve the issue of cross-platform data exchange.
The only possible disadvantage I can see is I don't know if file bookmarks can be passed to the subprocess and used (i.e. the UI opens a file or has a file dragged to it and wants to give access to the file to the worker process). I will update this answer with whatever I learn. (FINAL UPDATE: Passing the URL bookmark across the UNIX domain socket works fine, and the bookmark doesn't even need to be a security-scoped bookmark for this to work. There are no more impediments to this alternative to XPC).
Your assertion is incorrect about sub-processes not having their own entitlements; they do and are embedded into the executable and it must have "inherits sandbox" set for the sub-process to work correctly.
And the end-of-the-day the one-xpc-service-per-app is a show stopper so you have no choice but to find an alternative.
I'm developing a new client-server app (.Net) and have up until now been using WCF, which suits the app's request-response approach nicely. However I've been asked to replace this with a socket-based solution, partly to support non-.Net clients, and future pub-sub/broadcast requirements (I realise WCF is capable, but there are other drivers behind the decision). Having failed miserably at writing my own async socket solution, I'm now looking at ZeroMQ.
My client app has a couple of background threads that periodically request data from the server. Additionally, certain UI actions (e.g. a button click) can trigger a message to the server. WCF made this easy - the code simply called the relevant method on a singleton WCF service proxy (actually I use the Castle Windsor WCF facility which gives me async calling capabilities, but that's probably irrelevant to my question).
I'm not too sure how this approach would translate to ZeroMQ, particularly with regards to managing the sockets - I'm very new to ZeroMQ and still reading the guide. Am I right in saying that I'll need a separate socket for each thread (i.e. the two b/g threads and the UI)? What about socket lifetime - do I create one each time I want to send/receive (presumably inefficient), or create the socket when the thread starts and reuse it for the entire lifetime of the thread?
One thing has to be very clear. ZMQ sockets can connect and talk to ZMQ sockets only.
This means that if I am building an distributed application whose components communicate to each other, I have liberty to choose any communication approach as external clients are not exposed to it.
Choosing ZMQ Sockets for such means is a good idea. It allows you to instantly build on many communication patterns like req/rep, push/pull, pub/sub etc and also build more complicated topologies using ZMQ devices.
How ever, This constraint is not to be taken lightly when external clients are concerned. This will enforce all external clients to use ZMQ sockets which might not be ideal. If one of the client happens to be a browser consuming your web services then you will need to provide services through regular client.
Is your client app using regular sockets?
Can it be re-written to use ZMQ sockets?
if not then don't use ZMQ sockets for external interface
but only for your internal component communication.
[Edit: Further notes]
ZMQ is a wrapper over sockets but that does a few things which are hard to get done by hand
It manages messaging at higher throughput by batching multiple messages at the same time
Optimizes use of socket at the same time
A socket can send messages to only one another socket, ZMQ socket can connect to multiple ZMQ sockets
ZMQ socket based solution can take immediate advantage of various patterns - REQ/REP, PUSH/PULL, PUB/SUB etc
How ever, it is common to mistake ZMQ to be a messaging queue.
Messaging Queue as available has other properties like message persistence and delivery guarantee etc by implementing a queue for storage.
ZMQ stands for "Zero Messaging Queue"
I have only been learning ZMQ in recent times and have been very happy to use it.
Checkout my mini tutorial on ZMQ and See if it make sense for you to use it:
http://learning-0mq-with-pyzmq.readthedocs.org/en/latest/
Regarding castle integration, check out what Henry's done on his fork:
https://github.com/hconceicao/clrzmq2/tree/master/src/integration
My company is looking at implementing a new VPN solution, but require that the connection be maintained programatically by our software. The VPN solution consists of a background service that seems to manage the physical connection and a command line/GUI utilty that initiates the request to connect/disconnect. I am looking for a way to "spy" on the API calls between the front-end utilty and back-end service so that our software can make the same calls to the service. Are there any recommended software solutions or methods to do this?
Typically, communications between a front-end application and back-end service are done through some form of IPC (sockets, named pipes, etc.) or through custom messages sent through the Service Control Manager. You'll probably need to find out which method this solution uses, and work from there - though if it's encrypted communication over a socket, this could be difficult.
Like Harper Shelby said, it could be very difficult, but you may start with filemon, which can tell you when certain processes create or write to files, regmon, which can do the same for registry writes and reads, and wireshark to monitor the network traffic. This can get you some data, but even with the data, it may be too difficult to interpret in a manner that would allow you to make the same calls.
I don't understand why you want to replace the utility, instead of simply running the utility from your application.
Anyway, you can run "dumpbin /imports whatevertheutilitynameis.exe" to see the static list of API function names to which the utility is linked; this doesn't show the sequence in which they're called, nor the parameter values.
You can then use a system debugger (e.g. Winice or whatever its more modern equivalent might be) to set breakpoints on these API, so that you break into the debugger (and can then inspect parameter values) when the utility invokes these APIs.
You might be able to glean some information using tools such as Spy++ to look at Windows messages. Debugging/tracing tools (Windbg, or etc.) may allow you to see API calls that are in process. The Sysinternals tools can show you system information to some degree of detail of usage.
Although I would recommend against this for the most part -- is it possible to contact the solution provider and get documentation? One reason for that is fragility -- if a vendor is not expecting users to utilize that aspect of the interface, they are more likely to change it without notice.