I have server code on one box that needs to listen in on status coming from another box with about 10 chips with linux embedded in them. The 10 chips have their own ip addresses and each will send basically health status to the server which could (possibly) do something with it.
I would like the server just to passively listen and not have to send a response. So, this looks like a job for zmq's pub/sub. Where, each of the 10 chips have their own publication and the server would subscribe to each.
However, the server would need to know the well known address that each chip bound their publication to. But, in the field, these chips can be swapped or replace with a different ip address.
Instead, it's safer to have the chips know the server code's ip adddress.
What I would like a pub/sub where the receiver is the well known address. Or, a request/response pattern where the clients (the chips) send a messages to the server (the requests), but neither the server nor the chips need to send/receive a response.
Now, currently, there are two servers on the separate box. So, if possible I'd like a solution for one server and multiple servers.
Is this possible in zmq? And what pattern would that be?
thanks.
Yes, you can do this exactly the way you'd expect to do so. Just bind on your subscriber, then connect to that subscriber with your publishers. ZMQ doesn't designate which end should be the "server", or more reliable end, and which should be the "client", or more transient end, specifically for this reason, and this is an excellent reason to switch up the normal paradigm.
Edit to address the new clarification--
It should work fine with multiple servers. In general it would work like the following (the order of operations in this case is just to ensure no messages get lost, which is possible if the PUB socket starts sending messages before the SUB is ready):
Spin up server 1. Create SUB socket and bind on address:port.
Spin up server 2. Create SUB socket and bind on address:port.
Spin up a chip. That chip will create a PUB socket and connect on [server 1] address:port and connect on [server 2] address:port.
Repeat step (3) for the other nine chips.
Dual .SUB model
Oh yes, each .PUB-lishing entity may have numerous .SUB-s listening,
so having two <serverNode>-s meets the .PUB/.SUB-primitive Formal Communication Pattern ( one speaks - many listen )
As given above, each of your <serverNode> binds
.bind( aFixServer{A|B}_ipAddress_portNumber )
so as allow each .PUB-lishing <chipNode> to
.connect( anAprioriKnownServer{A|B}_bindingNode_ipAddress_portNumber )
And both <serverNode{A|B}> than .SUB-s to receive any messages from them.
Multi-Server model
As seen above, the {A|B} grammar is freely extensible to {A|B|C|D|...} so the principal messaging model will stand for any reasonable multi-server extension
Q.E.D.
Related
Currently I have a REP/REQ model up and running in my code.
However, I do not need for either to send replies. So replies are just wasting time.. I don't know if that matters in the real world or not.
Basically it looks like this.
Client PCs - Connect - REQ
these guys all connect to the Server and update the Server with Info they have on a regular basis. They don't care if the Server didn't receive a particular message, nor do they need any info back from the Server.
there are many of these guys but not excessive.. Let's say between 10 and 100.. all hitting the same server.. well probably not, probably it will be in groups.. a group of them hit one server, another group another.. clients would send messages several times a second. But not much more than several. I have not really done any timing, I don't know how really to time on my computer at less than 1-2 ms resolution so I really don't know what to expect or what is feasible in terms of performance and how many REQ clients can be served by 1 server REP.
Server PC - Bind - REP
this guy sits there running in a loop on his own separate thread waiting for REQs to come in. He sends replies to the REQs because he has to, not because he really wants to or needs to.
Alternate Models
from some googling it seems that PUSH PULL was recommended if you just want to sent messages and don't care about replies.
However, I couldn't figure out how to fit that into my architecture because the binds and connects seem to be reversed from what I need to have.. I would like my Bind to be on the Server because the Client "Connect" guys are not always available to be reached..
Solutions
1) good alternate model
A good alternate model that works and is relatively simple would be great. I'm not sure there really is one but apart from REP/REQ and PUB/SUB I don't really know too much about other models.
2) I'm worrying about nothing?
if message replies to REQ by REP are always going to be really fast and the reception of those replies by by REQ from REP also are really fast, then I guess I'm worrying about nothing. That would be good to know, so feel free to let me know if this is the case.
The Connection question
I don't really understand what connecting sockets does.
On a client REQ should I make a connect at the start of each loop before sending that one single message? Or should I connect before the loop to my socket that I also created before the loop?
I also don't understand what this means in terms of reliability or if I have to make special checks about connected status and reconnect, or if that is done automatically.
To sum up
I have a "global" context.. created at the start, disposed of at the end
This daddy context has 1 or 2 sockets (connected to the same address, including port) - I'm still debugging this dual socket on the same address thing so I'm not sure if that is ok or it just doesn't work that way - clarification would be nice
These context(s) are lazy initialized and outside the loop scope, so we are not recreating sockets on a regular basis
connect calls for the sockets occur currently outside of the loop scope, but I'm not sure if it is not better to have them inside the loop scope.
I think I'm getting mixed up here.. I think the dual sockets are on my PUB/SUB model .. 1 PUB with 2 SUB sockets on each client, but anyhow please let me know if that would be a problem as well.
If you do not need Request-Reply, do not use it.
Request-Reply is generally slow because you need a round trip to the server for every message. This means you get twice the network latency, which is the time a network package needs to travel over the network. That does not matter if network traffic is low but will become a bottleneck when the traffic is high, for example multiple messages per second.
As you already mentioned Push-Pull is a valid alternative for one-way traffic. With Push-Pull you create a Pull socket on the server and bind it to an endpoint (this is similar to the Reply socket). You create a Push socket on the clients and connect it to the server endpoint (this is similar to the Request socket).
If you send multiple messages from the client to the same server, you should connect only once. Setting up a network connection is a costly operation because it requires multiple network round trips, at least for TCP.
Hello this is preety straightforward question, i've seen and read around that people write that the address can point to multiple actors. I'm wondering why? What would be the use case of such.
Let's say router actor can have one address but then, on message it can dispatch the messages to multiple actors but each of it's children still might have only one address.
Thank you
First, let's analyze your example: indeed, a message payload can be delivered to multiple actors, as you have described (i.e. via router), as well as via pub-sub pattern .
I think, pub-sub is better because of convenience as there is no need of additional actor (router) in-between. Another reason, is less coupling in the pub-sub case: to subscribe and publish the message, there is need to know the address only, meanwhile in the router case, the donwstream actors have to know also at least router address and "subscription protocol", (e.g. donwstream.send<subscribe_me, message_type>(router_address, donwstream.address) or the donwstream actors have to know router class instance and (in the worse case) they have to be children of the router (or to be owned by router) - e.g. (router.subscribe(downstream_actor)).
The last reason on the point which might matter and depends on the implementation, that is is not the same message is delivered: in the pub-sub model the original message is delivered for multiple actors, in the router case multiple clones of the original message are delivered.
Second, the multiple patterns with actor multi-addressing are possible. Here are a few examples:
the actor's behavior might be different, whether a message of the same type is delivered to one or other actor's address, e.g. if actor owns some finite resource, say memory, and when it is about out of the resource, it might answer with rejections on it's "main" address, and still provide the resource on "critical" address. Without multi-addressing, you have to either share the resource between 2 different actors (discouraged) or to have 2 different types of messages (better).
in rotor (disclaimer: I'm the author), one actor can subscribe to the other service-actor address for some silent-side effects, i.e. for auditing or logging messages which are coming for the service-actor.
the message supervising can be implemented because of multi-addressing, e.g. in rotor the request-response pattern is implemented the following way: the request is routed via supervisor, which spawns a timer and a new address(X), and sends a payload copy to the original destination; then the response arrives at (X), it is just forwarded to the requesting-actor and timer is cancelled; however if the timer triggers, the supervisor creates a new message with error code (timeout) and delivers it back to the requesting-actor.
something like NAT is possible, i.e. when 2 nodes are connected, and here is need to deliver a message from actorA from node1 to actorB node2, the message can be actually just delivered to the unique address of NAT-actor on node1 representing actorB, serialized it special way and send over network. The reverse procedure will happen on node2. In that case NAT-actor will have multiple addresses (like ports in real routers), and still will be transparent for actorA and actorB.
Suppose TCP proxy has forwarded request back to the backend server. When it receives reply from the backend server, how does it knows which client to reply. What exact session information does a proxy stores?
Can anyone please throw some light on this
It depends on the protocol, it depends on the proxy, and it depends on whether transparency is a goal. Addressing all of these points exhaustively would take forever, so let's consider a simplistic case.
A network connection in software is usually represented by some sort of handle (whether that's a file descriptor or some other resource). In a C program on a POSIX system, we could simply keep two file descriptors associated with each other:
struct proxy_session {
int client_fd;
int server_fd;
}
This is the bare-minimum requirement.
When a client connects, we allocate one of these structures. There may be a protocol that lets us know what backend we should use, or we may be doing load balancing and picking backends ourselves.
Once we've picked a backend (either by virtue of having parsed the protocol or through having made some form routing decision), we initiate a connection to it. Simplistically, a proxy (as an intermediary) simply forwards packets between a client and a server.
We can use any number of interfaces for tying these two things together. On Linux, for example, epoll(2) allows us to associate a pointer to events on a file descriptor. We can provide it a pointer to our proxy_session structure for both the client and server side. When data comes in either of those file descriptors, we know where to map it.
Lacking such an interface, we necessarily have a means for differentiating connection handles (whether they're file descriptors, pointers, or some other representation). We could then use a structure like a hash table to look up the destination for a handle. The solution is found simply by being able to differentiate connections to each other, and holding some state that "glues" two connections together.
When building a pub-sub service using ZeroMQ on a Linux system, is there any way to enforce concurrent subscriber limits?
For example, I might want to create a ZeroMQ publisher service on a resource-limited system, and want to prevent overloading the system by setting a limit of, say, 100 concurrent connections to the tcp publisher endpoint. After that limit is reached, all subsequent connection attempts from ZeroMQ subscribers would fail.
I understand ZeroMQ doesn't provide notifications about connect/disconnect, but I've been looking for socket options that might allow such limits -- so far, no luck.
Or is this something that should be handled at some other level, perhaps within the protocol?
Yes, ZeroMQ is a Can-Do messaging framework:
Besides the trivial Formal Communication Pattern Framework elements ( the library primitives ), the strongest powers behind the ZeroMQ is the ability to develop one's own messaging system(s).
In your case, it is enough to enrich the scene with a few additional things ... a SUB-process -> PUB-process message-flow-channel, so as to allow PUB-side process to count a number of SUB-process instances concurrently connected and to allow for a disconnect ( a step delegated rather "back" to a SUB-process side suicside move, as the classical PUB-process, intentionally, has no instrumentation to manage subscriptions ) once a limit is dynamically achieved.
Plus add some dynamics for the inter-node signalling to start re-counting and/or to equip the SUB-process side(s) with a self-advertising mechanism to push-keepAliveSIG-s to the PUB-side and expect this signalling to be a weak and informative-only indication as there are many real-world collisions, where decentralised node simply fail to deliver a "guaranteed-delivery" message(s) and a well designed, distributed, low-latency, high-performance system has to cope well with this reality and have the self-healing state-recovery policies designed and in-built into own behaviour.
( Fig. courtesy imatix/ZeroMQ )
The ZeroMQ library can be thought of as a very powerful LEGO-tool-box for designing cool distributed systems, than a ready-made / batteries-included, stiff, quasi-solution-for-just-a-few-academic-cases ( well, it might be considered such, but just for some no-brainer's life, while our lives are much more colourful & teasing, aren't they ? )
So, "How to?"
Worth, definitely worth a few days to read the both of Pieter Hintjens' books & a few weeks for shifting one's mind to start designing with the ZeroMQ full-powers on one's side.
With just a few Python add-on habits ( a zmq.Context() early-setup, and not forgetting a finally: aContext.term() )
There's no way that I'm aware of to configure ZMQ to limit connections automatically... however, you have other options to accomplish what you're looking for. Perhaps the "traditional" way to accomplish this is with a second set of "network communication" sockets... perhaps REQ/REP from subscriber to publisher, asking for permission to connect.
You also have the option, depending on your version of ZMQ (and I've never used it and I can't find it in 5 minutes of searching, so I don't know how recent your version must be) to use XPUB/XSUB sockets, which can accomplish bi-directional communication. You can connect with XSUB, send a subscribe request, then receive a positive or negative response (you might have to play with your subscriber topics to communicate directly with just the single subscriber, I'm not sure), and react accordingly.
Either way, you'll be allowing a connection of some sort between the two systems and then either allowing it or terminating it depending on the situation. This could be less than completely ideal since you'll have to carve out a little overhead to handle connections that you'll be refusing... let's say you're saturated at 100 clients and all of a sudden get 100 new subscribe requests... you may or may not be able to cope with that sort of burst traffic.
You can test out the overhead in alternative communication mediums... like you could publish a webservice that indicates subscriber status that a client could check first, but that may not be any better to have clients connecting that way.
If you're absolutely at the limit of your resources, you'll have to set up a second server to handle subscriber status:
Server 1 is your publisher. You could set it up with a PUB socket and a REP socket.
Server 2 is your status server. It has a REQ socket. Have it subscribe to something like "system-status" or some such thing as that. It will also have your mechanism for communicating with new subscribers, be that a ZMQ socket or a web service or whatever else.
A client will request status from your status server. The status server will send a request to your publisher, which will increment it's subscriber count and reply with success, or keep its subscriber count and reply with failure. This success or failure will be communicated back to the subscriber, which will use that information to connect or not.
Disconnections will have to be communicated in a similar way... and you'll have to use some sort of heartbeating round-robin to confirm clients weren't a victim of catastrophic failure.
This will allow your publisher to make intelligent choices about whether it has resources or not. If you just want to set a static number, you don't even need the connection between the status server and the publisher, you can just keep count on the status server... but just to ensure the overall health of the network then it's probably best not to go that simplistic route.
Anyway, those are just some ideas to accomplish what you're looking for. ZMQ gives you options with which to craft your solutions moreso than actual solutions.
I currently have a pub/sub system running which allows clients to connect to a central message routing daemon, subscribe for a range of messages, and then start chattering away. The routing daemon tracks and maintains each subscriber's messages of interest (based on a simple tag) and delivers the appropriate messages of interest as each of the subscribers produce them. Essentially, each connection is considered a potential publisher OR subscriber AND USUALLY both, the daemon handles the routing and delivery as needed.
For example, three clients all connect and subscribe for their message tag(s) (MT) of interest:
Client 1(C1) subscribes to MT => 123
Client 2(C2) subscribes to MT => 123 & 456
Client 3(C3) subscribes to MT => 123 & 456 & 789
C1 produces MT 456: daemon delivers a copy to C2 and C3
C2 produces MT 123: daemon delivers a copy to C1 and C3 (not self)
C3 produces MT 999: daemon delivers it to none (nobody subscribed)
ZeroMQ came up in a discussion with a coworker and after tinkering with it for a few days I don't think I'm seeing the proper pattern for implementing/replacing the system that we currently have in place. Additionally, I would like to use EPGM in order to take advantage of the multicast gains and to eliminate the TCP based daemon, monkey in the middle, that I currently have.
Any suggestions?
It's possible to design a system like that using ZeroMQ. Basically speaking, you may create a daemon that binds two sockets: PULL to receive messages from clients and PUB to publish messages. Each of clients connects SUB socket and PUSH socket to server. EPGM might be used for PUB/SUB sockets, but PUSH/PULL sockets are still TCP.
The disadvantage of this design is that topic filtering and dropping out own messages must be done manually. For example, you might create message of three parts:
Topic
ID of producer
Message body
Client should read messages part by part immediately dropping tail of message it's not interested in. Working with PUB/SUB message envelopes is described in detail in this section of the guide: http://zguide.zeromq.org/page:all#Pub-Sub-Message-Envelopes. Client filtering shouldn't affect performance, since all PGM packets must be delivered to all connected receivers anyway.
This design is very simple yet pretty effective. It doesn't cover reliability, high availability, failure recovery and other important aspects - it's all doable with ZeroMQ and covered in the guide. Probably the best feature of ZeroMQ is the ability to start with something simple and add functionality as necessary without pain and/or major rewrites.
Something very similar (plus state snapshots, reliability and many more) is described in the chapter "Reliable Pub-Sub (Clone Pattern)" of the guide: http://zguide.zeromq.org/page:all#toc119
BTW, it's also possible to design p2p system with the central daemon used only as a name server, but it will be definitely more complex.