I want to create a networked architecture where a master process is connected with some slave processes and exchange messages in this way:
Every slave should be able to send a message to the master. The master should be able to send a message to every subset of connected slaves.
i.e.
Master sends a message to Slave 1
Master sends a message to Slave 2 and Slave 3
Master sends a message to all Slaves
Slave 1 send a message to Master
These messages could have answers, but this can be handled at an higher level if there's not a dedicated way of doing this with ZeroMQ.
This should work using only one port.
With plain sockets I could make the Master bind on a port, accept connections, spawn a thread for every slave to handle incoming data and use the single connection to contact the single slave.
Since this architecture will use message-based communication, I think that ZeroMQ is the proper tool to implement it, but browsing the doc i can't find a way do that.
I'm going to write this in python, but the problem should be language agnostic.
Only using one port I think its best to use Dealer/Router :
Master would be a Router socket
Slave would be a Dealer socket
When slaves start send an 'I'm here message' to the master which should store identity (first frame received) in a list of known slaves.
The master then sends to a slave by prepending the identity and an empty frame to the message. (You can only send to one client at a time with a router socket, but trivial to write a function to take a message and a list of slave identities and send to each in turn.)
The identities of the slaves can either be set by you, using the setsockopt call on the Dealer sockets, or ZeroMQ will auto assign unique ones if you don't.
Related
I am using ZMQ to allow multiple clients to connect to a server and send a video stream of data using Pull/Push pattern. Now issue is the client can disconnect from the server. I need to ask is there a reliability concern, what happens if the data being sent to the server is dropped. Will the client will start sending video streams from the start or from where it was dropped in ZeroMQ. Does using Pull/Push pattern has buffers to accommodate this problem.
Usually ZeroMQ sockets will automatically reconnect when a connection is lost due to a temporarily network error.
For a PUSH socket the documentation states:
When a PUSH socket enters the mute state due to having reached the high water mark for all downstream nodes, or if there are no downstream nodes at all, then any send operations on the socket will block until the mute state ends or at least one downstream node becomes available for sending; messages are not discarded.
So a PUSH socket will buffer data and eventually block if it is not connected to a downstream PULL socket.
You should be able to test this by temporarily disabling or unplugging the network.
I'm new to ZeroMQ and trying to figure out a design issue. My scenario is that I have one or more clients sending requests to a single server. The server will process the requests, do some stuff, and send a reply to the client. There are two conditions:
The replies must go to the clients that sent the request.
If the client disconnects, the server should queue messages for a period of time so that if the client reconnects, it can receive the messages it missed.
I am having a difficult time figuring out the simplest way to implement this.
Things I've tried:
PUB/SUB - I could tag replies with topics to ensure only the subscribers that sent their request (with their topic as their identifier) would receive the correct reply. This takes care of the routing issue, but since the publisher is unaware of the subscribers, it knows nothing about clients that disconnect.
PUSH/PULL - Seems to be able to handle the message queuing issue, but looks like it won't support my plan of having messages sent to specific clients (based on their ID, for example).
ROUTER/DEALER - Design seemed like the solution to both, but all of the examples seem pretty complex.
My thinking right now is continuing with PUB/SUB, try to implement some sort of heartbeat on the client end (allowing the server to detect the client's presence), and when the client no longer sends a heartbeat, it will stop sending messages tagged with its topic. But that seems sub-optimal and would also involve another socket.
Are there any ideas or suggestions on any other ways I might go about implementing this? Any info would be greatly appreciated. I'm working in Python but any language is fine.
To prepare the best proposition for your solution, more data about your application requirements. I have made a little research about your conditions and connnect it with my experience about ZMQ, here I present two possibilities:
1) PUSH/PULL pattern in two direction, bigger impact on scalability, but messages from server will be cached.
Server has one PULL socket to register each client and get all messages from clients. Each message should have client ID to for server knowledge where send response.
For each client - server create PUSH socket to send responses. Socket configuration was sent in register message. You can use also REQ/REP pattern for register clients (assign socket number).
Each client has own PULL socket, which configuration was sent to server in register message.
It means that server with three clients required to (example port numbers in []):
server: 1 x PULL[5555] socket, 3 x PUSH[5560,5561,5562] sockets (+ optional 1 X REQ[5556] socket for registrations, but I think it depends how you prepare client identity)
client: 1 x PUSH[5555] socket, 1 x PULL[5560|5561|5562] (one per client) (+ optional 1 X REP[5556])
You have to connect server to multiple client sockets to send responses but if client disconnects, messages will not lost. Client will get their own messages when it reconnect to their PULL socket. The disadvantage is requirements of creating few PUSH sockets on server side (number of clients).
2) PUB/SUB + PUSH/PULL or REQ/REP, static cocket configuration on server side (only 2), but server has to prepare some mechanism for retransmit or cache messages.
Server create PUB socket and PULL or REQ. Client register it identity by PULL or REQ socket. server will publish all messages to client with this identity as filter. Server use monitor() function on PUB socket to count number of connected and disconnected clients (actions: 'accept' and 'disconnect'). After 'disconnect' action server publish message to all clients to register again. For clients which not re-register, server stop publish messages.
Client create SUB socket and PUSH or REQ to register and send requests.
This solution requires maybe some cache on server side. Client could confirm each message after get it from SUB socket. It is more complicated and have to be connected with your requirement. If you just would like to know that client lost message. Client could send timestamps of last message received from server during registration. If you need guarantee that clients get all messages, you need some cache implementation. Maybe other process which subscribe all messages and delete each confirmed by client.
In this solution server with three clients required to (example port numbers in []):
server: 1 x PUB[5555] socket, 1 x REP or PULL[5560] socket + monitoring PUB socket
client: 1 x SUB[5555] socket and own identity for filter, 1 x REQ or PUSH[5560] socket
About monitoring you could read here: https://github.com/JustinTulloss/zeromq.node#monitoring (NodeJS implementation, but Python will be similar)
I think about other patterns, but I am not sure that ROUTER/DEALER or REQ/REP will cover your requirements. You should read more about patterns, because each of it is better for some solutions. Look here:
official ZMQ guide (a lot of examples and pictures)
easy ROUTER/DEALER example: http://blog.scottlogic.com/2015/03/20/ZeroMQ-Quick-Intro.html
I'm trying to figure out under what conditions I would want to implement a remote queue versus a local one for 2 endpoint applications.
Consider this scenario: App A on Server A needs to send messages to App B on Server B via MQServer1.
It seems like the simplest configuration would be to create a single local queue on MQServer1 and configure AppA to put messages to the local queue while configuring AppB to get messages from the same local queue. Both AppA and AppB would connect to the same Queue Manager but execute different commands.
What sort of circumstances would require the need to install another MQ server (e.g. MQServer2) and configure a remote queue on MQServer1 which instead sends the messages from AppA over a channel to a local queue on MQServer2 to be consumed by AppB?
I believe I understand the benefit of remote queuing but I'm not sure when it's best used over a more simpler design.
Here are some problems with what you call the simpler design that you don't have with remote queuing:-
Time Independance - Server1 has to be available all the time, whereas with a remote queue, once the messages have been moved to Server B, Server A and Server 1 don't need to be online when App B wants to get its messages.
Network Efficiency - with two client applications putting or getting from a central queue, you have two inefficient network hops, instead of one efficient channel batched network connection from Server A to Server B (no need for Server 1 in the middle)
Network Problems - No network, no messages. Whereas when they are stored locally, any that have already arrived can be processed even while the network is down. Likewise, the application putting messages is also not held up by a network problem, the messages sit on the transmit queue easy to be moved, and the application can get on with the next thing.
Of course your applications should be written so that they aren't even aware of the difference, and it's just configuration changes that switch you from one design to the other.
Here we can have separate Queue Manager for both the application.Application A will send the message on to the queue defined on local Queue Manager, which in turn transmit it to the Transmission queue via defined channels (Need to do configuration for that in the QueueManager) which in turn send it to the Local queue of the Application B.
How do master and slave communicate is Mesos. Does the master run a webserver? Is it using HTTP or TCP/IP requests ?
Thanks for your reply
Master and worker (aka slave) exchange protobuf messages packed in HTTP/1.1. Master has a tiny built-in webserver that processes messages from workers and requests coming via HTTP endpoints. If you want to learn more, you can start with looking at mesos/3rdparty/libprocess/src/encoder.hpp:107
I am using ZEROMQ for distributed messaging application. Need to connect client (DEALER socket) to multiple servers (ROUTER socket on server side). What are my options on CLIENT side ?
Create DEALER socket on client side for each server endpoint (ROUTER socket).
Create only ONE DEALER socket on client side and add multiple endpoints.
I tried option 2 - connecting to multiple endpoints but message always goes to the first connected endpoint. followed following steps:
create DEALER socket
connect to first endpoint
then at run time, add another endpoint to the socket by using socket.connect(endpoint).
Do I need to reconnect?
In DEALER socket, there is no option to send message on a particular endpoint in case it is connected to multiple endpoints.
Any idea?
ZeroMQ encodes certain behaviors into socket types. These mainly deal with:
handling multiple peers
handling undeliverable messages
handling excessive throughput (HWM)
A DEALER socket is one that can connect to multiple peers, and uses LRU (least recently used, aka round-robin) to decide which peer gets each message. If you do not want this behavior, then you do not want a DEALER socket with multiple peers.
If you want to decide which peer gets a message, there are two options for this:
create a DEALER per peer, and send on the appropriate socket
create a single ROUTER connected to all peers, and use IDENTITY prefixes to route messages. You may need to pass IDENTITIES via a side channel, in order to use ROUTER-ROUTER connections.
at run time, add another endpoint to the socket by using socket.connect(endpoint). Do I need to reconnect?
No, you do not need to reconnect. You can add (and remove) peers at any time during the program.