I started with zeromq just a few days ago. My aim is to design a publish subscribe system with multiple brokers (a network of brokers). I have read the relevant sections of the zeromq guide, and have written code for simple pub sub systems. If someone could please help me with the following questions:
From what I can conceive, the brokers(xpub-xsub sockets) will also have push/pull sockets to communicate the pub-sub messages. Is that correct? Any help in terms of how the brokers should communicate would be appreciated. Should there be any intermediaries between brokers?
Any design guidelines would be very helpful. Thank you.
The guide says that we should use xpub and xsub sockets when dynamic
discovery is required. Can someone please explain the difference
between the sockets: xpub and pub, and xsub and sub.
XPUB just means many publishers , compared to PUB, which means single publisher.
XSUB means many subscribers, compared to SUB, meaning single subscriber.
If you want to connect many subscribers to many publishers while still having the benefit dynamic discovery, you need a proxy in the middle; something like illustration below. PUB sockets sending messages to a proxy; the XSUB forwards message to XPUB, which then distributes those messages to all SUBs listening.
The code for creating such a proxy is simple (below), and the PUB and SUB ends are trivial, check the example code.
Socket xsub = ctx.createSocket(ZMQ.XSUB);
input.bind( "tcp://*:5500");
Socket xpub = ctx.createSocket(ZMQ.XPUB);
xpub.bind( "tcp://*:5600");
ZMQ.proxy( xsub, xpub, null);
From what I can conceive, the brokers will also have push/pull sockets
to communicate the pub-sub messages. Is that correct? Any help in
terms of how the brokers should communicate would be appreciated.
Should there be any intermediaries between brokers?
Check the guide for Inter-Broker Routing example
Related
My experience with setting up Tibco infrastructure is minimal, so please excuse any misuse of terminology, and correct me where wrong.
I am a developer in an organization where I don't have access to how the backend is setup for Tibco. However we have bandwidth issues between our regional centers, which I believe is due to how it's setup.
We have a producer that sends a message to multiple "regional" brokers. However these won't always have a client who needs to subscribe to the messages.
I have 3 questions around this:
For destination bridges: https://docs.tibco.com/pub/ems/8.6.0/doc/html/GUID-174DF38C-4FDA-445C-BF05-0C6E93B20189.html
Is a bridge what would normally be used, to have a producer send the same message to multiple brokers/destinations or is there something else?
It's not clear in the documentation, if a bridge exists to a destination where there is no client consuming a message, does the message still get sent to that destination? I.e., will this consume bandwidth even with no client wanting it?
If the above is true (and messages are only sent to destinations with a consumer), does this apply to both Topics and Message Selectors?
Is a bridge what would normally be used, to have a producer send the same message to multiple brokers/destinations or is there something else?
A bridge can be used to send messages from one destination to multiple destinations (queues or topic).
Alternatively Topics can be used to send a message to multiple consumer applications. Topics are not the best solution if a high level of integrity is needed(no message losses, queuing, etc).
It's not clear in the documentation, if a bridge exists to a destination where there is no client consuming a message, does the message still get sent to that destination? I.e., will this consume bandwidth even with no client wanting it?
If the bridge destination is a queue, messages will be put in the queue.
If the bridge destination is a Topic, messages will be distributed only if there are active consumers applications (or durable subscribers).
3 If the above is true (and messages are only sent to destinations with a consumer), does this apply to both Topics and Message Selectors?
This applies only to Topics (when there is no durable subscriber)
An alternative approach would be to use routing between EMS servers. In this approach Topics are send to remote EMS servers only when there is a consumer connected to the remote EMS server (or if there is a durable subscriber)
https://docs.tibco.com/pub/ems/8.6.0/doc/html/GUID-FFAAE7C8-448F-4260-9E14-0ACA02F1ED5A.html
I have the following setup:
zmq::proxy( acceptor, clients, nullptr );
My acceptor is a zmq::socket_type::stream andmy clients is a zmq::socket::type::dealer.
I am finding when the other end sends a large request (~ 16 [kB]), the request gets broken up and distributed to my dealer threads in pieces. One dealer gets the head of the message, others get pieces in the middle. I am not setting any special options so it seems like this is default zeromq behaviour.
I am using ZeroMQ 4.2.2.
Is there any way to override this behaviour and guarantee a delivery of a complete messages to my dealer threads?
#namdam deserved [+1] for posting version details
It there any wasy to override this... ?
Yes, kindly follow the API documented rules
A socket of type ZMQ_STREAM is used to send and receive TCP data from a non-ØMQ peer, when using the tcp:// transport. A ZMQ_STREAM socket can act as client and/or server, sending and/or receiving TCP data asynchronously.
Compatible peer sockets . . . . . none.
So either way, compose the proxy to handle compatible socket-archetypes ( w/o trying to hardwire ZMQ_STREAM into any other ZeroMQ native socket-archetype ) i.e. avoid using ZMQ_STREAM at all, or create a reading gateway, that decodes and mediates the ZMQ_STREAM compatible behaviour on one side and interfaces to other ZeroMQ native socket-archtypes on the other side of the gateway-logic.
If in doubts, you may enjoy a read into the main conceptual differencessketched in brief in the [ ZeroMQ hierarchy in less than a five seconds ] Section.
I've been reviewing Websockets messaging protocols. Looking at WAMP, it has the basic features I want. But in reading the docs, it appears that it requires a messages to route through the broker. Is this correct?
I am looking for real time messaging. While the broker role may be useful as a means of bringing together the publishers and subscribers, I would want the broker to only negotiate the connection, then hand-off sockets/IPs to the parties - allowing direct routing between the involved parties without requiring the broker to manage all the real time messaging. Can WAMP do this?
Two WebSocket clients (e.g. browsers) cannot talk directly to each other. So there will be an intermediary involved in any case.
WAMP is for real-time messaging. To be precise, WebSocket is for soft real-time. There are no hard real-time guarantees in any TCP based protocol running over networks.
Regarding publish & subscribe: a broker is always required, since it is exactly this part which decouples publishers and subscribers. If publisher would be directly connected to subscribers (not possible with 2 WebSocket client anyway, but ..), then you would introduce coupling. But decoupling a main point of doing PubSub anyway.
What exactly is your concern with having a broker (PubSub) or a dealer (RPC) being involved? Latency?
I was working with different patterns in zeromq in my project and right now i am using req/rep(later will shift to dealer/router) and pub/sub . The client sends messages to the server and the server publishes this information to other clients who have subscribed.
To use multiple sockets i followed the suggestions on this thread
Combining pub/sub with req/rep in zeromq and used zmq_poll . My server polls on req socket and pub socket.
While writing the code and while reading the above post i guessed that my pub socket will never get polledin and that's what i am observing now when i run the program. Only my request is polled in and publish is not happening at all.
If i don't use polling it works fine i.e as soon as the server gets the message i publish it.
So i am unclear on how polling will be useful in this pattern and how i can use it ?
You probably don't need to poll the pub socket. You certainly don't need to poll in on it - because that can never be triggered (pub sockets are send only).
The polling pattern might be useful in the case where you want to poll for "ready to send" on the req and the pub socket, allowing you to multiplex those channels. This will be particularly useful if/when you move to using a dealer/router.
The reason for that is that replacing req with a dealer (e.g.) can allow you to send multiple messages before receiving responses. Polling for inward and outbound messages will allow you to make maximum advantage of that.
Does ActiveMQ Apollo automatically convert messages between connected clients protocols ?
What I'm trying to do is have one producer use one protocol (STOMP or any other) but let the consumers choose their protocol as they wish, so they could be MQTT, STOMP, AMQP, OpenWire, etc.
Can Apollo convert messages depending on what protocol the consumer implemented ? If not, is there a different way than to have to implement all protocols as producers on different topics and tell clients which is their queue, based on protocol, something like
topic/stomp/chat
topic/mqtt/chat
Thanks!
You can't just convert messages depending on the consumer but, in theory with Apollo you can try Mirrored Queues feature to duplicate the queues/topics for a set of protocols.
http://activemq.apache.org/apollo/documentation/user-manual.html#Mirrored_Queues