On Windows, I have to build a relatively simple topology in ZeroMQ.
I have a process (let's call it a bridge) that recieves data from outside and introduces them in the ZeroMQ topology. I'd like to use a set of publishers (something like ipc:///bridge/entity1, ipc:///bridge/entity2, ipc:///bridge/entity3 and so on) but afaik, ZeroMQ does not support IPC transports on windows (due to the lack of named pipes in such OS).
So I've to move to a TCP transport, but I don't want to use one port for each entity: I'd like to use something like tcp:///bridge:12345/entity1, tcp:///bridge:12345/entity2 and so on.
However AFAIK, this is not possible with ZeroMQ.
Can you please point me to the right direction?
That's right, it's not possible to bind several ZeroMQ sockets to a single port.
Probably, your problem might be solved with a single PUB socket that publishes messages to different topics, and subscribers that connect with zmq_setsockopt(ZMQ_SUBSCRIBE, ...). Since ZeroMQ 3.x topic filtering is done on PUB side, so there won't be redundant data transmission (related question: ZeroMQ filtering at publisher)
Related
I'm trying to understand a low-level architecture of Microsoft's implementation of named pipes vs RPC. I understand that both can be used for client/server communication between applications on a local network. But why would I choose one over another?
PS. Is RPC a higher level implementation built on top of named pipes, or vice versa? I'm also trying to see which one would have less overhead.
Named pipes are a communications channel but they don't specify what data is to be sent (in a lot of ways they are a lot like either TCP or UDP on top of IP).
An RPC interface on the other hand specifies what data is needed for a successful communication and what data that communication will produce.
Note that RPC is capable of using a large number of communications channels and named pipes are just one of the possible options.
I assume that there are hundreds and thousands IoT devices that publish the data to the (broker)MQTT cluster via the MQTT protocol, behind the broker i have the data processing module which subscribe the data from the broker and maintain a status table for all these devices. The number of the devices is still rising, therefor I have to scale out the broker cluster and data processing module accordingly, for the MQTT broker such as Kafka/Rabbit MQ/Hive MQ can be scaled out very easily, but for the data processing module I'm not quite sure whether there is any best practice, or any framework/architecture can achieve this very easily:
I assume I have to create many daemon processes with hundreds and thousands threads to listen on the MQTT broker, the question is how to scale out these services dynamically?
Thanks.
One way of doing this would be using Node.js as it uses an event-driven approach and you don't have to deal with threads, etc.
I found this library for Node.js which is specific to MQTT:
https://www.npmjs.com/package/mqtt
You can use this to subscribe to different topics.
You may also find this project interesting:
http://nodered.org/
The other solution can be using Apache Kafka which has scalability as an important feature. However, the problem here is that Kafka does not support MQTT out of the box and has its own conventions. Therefore, there is a need for some sort of adapter to make them work together. For that, take a look at this:
using mqtt protocol with kafka as a message broker
I'm attempting to do a pub/sub architecture where multiple publishers and multiple subscribers exist on the same bus. According to what I've read on the internet, only one socket should ever call bind(), and all others (whether pub or sub) should call connect().
The problem is, with this approach I'm finding that only the publisher that actually calls bind() on the socket ever publishes messages. All of my publishers that call connect() seem to fail silently and don't actually publish any messages to the bus. I've confirmed this isn't a subscriber key issue, as I've written a simple "sniffer" app that subscribes to all messages on the bus, and it is only showing the publisher that called bind().
If I attempt multiple binds with the publisher, the "expected" zmq behavior of silently stealing the bus occurs with ipc, and a port in use error is thrown with tcp.
I've verified this behavior with ipc and tcp endpoints, but ultimately the full system will be using epgm. I assume (though of course may be wrong) that in this situation I wouldn't need a broker since there's no dynamic discovery occurring (endpoints are known, whether ipc, tcp, or epgm multicast).
Is there something I'm missing, perhaps a socket setting, that would be causing the connecting publishers to not actually send their data? According to the literature I've seen on the internet, I'm doing things the "correct" way but it still doesn't work.
For reference, my publisher class has the following methods for setting up the endpoint:
ZmqPublisher::ZmqPublisher()
: m_zmqContext(1), m_zmqSocket(m_zmqContext, ZMQ_PUB)
{}
void ZmqPublisher::bindEndpoint(std::string ep)
{
m_zmqSocket.bind(ep.c_str());
}
void ZmqPublisher::connect(std::string ep)
{
m_zmqSocket.connect(ep.c_str());
}
So ultimately, my question is this: What is the proper way to handle multiple publishers on the same endpoint, and why am I not seeing messages from more than one publisher?
It may or may not be relevant, but The 0MQ Guide has the following slightly enigmatic remark:
In theory with ØMQ sockets, it does not matter which end connects and which end binds. However, in practice there are undocumented differences that I'll come to later. For now, bind the PUB and connect the SUB, unless your network design makes that impossible.
I've not yet discovered where the "come to later" actually happens, but I don't use pub/sub so much, and haven't read the "Advanced Pub-Sub Patterns" part of the guide in great detail.
However, the idea of multiple publishers on a single end-point, to me, suggests the need for an XPUB/XSUB style broker; it's not about dynamic discovery, it's about single point of contact and routing. Ultimately, I think a broker-based topology would simplify your application, and make it easier to identify problems.
Your mistake was that you call a single publisher with bind and others with connect. This is not supported with plain PUB-SUB pattern.
Plain PUB-SUB in ZeroMQ supports only two scenarios (see the image below):
single publisher, multiple subscribers
single subscriber, multiple publishers
In both cases, the party that is "single" must bind and the party that is "multiple" must connect. Otherwise, if you want many-to-many, you can use XPUB-XSUB or some other pattern.
I'm just starting understanding and trying ZeroMQ.
It's not clear to me how could I have a two way communication between more than two actors (publisher and subscriber) so that each component is able both to read and write on the MQ.
This would allow to create event-driven architecture, because each component could be listening for an event and reply with another event.
Is there a way to do this with ZeroMQ directly or I should implement my own solution on top of that?
If you want simple two-way communication then you simply set up a publishing socket on each node, and let each connect to the other.
In an many to many setup this quickly becomes tricky to handle. Basically, it sounds like you want some kind of central node that all nodes can "connect" to, receive messages from and, if some conditions on the subscriber are met, send messages to.
Since ZeroMq is a simple "power-socket", and not a message queue (hence its name, ZeroMQ - Zero Message Queue) this is not feasible out-of-the-box.
A simple alternative could be to let each node set up an UDP broadcast socket (not using ZeroMq, just regular sockets). All nodes can listen in to whatever takes place and "publish" its own messages back on the socket, effectively sending it to any nodes listening. This setup works on a LAN and in a setting where it is ok for messages to get lost (like periodical state updates). If the messages needs to be reliable (and possibly durable) you need a more advanced full-blown message queue.
If you can do without durable message queues, you can create a solution based on a central node, a central message handler, to which all nodes can subscribe to and send data to. Basically, create a "server" with one REP (Response) socket (for incoming data) and one PUB (Publisher) socket (for outgoing data). Each client then publishes data to the servers REP socket over a REQ (Request) socket and sets up a SUB (Subscriber) socket to the servers PUB socket.
Check out the ZeroMq guide regarding the various message patterns available.
To spice it up a bit, you could add event "topics", including server side filtering, by splitting up the outgoing messages (on the servers PUB socket) into two message parts (see multi-part messages) where the first part specifies the "topic" and the second part contains the payload (e.g. temp|46.2, speed|134). This way, each client can register its interest in any topic (or all) and let the server filter out only matching messages. See this example for details.
Basically, ZeroMq is "just" an abstraction over regular sockets, providing a couple of messaging patterns to build your solution on top of. However, it relieves you of a lot of tedious work and provides scalability and performance out of the ordinary. It takes some getting used to though. Check out the ZeroMq Guide for more details.
I am looking for a way to send requests and receive call backs from another party.
The only gotcha is that we do not now how it will be designed/deployed on the receiver side.
We do have the text/JSON based messages defined and agreed upon.
Looked at RabbitMQ and others, but each requires a server that would need to be maintained.
Thanks,
RabbitMQ is pretty easy to maintain. You would use two queues, one for requests and the other for replies. Use the AMQP correlation_id header to tag requests and replies so that when a reply message is received it can be matched with the orginal request.
However, if a broker is not for you, then use ZeroMQ. It is a client library available for a dozen or more languages and it enforces messaging patterns over top of sockets. This means that your app does not have to do all the low level socket management. Instead you declare the socket as REQ/REP and ZeroMQ handles all the rest. You just send messages in any format you desire, and you get messages back.
I've used ZeroMQ to implement a memcache style application in Python using REQ/REP.
#user821692: You have to agree not only message format but also destination/transport protocol. For e.g. if both communicating parties has access to same queue physically located anywhere, then they can communicate pre-defined messages. You may also look of sending messages over HTTP..