although I went through the guide I cant find a way to do the following:
we have n publishers,
we have m subscribers.
Each subscriber subscribes to certain type of msg(one publisher can send more than one kind of the message, multiple pubs can emit msgs of same type).
How to create N to N(or N to 1 to N) pattern in 0MQ that does that?
Pattern? A publisher will accept any number of subscribers, and your subscribers can connect to multiple publishers. I don't see any issue here, just go ahead and do it.
In the future, perhaps you'll be better served by stating what you want to accomplish, rather than how you think it should be done. Just hooking up a bunch of subscribers to a bunch of publishers might not be the best way to go about whatever it is that you're doing.
0MQ automatically allows N to M networks; just connect each subscriber to each publisher. That can become a burden to manage, in which case you can use a simple proxy that all publishers and subscribers connect to, see http://zguide.zeromq.org/page:all#toc36.
I have done a simple example in C which uses the built in zmq forwarder device to forward messages from N publishers to N subscribers. The code is available at https://gitorious.org/hellozmq
Related
I want to check how many users are connected to my pubsub pattern. Is there a simple way to do it in Go? Thank you.
pubsub := env.redisCli.PSubscribe(name)
defer pubsub.Close()
I have tried this:
val, _ := env.redisCli.Do("pubsub", "numpat").Int()
But it shows me other patterns also and I want to count only in that specific pattern.
The redis documentation states you can limit the result to a single subscription with the NUMSUB command, but this will not list clients that are subscribed to patterns:
Returns the number of subscribers (not counting clients subscribed to patterns) for the specified channels.
The NUMPAT on the other hand will count all patterns all clients are connected to:
Note that this is not just the count of clients subscribed to patterns but the total number of patterns all the clients are subscribed to.
(from https://redis.io/commands/pubsub)
I don't find a way to list all subscribers with their subscriptions in Redis. So the only way I can think of is to store that information in redis (or somewhere else) independently and manage it yourself.
Can I publisher service receive data from an external source and send them to the subscribers?
In the wuserver.cpp example, the data are generated from the same script.
Can I write a ZMQ_PUBLISHER entity, which receives data from external data source / application ... ?
In this affirmation:
There is one more important thing to know about PUB-SUB sockets: you do not know precisely when a subscriber starts to get messages. Even if you start a subscriber, wait a while, and then start the publisher, the subscriber will always miss the first messages that the publisher sends. This is because as the subscriber connects to the publisher (something that takes a small but non-zero time), the publisher may already be sending messages out.
Does this mean, that a PUB-SUB ZeroMQ pattern is performed to a best effort - UDP style?
Q1: Can I write a ZMQ_PUBLISHER entity, which receives data from external data source/application?
A1: Oh sure, this is why ZeroMQ is so helping us in designing smart distributed-systems. Just imagine the PUB-side process to also have other { .bind() | .connect() }-calls, so as to establish such other links to data-feeder(s), and you are done to operate the wished to have scheme. In distributed-systems this gives you a new freedom to smart integrate heterogeneous systems to talk to each other in a very efficient way.
Q2:Does this mean, that a PUB-SUB ZeroMQ pattern is performed to a best effort - UDP style?
A2: No, it has another meaning. The newly declared subscriber entities at some uncertain moment start to negotiate their respective subscription-topic filtering and such a ( distributed ) process takes some a-priori unknown time. Unless until the new / changed topic-filter policy was established, there is nothing to go into the SUB-side exgress interface to meet a .recv()-call, so no one can indeed tell, when that will get happened, can he?
On a higher level, there is another well known dichotomy of ZeroMQ -- Zero-Warranty Principle -- expect to either get delivered a complete message or none at all, which prevents the framework users from a need to handle any kind of damaged / inconsistent message-payloads. Either OK, or None. That's a great warranty. The more for distributed-systems.
I'm currently looking for the best architecture for an IM app I'm trying to build.
The app consists of channels each having a couple thousands of subscribed users. Each user is subscribed only to one channel at a time and is able to publish and read from that channel. Users may move rapidly between channels.
I initially considered using the XMPP PubSub (via Ejabbered or MongooseIM) but as far as I understand it was added as an afterthought and is not very scalable.
I also thought about using using a message queue protocol like AMPQ but I'm not sure if that's what I'm looking for from the IM aspect.
Is my concern regarding the XMPP PubSub justified? And if so, do you know of a better solution?
Take a look at Redis and Kafka. Both are scalable and performant.
I imagined below primary usecases for above IM application based on your inputs.
**
Usecases
**
Many new users keep registering with system and subscribing to one
of the channels
Many existing users changing their subscription from one channel to
other channel
Many existing users keep publishing messages to channels
Many existing users keep receiving messages as subscribers
XMPP is natural fit for 3rd and 4th usecases. "ejabbered" is one of proven highly scalable platform to go ahead.
In case 2nd usecase, You probably may have logic some thing like this.
- a)update channel info of the user in DB
- b)make him listen to new channel
- c)change his publishing topic to other channel...so on
When ever you need to do multiple operations, I strongly recommend to use "KAFKA" to perform above operations in async manner
In case of 1st usecase, Provide registration through rest APIs.So that registration can be done from any device.While registering an user,You may have many operations as follows.
- 1) register user in DB
- 2) create internally IM account
- 3) send email OR SMS for confirmation...so on
Here also perform 1st operation as a part of rest API service logic. Perform 2nd and 3rd operations in async manner using KAFKA. That means your service logic perform 1st operation in sync manner and raise an event to KAFKA. Every consumer will handle 2nd and 3rd operations in async manner.
System could scale well if all layers/subsystems can scale well. In that perspective, Below tech stack may help you scale well.
REST APIS + KAFKA + EJABBERED(XMPP)
if I am sending a message onto a multicast topic using:
TibrvMsg replyMessage = TibRvdTransport.sendRequest(message,timeout)
and there are two subscribers, which one actually sends the replyMessage, and what happens to the other replyMessage ?
I can only guess the fastest one that that answers. But I cannot see this documented anywhere.
Since your components are decoupled, they are unaware of each other. Rendezvous is pub-sub, which means that all subscribers receive all messages published to subjects that they have subscribed to. Furthermore, Rendezvous uses a peer-to-peer messaging approach vis-a-vis a centralized message forwarding approach. Therefore both components will receive the message and both components will reply.
If this is not the desired behavior, with Rendezvous you can use a distributed queue (RVDQ). With that approach a "scheduler" assigns work to workers, ensuring that messages get processed only once.
Say I have one JMS message FooCompleted
{"businessId": 1,"timestamp": "20140101 01:01:01.000"}
and another JMS message BazCompleted
{"businessId": 1,"timestamp": "20140101 01:02:02.000"}
The use case is that I want some action triggered when both messages have been received for the business id in question - essentially a join point of reception of the two messages. The two messages are published on two different queues and order between reception of FooCompleted and BazCompleted may change. In reality, I may need to have join of reception of several different messages for the businessId in question.
The naive approach was that to store the reception of the message in a db and check if message(s) its dependent join arm(s) have been received and only then kick off the action desired. Given that the problem seems generic enough, we were wondering if there is a better way to solve this.
Another thought was to move messages from these two queues into a third queue on reception. The listener on this third queue will be using a special avataar of DefaultMessageListenerContainer which overrides the doReceiveAndExecute to call receiveMessage for all outstanding messages in the queue and adding messages back to the queue whose all dependent messages have not yet arrived - the remaining ones will be acknowledged and hence removed. Given that the quantum of messages will be low, probing the queue over and adding messages again should not be a problem. The advantage would be avoiding the DB dependency and the associated scaffolding code. Wanted to see if there is something glaringly bad with this
Gurus, please critique and point out better ways to achieve this.
Thanks in advance!
Spring Integration with a JMS message-driven adapter and an aggregator with custom correlation and release strategies, and a peristent (JDBC) message store will provide your first solution without writing much (or any) code.