I am looking into using ZeroMQ, and have tried a few examples. However there are a few important requirements that I can't verify that ZeroMQ supports. I hope you can help me.
I'll ask my question by using this simple scenario:
A publisher (say an exchange) provides price updates on (a large number of) shares. N clients subscribes to price updates of 1..n of these shares.
I have seen the PUB/SUB feature, but I cant verify whether the filtering takes place on the publisher or subscriber side? In a use case like the above, anything except filtering on the publisher side is not acceptable to me.
However my investigation tells me that filtering is only available on the subscriber side, meaning that ALL data is sent to the subscriber, and filtered there.
I suspect I have got it completely wrong, and would appreciate any pointers on how one would solve such a scenario with ZeroMQ
That's simple, in ZeroMQ 3.x PUB/SUB filtering is done at publisher side, source: http://www.zeromq.org/docs:3-1-upgrade
Related
We give support to an organization that uses IBM MQ (v9.0) to communicate to an external partner. It's a simple layout, a single local queue manager connected to an external one trough one single channel. Various applications use different queues to communicate trough this QM.
Last week, after solving a support ticket, my boss came to the conclusion that having one single channel to funnel all the messages from the different queues (and applications that use such queues) is not good.
I have to write a report detailing what criteria should we use to separate the message flows into different channels, and what benefits and inconveniences would this entail.
I'm fairly new to IBM MQ, and I'm having trouble finding information on this topic. So far I've read the official documentation and other bibliography and i have a superficial idea about it, but i cant find concrete information about this topic.
Is separating message flows into different channels a good practice? What would be a good criteria to use?
Here is a good write up on channels and it discusses reasons for parallel channels. You may be able to use this in your report. MQ Channels
If you have applications with large messages that are of the fire-and-forget pattern it is common to put these on a different path than the real time apps that are of the request-response pattern.
Whether or not this configuration change will lessen the chance of problem tickets depends largely on what led to the issue. Good luck.
I have an architectural problem.
I know what represents a (fanout, direct, topic, headers) exchange, bindings, queues and almost everything about message architectures. I have the following problem and I need some pieces of advice.
I would like to implement notification logic to my application, where each user will receive in real-time notification only intended for him. ( Actually, I don't want mention what are my UI and BE languages/frameworks, because of an additional level of abstraction) The UI will make a connection to RabbitMQ with WebSocket, SockJS and STOMP. My UI will be only a consumer, the BE is the writer - that one, which will add some Messages to RabbitMQ.
It's perfectly clear to me if I have a direct Exchange with routing key, which uniquely to identify the specific user (for example: my-routing-to-empoyee-with-id-1) and N-number of Queues for each user. This is too heavy to me (I don't know actually whether is normal situation to have so much queues).
Is there any solution, where I can use only one Queue and the message to be delivered only to user for who is intended ?
I know a solution, where I can have a topic exchange and to have one writer and many subscribers, but on this way, I can filter the message only on clients level, which is not so secure. :(
Actually, I found a very interesting article, which describes me a problem that I have pretty well.
What I want is called Selective Consumers and for Enterprise Integration Patterns this is an anti-pattern and we should not use it.
For more details, all who want can read this article: https://derickbailey.com/2015/07/22/airport-baggage-claims-selective-consumers-and-rabbitmq-anti-patterns/
As stated in the docs in version 3.x of zeromq in PUB/SUB scenarios messages are being filtered on publisher side (rather than on subscriber side which is trivial).
To me this sounds like that the publisher has to hold a list of all connected sockets and message filters to accomplish this.
Would you agree?
Based on this assumption I'd now like to know whether or not a specific filter is active or not. This would make it possible for me to not retrieve specific data from some (maybe very slow) other data provider when I know it's not being used anyway.
Is there a way to see what filters are active on a given PUB socket in a recent version of ZeroMQ?
I know there already has been some work on this, see here but that's been two years now..
So far as I know, there's no way to get this information from ZMQ. If you want the most up-to-date information on this, the best place to ask would be the ZMQ dev mailing list, the actual developers are over there.
Looking a little further back, I found this discussion on the mailing list that, while it doesn't speak specifically about subscriber topics, does address why that information isn't available - namely, that knowing a subscriber is subscribed to a topic means knowing that they're connected, and that information goes against the ZMQ abstraction design concept of letting connections/disconnections be more seamless.
There is a solution, just probably not the one you're looking for: spin up another pair of meta-sockets to communicate from client to server what topics it is interested in, so this information goes from ZMQ abstraction into explicit message passing. You keep track of that info there and use it to control your information gathering. It may seem a bit of a kludge (when the information is already technically there in the publisher, as you note), but it's the ZMQ way of doing things.
I am looking for a realtime hosted push/socket service (paid is fine) which will handle many connections/channels from many clients (JS) and server api which can subscribe/publish to those channels from a PHP script.
Here is an example:
The client UI has a fleet of 100 trucks rendered, when a truck is modified its data is pushed to channel (eg. /updates/truck/34) to server (PHP subscriber), DB is updated and receipt/data is sent back to the single truck channel.
We have a prototype working in Firebase.io but we don't need the firebase database, we just need the realtime transmission. One of the great features of firebase.io is that its light and we can subscribe to many small channels at once. This helps reduce payload as only that object data that has changed is transmitted.
Correct me if I am wrong but I think pusher and pubnub will allow me to create 100 truck pub/subs (in this example) for each client that opens the site?
Can anyone offer a recommendation?
I can confirm that you can use Pusher to achieve this - I work for Pusher.
PubNub previously counted each channel as a connection, but they now seem to have introduced multiplexing. This FAQ states you can support 100 channels over the multiplexed connection.
So, both of these services will be able to achieve what you are looking for. There will also be more options available via this Realtime Web Tech guide which I maintain.
[I work for Firebase]
Firebase should continue to work well for you even if you don't need the persistence features. We're not aware of any case where our persistence actually makes things harder, and in many cases it actually makes your life a lot easier. For example, you probably want to be able to ask "what is the current position of a truck" without needing to wait for the next time an update is sent.
If you've encountered a situation where persistence is actually a negative for you, we'd love to hear about it. That certainly isn't our intention.
Also - we're not Firebase.io -- we're just Firebase (though we do own the firebase.io domain name).
I'm having a play around with websockets and I'm having a bit of trouble wrapping my head around some stuff. Specifically, being able to send a whole bunch of subscribers different data without using a stupid amount of resources.
For example, if you had some sort of twitter like service, how would you send all followers of a person a newly posted tweet that they have made (and do the same for the other hundreds of people doing the same). It just seems that handling that many separate people is a bit absurd.
Can someone talk me through how you would go about treating each client individually? Please tell me if I have the whole idea of websockets wrong.
Thanks in advance!
P.S. for reference, I'm probably going to play around using either node or clojure (with aleph)
Use an established messaging protocol and broker on top of websockets.
It seems you are looking at websockets at the application layer when it is more of a network protocol. A variety of messaging APIs exists (such as JMS) with open source message brokers that are designed to do the complex and scalable message routing.