When using Ably for Pub/Sub over WebSockets, can I use wildcards to subscribe to multiple channels like so
var channel = ably.channels.get('foo:*')
channel.attach()
(disclaimer: I am a developer advocate for Ably, and posting and self-answering a commonly asked support question here on Stack Overflow so our users can find this more easily)
When attaching to a channel, you need to explicitly provide the channel name you are attaching to such as:
var channel = ably.channels.get('announcements')
channel.attach()
Attaching to more than one channel in a single operation is not possible, i.e. the following is not supported:
var channel = ably.channels.get('foo:*')
channel.attach()
/* attempting to attach to all channels matching the name foo:* will not work */
This is not possible for a number of reasons:
Attaching to unbounded number of channels in a single operation will not scale for your client devices or our servers terminating those connections
Channels in Ably's cluster are dynamically distributed across the available resources and move frequently. Each channel is largely autonomous and this is important to ensure the system remains reliable without a single point of failure or congestion. If a client were to subscribe to all channels matching a wildcard, then a connection would need to be maintained to every server in the cluster that could possibly be running a channel in case a channel is created that matches that wildcard. This does not scale.
If you are subscribed to wildcard channels then it is impossible to offer the data delivery guarantees and quality of service Ably provides on channels because:
At no point in time is there a way to deterministically know which channels a client is actually attached to
If the client device becomes overloaded (can't keep up with the stream) or exceeds rate limits, Ably's servers will have to selectively start dropping messages across random channels to ensure the client can continue to receive messages. Which messages should be dropped? How does a customer then work out which messages he/she missed?
However, because Ably's connections are multiplexed thus allowing you to attach and detach from any channels dynamically over the same connection, it is of course possible to effectively subscribe to wildcard channels by attaching to channels as you need them.
Related
According to the documentation here: https://github.com/OpenHFT/Chronicle-Engine one is able to do pub/sub using maps. This allows one to create a construct similar to topics that are available in middleware such as Tibco, 29W, Kafka and use that as a way of sending events across processes. Is this a recommended usage of chronicle map? What kind of latency can I expect if both publisher and subscriber stay in the same machine?
My second question is, how can this be extended to send messages across machines? How does this work with enterprise TCP replication?
My requirement is to create thousands of topics and use them to communicate across processes running in different machines (in a LAN). Each of these topics would be written by a single source and read by multiple readers running in same or different machines. If the source of a particular topic dies, that source's replica would start writing to the topic and listeners will continue to receive messages. These messages need not be stored for replay.
Is this a recommended usage of chronicle map?
Yes, you can use engine to support event notification across a machine. However, if you want lowest latencies you might need to send a notification via Queue and keep the latest value in a map.
What kind of latency can I expect if both publisher and subscriber stay in the same machine?
It depends on your use case esp the size of the data (in maps case the number of entries as well) The Latency for Map in Engine is around 30 - 100 us, however the latency for Queue is around 2 - 5 us.
My second question is, how can this be extended to send messages across machines?
For this you need our licensed product but the code is the same.
Each of these topics would be written by a single source and read by multiple readers running in same or different machines. If the source of a particular topic dies, that source's replica would start writing to the topic and listeners will continue to receive messages.
Most likely, the simplest solution is to have a Map where each topic is a different key. This will send the latest value for that topic to the consumers.
If you need to recorded every event, a Queue is likely to be a better choice. If you don't need to retain the data for long, you can use a very sort file rotation.
Can I publisher service receive data from an external source and send them to the subscribers?
In the wuserver.cpp example, the data are generated from the same script.
Can I write a ZMQ_PUBLISHER entity, which receives data from external data source / application ... ?
In this affirmation:
There is one more important thing to know about PUB-SUB sockets: you do not know precisely when a subscriber starts to get messages. Even if you start a subscriber, wait a while, and then start the publisher, the subscriber will always miss the first messages that the publisher sends. This is because as the subscriber connects to the publisher (something that takes a small but non-zero time), the publisher may already be sending messages out.
Does this mean, that a PUB-SUB ZeroMQ pattern is performed to a best effort - UDP style?
Q1: Can I write a ZMQ_PUBLISHER entity, which receives data from external data source/application?
A1: Oh sure, this is why ZeroMQ is so helping us in designing smart distributed-systems. Just imagine the PUB-side process to also have other { .bind() | .connect() }-calls, so as to establish such other links to data-feeder(s), and you are done to operate the wished to have scheme. In distributed-systems this gives you a new freedom to smart integrate heterogeneous systems to talk to each other in a very efficient way.
Q2:Does this mean, that a PUB-SUB ZeroMQ pattern is performed to a best effort - UDP style?
A2: No, it has another meaning. The newly declared subscriber entities at some uncertain moment start to negotiate their respective subscription-topic filtering and such a ( distributed ) process takes some a-priori unknown time. Unless until the new / changed topic-filter policy was established, there is nothing to go into the SUB-side exgress interface to meet a .recv()-call, so no one can indeed tell, when that will get happened, can he?
On a higher level, there is another well known dichotomy of ZeroMQ -- Zero-Warranty Principle -- expect to either get delivered a complete message or none at all, which prevents the framework users from a need to handle any kind of damaged / inconsistent message-payloads. Either OK, or None. That's a great warranty. The more for distributed-systems.
I want to be able to process messages with Spring Integration in parallel. The messages come from multiple devices and we need to process messages from the same device in sequential order but the devices can be processed in multiple threads. There can be thousands of devices so I'm trying to figure out how to assign processor based on mod of the device ID using Spring Integration's semantics as much as possible. What approach should I be looking at?
It's difficult to generalize without knowing other requirements (transaction semantics etc) but probably the simplest approach would be a router sending messages to a number of QueueChannels using some kind of hash algorithm on the device id (so all messages for a particular device go to the same channel).
Then, have a single-threaded poller pulling messages from each queue.
EDIT: (response to comment)
Again, difficult to generalize, but...
See AbstractMessageRouter.determineTargetChannels() - a router actually returns a physical channel object (actually a list, but in most cases a list of 1). So, yes, you can create the QueueChannels programmatically and have the router return the appropriate one, based on the message.
Assuming you want all the messages to then be handled by the same downstream flow, you would also need to create a <bridge/> for each queue channel to bridge it to the input channel of the next component in the flow.
create a QueueChannel
create a BridgeHandler (set the outputChannel to the input channel of the next component)
create a PollingConsumer (constructor takes the channel and handler; set trigger etc)
start() the consumer.
All of this can be done in your custom router initialization and implement determineTargetChannels() to select the queue.
Depending on the processing time for your events, I would generally recommend running the downstream flow on the poller thread rather than setting a taskExecutor to avoid issues with the next poll trying to schedule another task before this one's done. You might need to increase the default taskScheduler's pool size.
I can make some senders to send some messages, but when I create two receivers in one Session, the first one works and the second one blocked. In debug, I see the queue list size which the second receiver received is zero. I found that session is made for one thread, I don't know whether this problem involves in the unsafe thread?
I use ActiveMQ implementation.
A JMS Session is absolutely single threaded. As such, it can only have one active receiver. You have 2 options:
Use one connection with multiple sessions, each session having a receiver. Connections are thread safe and you can create many sessions from that single connections.
ActiveMQ gives you a number of options regarding multiplexing multiple destinations, so rather than having multiple receivers, you might want to focus on one, but use ActiveMQ's facilities to create virtual destinations that will funnel all the messages you want through the one receiver.
See this question.
This is a follow up on the question below, but it deserves another question.
Are server-conn and client-conn channels the only channels that could have more than one instances?
To my understanding, a receiver (or cluster receiver) channel usually pair up with a single sender (or cluster sender) channel. How can one side being single instance while the other side being multiple instances?
Thanks.
Consider a CLUSRCVR channel for a minute. You define one of these and then all members of the cluster use the same one to talk to the local QMgr. Similarly, a single SVRCONN channel definition can server connections from many users (think SYSTEM.ADMIN.SVRCONN for instance, with 50 users attached to it.)
The same principle works for RCVR or RQSTR channels. For example, suppose that you create a channel on your local QMgr called BRANCH.TO.CENTRAL with the intention that each branch office uses the channel to send daily sales numbers to the home office. Each of the queue managers can have only one instance of a BRANCH.TO.CENTRAL channel defined. The branch queue managers can have only one instance of the channel running but the central office can have many instances of the channel active at the same time. Typically you would not do this and would either use a cluster or named channel pairs, but it does serve to illustrate the case.
So the rule of thumb is that inbound channels (SVRCONN, RCVR, RQSTR and CLUSRCVR) can have many running instances using a single channel definition while outbound channels can have only one running instance.
Does that help or just muddy things up?