I am developing some code to process events off of an Azure Event Hub using AMQP with the Apache Qpid library. One of the things I'm noticing is that when my application restarts, all messages are re-read from the consumer group / partition.
My assumption is that my consumer is not checkpointing as it should (based on https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-features#event-consumers), but I am not sure what I options would need to be set on the JMS consumer to do this.
My current connection code (prior to attaching message listeners) looks something like this:
final ConnectionFactory factory = new JmsConnectionFactory(uri);
final Connection connection = factory.createConnection();
connection.start();
final Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
Is there something I need to do in terms of URL options to cause checkpointing to occur?
The short answer to this question is that AMQP has no notion of checkpointing built in and, by extension, neither does JMS. The result of this is that every time an application that reads via AMQP starts up, it will begin reading from the beginning of the event stream and reprocess everything.
If the application is developed properly (because an intentional rewind is quite possible), this shouldn't cause a functional problem, but it does have the potential to be very wasteful of resources. In the end I settled on using Microsoft's Event Hub client for Java, which has checkpointing support built in.
I sketched this out in some sample code on my github page, comparing https://github.com/michaeljmcd/eventhub-qpid-example and https://github.com/michaeljmcd/eventhub-client-example
Related
I have multiple instances of a worker connected to a queue and all requests will be distributed to worker instances in a load balanced way. When a new worker instance is connected to the queue, I should dump a small data from mainstream app to this new worker instance (one time job).
Currently I'm using REST endpoint from mainstream app for doing this at application start-up but can we leverage the messaging queue for this? Once a new worker instance connected to queue, it will ask the initial data dump to mainstream app through queue and then app will reply with initial data.
Is it possible using messaging queue/topic? Kindly share your views/suggestions to achieve this using activemq
If you're using ActiveMQ Artemis this kind of requirement is typically fulfilled with a queue that supports both non-destructive and last-value semantics. The last-value semantics allows the queue to stay up-to-date with the latest messages and the non-destructive semantics means that even when consumers acknowledge the messages they will remain on the queue for the next client which connects. When using this combination clients can first consume all the messages from this special "initialization" queue and then continue on with whatever other messaging work they need to do.
Unfortunately ActiveMQ "Classic" doesn't support either of these semantics and there is no straight-forward way to get equivalent behavior.
I'm investigating ActiveMQ to see if it will work for a project. The current use case I need to demonstrate is that late-joining subscribers will receive topics published prior to the creation of the subscription. It seemed that ActiveMQ Retroactive Consumers would satisfy this need, but I can't get the code to work.
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory("[url]");
Connection connection = connectionFactory.createConnection();
connection.start();
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
Destination destination = session.createTopic("testAddress?consumer.retroactive=true");
MessageProducer producer = session.createProducer(destination);
TextMessage message = session.createTextMessage("Hello, World!");
producer.send(message);
Thread.sleep(5000);
session.createConsumer(destination).setMessageListener(message2 -> processMessage(message2));
session.close();
connection.close();
connectionFactory.close();
All I'm trying to demonstrate here is that a topic can be published, and then some arbitrary amount of time later (eg. 5 seconds) a consumer can subscribe to the topic and receive the previous message.
As far as I can tell, the issue seems to be that creating the topic creates an address but doesn't create any associated queues. If I send a topic to the address before the queue is made (either in code or manually via the web interface to the broker), the message seems to be ignored and the "un routed message count" is immediately incremented.
The ActiveMQ documentation ( https://activemq.apache.org/retroactive-consumer ) doesn't provide any greater detail on how to set up a retroactive consumer than appending "?consumer.retroactive=true" when making the topic, so I wonder if there are some other configuration aspects I'm missing.
To my knowledge ActiveMQ Artemis doesn't support the retroactive consumer feature that 5.x does. The client side option just tells the broker you want it, but since Artemis doesn't handle that you won't see any difference from sending it. The feature itself in 5.x shouldn't be relied upon as a 100% stand in for a durable consumer, broker restart for instance will cause all those messages (of which the amount stored is finite) to be lost.
If you want to guarantee that you get messages sent when the topic consumer is offline then a durable consumer is the safe way to do this
To accomplish the desired behavior (a subscriber receives topics that were published before the subscription was made) in ActiveMQ Artemis, I used a Last Value Queue with non-destructive reads. This has the limitation that I'm only receiving only the most recent copy of a topic published, but that will work for my situation.
I'm designing a quite complicated system and was wondering what the best way is to put a jms consumer (activemq, vm protocol, non persitent) inside a netty handler.
Let me explain, i have several clients connecting to my netty server using websockets. For every client connection i create a jms consumer that listens for interesting messages on one or more topics. If a interesting message arrives i need to do a extra step (additional filtering) before sending the message to the client using the websocket.
Is the following a good way to do this:
inside a SimpleChannelInboundHandler i declare a private non static consumer
the consumer is initialized in channelActive
the consumer is destroyed in channelInactive
when a message is received by consumer i do the extra filter a send it using ctx.channel().write()
In this setup i'm a bit worried that the consumer might turn into slow consumer and slow everything down, cause the websocket goes over the internet.
I came up with a more complex one to decouple the "receiving of message by consumer" and "sending of message through a websocket".
inside a SimpleChannelInboundHandler i declare a private non static consumer
the consumer is initialized in channelActive
the consumer is destroyed in channelInactive
when a message is received by consumer i put it in a blockedqueue
every minute i let a thread (created for every client) look in the queue and send the found messages to the client using ctx.channel().write().
At this point i'm a bit worried about the extra thread per client.
Or is there maybe a better way to accomplish this task?
This is a classic slow consumer problem and the first step to resolving it is to determine what the appropriate action is when a slow consumer is detected. If it is acceptable that the slow consumer misses messages then the solution is some variation on dropping messages or unsubscribing them from the feed. For example, if it's acceptable that the client misses messages then, when one is received from JMS, check if the channel is writable. If it isn't, drop the message. If you want to give yourself a bit more of a buffer (although OS buffers are quite large) you can track the number of write completion future's that haven't completed (ie the messages haven't been written to the OS send buffer) and drop messages if there are too many outstanding write requests.
If the client may not miss messages, and is consistently slow, then the problem is more difficult. One option might be to divert messages to a JMS queue with a specific header value, then open a new consumer that reads messages from that queue using a JMS selector. This will put more load on the JMS server but might be appropriate for temporary slowness and hopefully it won't interfere with you main topic feeds. Alternatively you might want to stash the messages in a different store, such as a database, so you can poll for messages when they can be sent. If you do this right a single polling thread can cope with many clients (query for clients which have outstanding messages, then for each client, load a bunch of messages). However this isn't as convenient as using JMS.
I wouldn't go with option 2 because the blocking queue is only going to solve the problem temporarily, and you can achieve the same thing by tracking how many write operations are waiting to complete.
Sorry, if it is a duplicate question.
I have a legacy web application which uses Queues (yes. normal Java Queue) and custom polling (every 500ms). A REST web service (/message) will be called, which will return the message if any otherwise empty string.
My need: If any message is available in Queue, in Real-Time, the client should get the message. So I can save 500ms.
Is there any advantage to moving to JMS from current approach? From this link JMS MessageConsumer's messageListener makes push or pull? it seems, MessageListener (process is asynchronous) uses polling which is no different from current approach.
If it is vendor based, how HornetQ/ActiveMQ supports MessageListener?
EDIT:
The queue is used for integration of two systems. A web app & standlone java program.
Either receive or a MessageListener will be asynchronous and will be called as soon as you receive a message.
you could control the pre-fetch size of your client.
Now, if all you need is to avoid the delay of poling every 500 ms, using a Queue system may be an overkill? It's perfect fine to use java.util.Queue (or any other subclass).
If all you need is to block until an element of a java.util.Queue is available, and you don't need distributed messaging, persistence or anything like you could simply using BlockingDequeue and your thread would unblock as soon as you have a message..
Look at this:
http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/BlockingDeque.html
The Async MessageListener is implemented using a push based model. In ActiveMQ the broker sends a number of messages to the client based in it's set prefetch value so that messages are ready for consumption. Whether or not this helps with your particular use case is a question you need to answer for yourself.
I’m writing a server/client game, a typical scenario looks like this: one client (clientA) send a message to the server, there is a MessageDrivenBean in server to handle such messages. After the MDB finished its job, it sends the result message back to another client (clientB).
In my opinion I only need two queues for such communication, one for input the other for output. Creating new queue for each connection is not a good idea, right?
The Input queue is relative clear, if more clients are sending message at the same time, the messages are just waiting in the queue, while there are more MDB instances in server, that should not a big performance issue.
But on the other side I am not quite clear about the output queue, should I use a topic instead of a queue? Every client is listening the output queue, one of them gets the new message and checks the property to determine if the message is to it, if not, it rollback the transaction, the message goes back to queue and be ready for other client … It should work but must be very slow. If I use topic instead, every client gets a copy of the message, if it’s not to it, just ignores the message. It should be better, right?
I’m new about message system. Is there any suggestion about my implementation? Thanks!
To begin with, choosing JMS as a gaming platform is, well, unusual — businesses use JMS brokers for delivery reliability and transaction support. Do you really need this heavy lifiting in a game? Shouldn't you resort to your own HTTP-based protocol, for example?
That said, two queues are a standard pattern for point-to-point communication. Creating a queue for a new connection is definitely not OK — message-driven beans are attached to queues at deployment time, so you won't be able to respond to queue creation events. Besides, queues are not meant to be created and destroyed in short cycles, they're rather designed to be long-living entities. If you need to deliver a message to one precise client, have the client listen on the server response queue with a message selector set to filter only the messages intended for this client (see javax.jms.Message API).
With topics it's exactly as you noted — each connected client will get a copy of the message — so again, it's not a good pattern to send to n clients a message that has to be discarded by n-1 clients.
MaDa;
You could stick one output queue (or topic) and simply tag the message with a header that identifies the intended client. Then, clients can listen on the queue/topic using a selector. Hopefully your JMS implementation has efficient server-side listener evaluation.