I have some Java code that reads messages from an ActiveMQ queue. The code uses a JmsTemplate from Spring and I use the "browseSelected" method to retrieve any messages from the queue that have a timestamp in their header older than 7 days (by creating the appropriate criteria as part of the messageSelector parameter).
myJmsTemplate.browseSelected(myQueue, myCriteria, new BrowserCallback<Integer>() {
#Override
public Integer doInJms(Session s, QueueBrowser qb) throws JMSException {
#SuppressWarnings("unchecked")
final Enumeration<Message> e = qb.getEnumeration();
int count = 0;
while (e.hasMoreElements()) {
final Message m = e.nextElement();
final TextMessage tm = (TextMessage) MyClass.this.jmsQueueTemplate.receiveSelected(
MyClass.this.myQueue, "JMSMessageID = '" + m.getJMSMessageID() + "'");
myMessages.add(tm);
count++;
}
return count;
}
});
The BrowserCallback's "doInJms" method adds the messages which match the criteria to a list ("myMessages") which subsequently get processed further.
The issue is that I'm finding the code will only process 400 messages each time it runs, even though there are several thousand messages which match the criteria specified.
When I previously used another queueing technology with this code (IBM MQ), it would process all records which met the criteria.
I'm wondering whether I'm experiencing an issue with ActiveMQ's prefetch limit: http://activemq.apache.org/what-is-the-prefetch-limit-for.html
Versions: ActiveMQ 5.10.1 and Spring 3.2.2.
Thanks in advance for any assistance.
The broker will only return up to 400 message by default as configured by the maxBrowsePageSize option in the destination policies. You can increase that value but must use caution as the messages are paged into memory and as such can lead you into an OOM situation.
You must always remember that a message broker is not a database, using it as one will generally end in tears.
Related
I send a simple text message to an MQ Queue (MQ 7.0.1):
"abc"
Using spring JMS the total length of the message is: 291
But putting the same message in the queue using IBM MQ libraries the total length of the message is: 3
How can I get total data length 3 with JMS?
Spring JMS code:
#EnableJms
public class JMSTestController {
...
#Autowired
private JmsTemplate jmsTemplate;
#Autowired
JmsMessagingTemplate jmsMessagingTemplate;
...
public String send() throws JMSException{
jmsTemplate.setReceiveTimeout(10000);
jmsMessagingTemplate.setJmsTemplate(jmsTemplate);
Session session = jmsMessagingTemplate.getConnectionFactory().createConnection()
.createSession(false, Session.AUTO_ACKNOWLEDGE);
Queue entryQueue = session.createQueue("hereQueueName");
Queue replyQueue = session.createQueue("hereReplyQueueName");
TextMessage message = session.createTextMessage("abc");
message.setJMSDeliveryMode(DeliveryMode.NON_PERSISTENT);
message.setJMSDestination(entryQueue);
message.setIntProperty(WMQConstants.JMS_IBM_CHARACTER_SET, 819);
message.setIntProperty(WMQConstants.JMS_IBM_ENCODING, 273);
jmsMessagingTemplate.convertAndSend(entryQueue, message);
String messageId = message.getJMSMessageID();
...
}
Native code:
MQQueueManager qm = createQueueManager(queueManager, host, port,
channel, username, password, connectionType);
MQQueue m_receiver = null;
MQMessage msg = new MQMessage();
msg.format = MQC.MQFMT_STRING;
msg.expiry = timeout / 1000;
msg.replyToQueueName = qReceiver;
msg.replyToQueueManagerName = queueManager;
msg.write("abc".getBytes());
MQPutMessageOptions pmo = new MQPutMessageOptions();
try {
qm.put(qSender, msg, pmo);
} catch (MQException e) {
MQTalkerException ex = new MQTalkerException(
"An error happened sending a message", e);
logger.error(ex);
throw ex;
}
Solution
Following JoshMc's comment I made the following modification and reached the expected result:
Check out these answers, you want to set targetClient to MQ to remove
those properties. There are many ways to accomplish this, changing
your CreateQueue to use a URI is probably the easiest.
JMS transport v/s MQ transport
That is, modify the creation of the queue using the URI instead of just its name.
Queue entryQueue = session.createQueue("queue:///QUEUE_NAME?targetClient=1");
I reached the solution by following JoshMc's comment. That is, modify the creation of the queue using the URI instead of just its name.
Queue entryQueue = session.createQueue("queue:///QUEUE_NAME?targetClient=1");
This removes the MQRFH2 header (the extra bytes I didn't know where they came from)
and with that the message has a total length of 3 bytes.
Spring is counting the bytes of the message body (aka data)
IBM MQ native is counting the bytes of the message headers plus body
In your screenshot, the field directly above shows '3' bytes.
Longitud dataos = length of body = 3
Longitud total = length of headers + body = 291
I have requirement where need to have a Spring Boot Rest Service that a client application will call every 30 minutes and service is to return
number of latest messages based on the number specified in query param e.g. http://messages.com/getNewMessages?number=10 in this case should return 10 messages
number of messages based on the number and offset specified in query param e.g. http://messages.com/getSpecificMessages?number=5&start=123 in this case should return 5 messages starting offset 123.
I have simple standalone application and it works fine. Here is what I tested and would lke some direction of incorporating it in the service.
public static void main(String[] args) {
// create kafka consumer
Properties properties = new Properties();
properties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
properties.put(ConsumerConfig.GROUP_ID_CONFIG, "my-first-consumer-group");
properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
properties.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
properties.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, args[0]);
Consumer<String, String> consumer = new KafkaConsumer<>(properties);
// subscribe to topic
consumer.subscribe(Collections.singleton("test"));
consumer.poll(0);
//get to specific offset and get specified number of messages
for (TopicPartition partition : consumer.assignment())
consumer.seek(partition, args[1]);
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(5000));
System.out.println("Total Record Count ******* : " + records.count());
for (ConsumerRecord<String, String> record : records) {
System.out.println("Message: " + record.value());
System.out.println("Message offset: " + record.offset());
System.out.println("Message: " + record.timestamp());
Date date = new Date(record.timestamp());
Format format = new SimpleDateFormat("yyyy MM dd HH:mm:ss.SSS");
System.out.println("Message date: " + format.format(date));
}
consumer.commitSync();
As my consumer will be on-demand wondering in Spring Boot Service how I can achieve this. Where do I specify the properties if I put in application.properties those get's injected at startup time but how do i control MAX_POLL_RECORDS_CONFIG at runtime. Any help appreciated.
MAX_POLL_RECORDS_CONFIG only impact your kafka-client return the records to your spring service, it will never reduce the bytes that the consumer poll from kafka-server
see the above picture, no matter your start offset = 150 or 190, kafka server will return the whole data from (offset=110, offset=190), kafka server even didn't know how many records return to consumer, he only know the byte size = (220 - 110)
so i think you can control the record number by yourself,currently it is controlled by the kafka client jar, they are both occupy your jvm local memory
The answer to your question is here and the answer with code example is this answer.
Both written by the excellent Gary Russell, the main or one of the main person behind Spring Kafka.
TL;DR:
If you want to arbitrarily rewind the partitions at runtime, have your
listener implement ConsumerSeekAware and grab a reference to the
ConsumerSeekCallback.
I have following code to read xml files from a sftp server as InputStream:
#Configuration
public class SftpConfig {
...
#Bean
#InboundChannelAdapter(channel = "stream", poller = #Poller(fixedDelay="60000"))
public MessageSource<InputStream> messageSource() {
SftpStreamingMessageSource messageSource = new SftpStreamingMessageSource(template());
messageSource.setRemoteDirectory(sftpProperties.getBaseDir());
messageSource.setFilter(new SftpSimplePatternFileListFilter("*.xml"));
// messageSource.setMaxFetchSize(-1); no matter what i set this to, it only fetches one file
return messageSource;
}
#ServiceActivator(inputChannel = "stream", adviceChain = "after")
#Bean
public MessageHandler handle() {
return message -> {
Assert.isTrue(message.getPayload() instanceof InputStream, "Payload must be of type $InputStream");
String filename = (String) message.getHeaders().get(FileHeaders.REMOTE_FILE);
InputStream is = (InputStream) message.getPayload();
log.info("I am here"); // each poll only prints this once
};
}
...
}
When I debugged or checked the logs for MessageHanlder$handleMessage, I continuously only saw one message (file object) came through. And there are more than one .xml file sitting on the sftp server as I could verify by seeing file coming through in the next poll. The documentation says
/**
* Set the maximum number of objects the source should fetch if it is necessary to
* fetch objects. Setting the
* maxFetchSize to 0 disables remote fetching, a negative value indicates no limit.
* #param maxFetchSize the max fetch size; a negative value means unlimited.
*/
void setMaxFetchSize(int maxFetchSize);
So that I fiddled with different numbers but to no avail. What am I missing here?
Sorry for misleading, but fetch doesn't mean poll. The fetch options just take as many remote entities to the local cache on a first poll and every single subsequent polls just take entries from that cache until it is exhausted.
The option about max messages per poll belongs to that #Poller configuration. See a respective option:
/**
* #return The maximum number of messages to receive for each poll.
* Can be specified as 'property placeholder', e.g. {#code ${poller.maxMessagesPerPoll}}.
* Defaults to -1 (infinity) for polling consumers and 1 for polling inbound channel adapters.
*/
String maxMessagesPerPoll() default "";
Pay attention to that 1 for polling inbound channel adapters. That's how you see only one message coming through.
Nevertheless the logic is like push only one message to the channel. There is no batching for how many files you have a the moment. Independently of fetch perPoll only one message is sent to the channel. Although I agree that with infinite perPoll all the messages are sent in the same thread and during the same poll cycle.
I need to redeliver a message if it can't be processed, for example because of some external endpoint failure. So I'm using following MDB configuration (it worth to mention that I'm using openMQ (Glassfish 4.1)):
#MessageDriven(mappedName = "MyQueue", name = "MyQueue", activationConfig = {
#ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue"),
#ActivationConfigProperty(propertyName = "endpointExceptionRedeliveryAttempts", propertyValue = "10"),
#ActivationConfigProperty(propertyName = "endpointExceptionRedeliveryInterval", propertyValue = "30000")})
Here's onMessage() method:
#Override
#TransactionAttribute(TransactionAttributeType.REQUIRED)
public void onMessage (Message message)
{
try
{
//some processing here
}
catch (JMSException jmsException)
{
logger.log (Level.SEVERE, "Exception processing notification message", jmsException);
}
catch (BackingStoreException e)
{
// because of throwing RuntimeException, the message is going to be redelivered according to mdb configuration params(interval and attempts count)
throw new RuntimeException ();
}
}
In order to redeliver a message it's also possible to rollback a transaction, however openMQ lacks the property for redelivery interval, so it doesn't suites me:
https://github.com/javaee/openmq/issues/220
https://github.com/javaee/openmq/issues/23
https://github.com/javaee/openmq/issues/134
All in all, redelivery works fine, besides one moment: if message is going to be redelivered, mdb doesn't release the connection and holds it for endpointExceptionRedeliveryInterval * endpointExceptionRedeliveryAttempts, in my case 5 minutes. So, cause the default values for maxPoolSize is 32, 32 "bad" messages are enough to block the mdb.
Is there are way to release the connection in case of message redelivering?
This is the expected behavior as per JMS specification and I do not think that there could be some way to release the connection object while some sort of message processing is under progress. Read JMS specs here and below is relevant except from "4.3.5 Closing a Connection":
If one or more of the connection’s session’s message listeners is
processing a message at the point connection close is invoked, all the
facilities of the connection and it’s sessions must remain available
to those listeners until they return control to the JMS provider.
When connection close is invoked it should not return until message
processing has been orderly shut down. This means that all message
listeners that may have been running have returned and that all
pending receives have returned.
I am not sure why you want to use such high numbers for retry mechanism, 10 attempts are too big to go for, maximum I have seen till now is 3 times, I think you can try to adjust your retry mechanism numbers, or may be having another connection which is dedicated to re-delivery.
I'm looking to increase the performance of a high-throughput producer that I'm writing against ActiveMQ, and according to this useAsyncSend will:
Forces the use of Async Sends which adds a massive performance boost;
but means that the send() method will return immediately whether the
message has been sent or not which could lead to message loss.
However I can't see it making any difference to my simple test case.
Using this very basic application:
const string QueueName = "....";
const string Uri = "....";
static readonly Stopwatch TotalRuntime = new Stopwatch();
static void Main(string[] args)
{
TotalRuntime.Start();
SendMessage();
Console.ReadLine();
}
static void SendMessage()
{
var session = CreateSession();
var destination = session.GetQueue(QueueName);
var producer = session.CreateProducer(destination);
Console.WriteLine("Ready to send 700 messages");
Console.ReadLine();
var body = new byte[600*1024];
Parallel.For(0, 700, i => SendMessage(producer, i, body, session));
}
static void SendMessage(IMessageProducer producer, int i, byte[] body, ISession session)
{
var message = session.CreateBytesMessage(body);
var sw = new Stopwatch();
sw.Start();
producer.Send(message);
sw.Stop();
Console.WriteLine("Running for {0}ms: Sent message {1} blocked for {2}ms",
TotalRuntime.ElapsedMilliseconds,
i,
sw.ElapsedMilliseconds);
}
static ISession CreateSession()
{
var connectionFactory = new ConnectionFactory(Uri)
{
AsyncSend = true,
CopyMessageOnSend = false
};
var connection = connectionFactory.CreateConnection();
connection.Start();
var session = connection.CreateSession(AcknowledgementMode.AutoAcknowledge);
return session;
}
I get the following output:
Ready to send 700 messages
Running for 2430ms: Sent message 696 blocked for 12ms
Running for 4275ms: Sent message 348 blocked for 1858ms
Running for 5106ms: Sent message 609 blocked for 2689ms
Running for 5924ms: Sent message 1 blocked for 2535ms
Running for 6749ms: Sent message 88 blocked for 1860ms
Running for 7537ms: Sent message 610 blocked for 2429ms
Running for 8340ms: Sent message 175 blocked for 2451ms
Running for 9163ms: Sent message 89 blocked for 2413ms
.....
Which shows that each message takes about 800ms to send and the call to session.Send() blocks for about two and a half seconds. Even though the documentation says that
"send() method will return immediately"
Also these number are basically the same if I either change the parallel for to a normal for loop or change the AsyncSend = true to AlwaysSyncSend = true so I don't believe that the async switch is working at all...
Can anyone see what I'm missing here to make the send asynchronous?
After further testing:
According to ANTS performance profiler that vast majority of the runtime is being spent waiting for synchronization. It appears that the issue is that the various transport classes block internally through monitors. In particular I seem to get hung up on the MutexTransport's OneWay method which only allows one thread to access it at a time.
It looks as though the call to Send will block until the previous message has completed, this explains why my output shows that the first message blocked for 12ms, while the next took 1858ms. I can have multiple transports by implementing a connection-per-message pattern which improves matters and makes the message sends work in parallel, but greatly increases the time to send a single message, and uses up so many resources that it doesn't seem like the right solution.
I've retested all of this with 1.5.6 and haven't seen any difference.
As always the best thing to do is update to the latest version (1.5.6 at the time of this writing). A send can block if the broker has producer flow control enabled and you've reached a queue size limit although with async send this shouldn't happen unless you are sending with a producerWindowSize set. One good way to get help is to create a test case and submit it via a Jira issue to the NMS.ActiveMQ site so that we can look into it using your test code. There have been many fixes since 1.5.1 so I'd recommend giving that new version a try as it could already be a non-issue.