I'm using this part of code for connect to a queue manager.
jmsFactory = JmsFactoryFactory.getInstance(WMQConstants.WMQ_PROVIDER);
jcf = jmsFactory.createConnectionFactory();
// Set the properties
jcf.setStringProperty(WMQConstants.WMQ_HOST_NAME, hostName);
jcf.setIntProperty(WMQConstants.WMQ_PORT, port);
jcf.setIntProperty(WMQConstants.WMQ_CONNECTION_MODE, WMQConstants.WMQ_CM_CLIENT);
jcf.setStringProperty(WMQConstants.WMQ_CHANNEL, channel);
connection = jcf.createConnection();
When everything is OK, there is no problem. But when host is unreachable; there is a long wait for get an exception. I mean jcf.createConnection() waiting and trying to get connection around 30 seconds and then raise an exception.
Can I decrease this time? and how?
when in another application I tried to send thousand of messages by thread to a queue manager and each thread waiting for 30 second to get connection, it caused serious problems.
My suspicion is that your reconnectionRetryInterval is set at the default value, which I believe is 30 seconds.
Related
I am facing a scenario where the reply queue I connect to, runs out of handles. I have traced it to the fact that my JMS Producers are being cached but not my JMS consumers. I am able to send and receive messages just fine so there is no problem with connecting-sending-receiving to/from the queues. I am using the CachedConnectionFactory (SessionCacheSize = 10)with the target factory as com.ibm.mq.jms.MQQueueConnectionFactory while instantiating the jmsTemplate. Code snippet is as follows
:
:
String replyQueue = "MyQueue";// replyQueue which runs out of handles
messageCreator.setReplyToQueue(new MQQueue(replyQueue));
jmsTemplate.setReceiveTimeout(receiveTimeout);
jmsTemplate.send(destination, messageCreator);// Send to destination queue
Message message = jmsTemplate.receiveSelected(replyQueue,
String.format("JMSCorrelationID = '%s'", messageCreator.getMessageId()));
:
:
From the logs (jms TRACE is enabled) Producer is cached, so the destination queue "handle count" does not increase.
// The first time around (for Producer)
Registering cached JMS MessageProducer for destination[queue:///<destination>:com.ibm.mq.jms.MQQueueSender#c9x758b
// Second time around, the cached producer is reused
Found cached JMS MessageProducer for destination [queue:///<destination>]: com.ibm.mq.jms.MQQueueSender#c9x758b
However, the handles for the replyQueue keep increasing because for every call to that queue, I see a new JMS Consumer being registered. Ultimately the calls to open the replyQueue fail because of MQRC_HANDLE_NOT_AVAILABLE
// First time around
Registering cached JMS MessageConsumer for destination [queue:///<replyQueue>]:com.ibm.mq.jms.MQQueueReceiver#b3ytd25b
// Second time around, another MessageConsumer is registered !
Registering cached JMS MessageConsumer for destination [queue:///<replyQueue>]:com.ibm.mq.jms.MQQueueReceiver#re25b
My memory is a bit dim on this, but here is what is happening. You are receiving messages based on a message selector. This selector is always changing, however. As a test, either remove the selector or make it a constant and see what happens. So when you try to cache/pool based on connection/session/consumer, the consumer is always changing. This requires a new cache entry.
After you go through your 10 sessions, a new connection will be created, but the existing one is not closed. Increase your session count to 100, for example, and your connection count on the MQ broker should climb 10 time slower.
You need to create a new consumer for every message receive as your correlation ID is always changing. So just cache connection/session. No matter what you do, you will always have to round trip to the broker to ask for the new correlation ID.
I'm hoping this is a simple configuration issue but I can't seem to figure out what it might be.
Set-up
Spring-Boor 2.2.2.RELEASE
cloud-starter
cloud-starter-aws
spring-jms
spring-cloud-dependencies Hoxton.SR1
amazon-sqs-java-messaging-lib 1.0.8
Problem
My application starts up fine and begins to process messages from Amazon SQS. After some amount of time I see the following warning
2020-02-01 04:16:21.482 LogLevel=WARN 1 --- [ecutor-thread14] o.s.j.l.DefaultMessageListenerContainer : Number of scheduled consumers has dropped below concurrentConsumers limit, probably due to tasks having been rejected. Check your thread pool configuration! Automatic recovery to be triggered by remaining consumers.
The above warning gets printed multiple times and eventually I see the following two INFO messages
2020-02-01 04:17:51.552 LogLevel=INFO 1 --- [ecutor-thread40] c.a.s.javamessaging.SQSMessageConsumer : Shutting down ConsumerPrefetch executor
2020-02-01 04:18:06.640 LogLevel=INFO 1 --- [ecutor-thread40] com.amazon.sqs.javamessaging.SQSSession : Shutting down SessionCallBackScheduler executor
The above 2 messages will display several times and at some point no more messages are consumed from SQS. I don't see any other messages in my log to indicate an issue, but I get no messages from my handlers that they are processing messages (I have 2~) and I can see the AWS SQS queue growing in the number of messages and the age.
~: This exact code was working fine when I had a single handler, this problem started when I added the second one.
Configuration/Code
The first "WARNing" I realize is caused by the currency of the ThreadPoolTaskExecutor, but I can not get a configuration which works properly. Here is my current configuration for the JMS stuff, I have tried various levels of max pool size with no real affect other than the warings start sooner or later based on the pool size
public ThreadPoolTaskExecutor asyncAppConsumerTaskExecutor() {
ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
taskExecutor.setThreadGroupName("asyncConsumerTaskExecutor");
taskExecutor.setThreadNamePrefix("asyncConsumerTaskExecutor-thread");
taskExecutor.setCorePoolSize(10);
// Allow the thread pool to grow up to 4 times the core size, evidently not
// having the pool be larger than the max concurrency causes the JMS queue
// to barf on itself with messages like
// "Number of scheduled consumers has dropped below concurrentConsumers limit, probably due to tasks having been rejected. Check your thread pool configuration! Automatic recovery to be triggered by remaining consumers"
taskExecutor.setMaxPoolSize(10 * 4);
taskExecutor.setQueueCapacity(0); // do not queue up messages
taskExecutor.setWaitForTasksToCompleteOnShutdown(true);
taskExecutor.setAwaitTerminationSeconds(60);
return taskExecutor;
}
Here is the JMS Container Factory we create
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory(SQSConnectionFactory sqsConnectionFactory, ThreadPoolTaskExecutor asyncConsumerTaskExecutor) {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(sqsConnectionFactory);
factory.setDestinationResolver(new DynamicDestinationResolver());
// The JMS processor will start 'concurrency' number of tasks
// and supposedly will increase this to the max of '10 * 3'
factory.setConcurrency(10 + "-" + (10 * 3));
factory.setTaskExecutor(asyncConsumerTaskExecutor);
// Let the task process 100 messages, default appears to be 10
factory.setMaxMessagesPerTask(100);
// Wait up to 5 seconds for a timeout, this keeps the task around a bit longer
factory.setReceiveTimeout(5000L);
factory.setSessionAcknowledgeMode(Session.CLIENT_ACKNOWLEDGE);
return factory;
}
I added the setMaxMessagesPerTask & setReceiveTimeout calls based on stuff found on the internet, the problem persists without these and at various settings (50, 2500L, 25, 1000L, etc...)
We create a default SQS connection factory
public SQSConnectionFactory sqsConnectionFactory(AmazonSQS amazonSQS) {
return new SQSConnectionFactory(new ProviderConfiguration(), amazonSQS);
}
Finally the handlers look like this
#JmsListener(destination = "consumer-event-queue")
public void receiveEvents(String message) throws IOException {
MyEventDTO myEventDTO = jsonObj.readValue(message, MyEventDTO.class);
//messageTask.process(myEventDTO);
}
#JmsListener(destination = "myalert-sqs")
public void receiveAlerts(String message) throws IOException, InterruptedException {
final MyAlertDTO myAlert = jsonObj.readValue(message, MyAlertDTO.class);
myProcessor.addAlertToQueue(myAlert);
}
You can see in the first function (receiveEvents) we just take the message from the queue and exit, we have not implemented the processing code for that.
The second function (receiveAlerts) gets the message, the myProcessor.addAlertToQueue function creates a runnable object and submits it to a threadpool to be processed at some point in the future.
The problem only started (the warning, info and failure to consume messages) only started when we added the receiveAlerts function, previously the other function was the only one present and we did not see this behavior.
More
This is part of a larger project and I am working on breaking this code out into a smaller test case to see if I can duplicate this issue. I will post a follow-up with the results.
In the Mean Time
I'm hoping this is just a config issue and someone more familiar with this can tell me what I'm doing wrong, or that someone can provide some thoughts and comments on how to correct this to work properly.
Thank you!
After fighting this one for a bit I think I finally resolved it.
The issue appears to be due to the "DefaultJmsListenerContainerFactory", this factory creates a new "DefaultJmsListenerContainer" for EACH method with a '#JmsListener' annotation. The person who originally wrote the code thought it was only called once for the application, and the created container would be re-used. So the issue was two-fold
The 'ThreadPoolTaskExecutor' attached to the factory had 40 threads, when the application had 1 '#JmsListener' method this worked fine, but when we aded a second method then each method got 10 threads (total of 20) for listening. This is fine, however; since we stated that each listener could grow up to 30 listeners we quickly ran out of threads in the pool mentioned in 1 above. This caused the "Number of scheduled consumers has dropped below concurrentConsumers limit" error
This is probably obvious given the above, but I wanted to call it out explicitly. In the Listener Factory we set the concurrency to be "10-30", however; all of the listeners have to share that pool. As such the max concurrency has to be setup so that each listeners' max value is small enough so that if each listener creates its maximum that it doesn't exceed the maximum number of threads in the pool (e.g. if we have 2 '#JmsListener' annotated methods and a pool with 40 threads, then the max value can be no more than 20).
Hopefully this might help someone else with a similar issue in the future....
Two questions:
I have an #StreamListener reading from a RabbitMQ channel. I have a pool of 500 ThreadTaskExecutor instances to process the messages as they are read.
The problem is that #StreamListener is reading messages even if the pool is completely utilized.
Caused by: org.springframework.core.task.TaskRejectedException:
Executor [java.util.concurrent.ThreadPoolExecutor#4c15ce96
[Running, pool size = 500, active threads = 500, queued tasks = 1500,
completed tasks = 1025020]] did not accept task:
org.springframework.cloud.sleuth.instrument.async.SpanContinuingTraceCallable#4dc03919
Is there a way to configure #StreamListener so that it only reads from the queue if it has capacity?
In addition, this error trickles up to an UndeclaredThrowableException. IO think its trying to throw the exception back to RabbitMQ so it reques the message. However the end is this:
[WARN] o.s.a.r.l.ConditionalRejectingErrorHandler
Execution of Rabbit message listener failed.
org.springframework.amqp.rabbit.listener.exception
.ListenerExecutionFailedException:
Retry Policy Exhausted
The final result is my message is lost.
Any suggestions for this second issue?
Did you try CallerRunsPolicy for your ThreadPoolTaskExecutor? This way the task won't finish with error and the thread from the SimpleMessageListenerContainer will be busy to do the latest task for just arrived message. As far as you don't use maxConcurrentConsumers option not new concurrent listeners will be raised and the current one (concurrentConsumers = 1 by default) will be busy and no new message is pulled from the Rabbit MQ.
See more info about listener container concurrency in the Docs. This way you may even reconsider your custom ThreadPoolTaskExecutor solution and will fully rely on the built-in mechanism in the Framework.
The maxConcurrency option is exposed for the RabbitMQ Binder Consumer as well.
I am successful in creating a custom sampler by extending AbstractJavaSamplerClient. In the sampler implementation I am making a JMS connection and sending message to a queue. This is working great. If I configure my Thread Group to 100 with Ramp-up 1, my sampler is pushing 100 messages.
Now what I wanted to do is to make the connection only once at the JMeter startup and then reuse the same connection to send messages on each run.
Can anyone explain how to create a connection at JMeter startup and then share the same connection with the sampler.
Note: I can't use the existing JMS publisher because I want to calculate my response time based on different application event not just calculating the time taken to publish the message to JMS.
Thanks in advance.
You can use testStarted method to initialize connections for all threads. Note that this method runs once, before threads are cloned, so in testStarted you need a connection pool, which threads then can take from. For example a very primitive connection pool would be a map with some sequential ID for a key, and connection object. Each thread would take one connection from that pool, based on thread number:
So such simple pool could be initialized as:
#Override
public void testStarted()
{
int maxConnections = getThreadContext().getThreadGroup().getNumThreads();
ConcurrentMap<Integer, Object> connections = new ConcurrentHashMap<Integer, Object>();
for(int i = 0; i < maxConnections; i++)
{
Object connection = //... whatever you need to do to connect
connections.put(new Integer(i), connection);
}
// Put in the context of thread group
JMeterContextService.getContext().getVariables().putObject("MyConnections", connections);
}
(connection object could be a more specific type, based on your needs).
Later you can use it in sample method:
// Get connections pool from context
ConcurrentMap<Integer, Object> connections = (ConcurrentHashMap<Integer, Object>) JMeterContextService.getContext().getVariables().getObject("MyConnections");
// Find connection by thread ID, so each thread goes to a different connection
connections.get(getThreadContext().getThreadNum());
Here I naively assume a perfect mapping between thread number returned at run-time and initial sequential integer I used for connection initialization. Not the best assumption, could be improved, but it's a valid starting point.
You can then close and remove connection in testEnded method. This method also runs once, so we close all connections:
#Override
public void testEnded()
{
for(Entry<Integer, Object> connection : connections.entrySet())
{
connection.close(); // or do whatever you need to close it
connections.remove(connection.getKey());
}
}
Or you could just call connections.clear() when all connections are closed.
Disclosure: I did not test code in this answer directly, but used similar code fragments in the past, and reused them to answer this question. If you find any problems, feel free to update this answer.
when i send message to broker,this exception occasionally occurs.
MQBrokerException: CODE: 2 DESC: [TIMEOUT_CLEAN_QUEUE]broker busy, start flow control for a while
This means broker is too busy(when tps>1,5000) to handle so many sending message request.
What would be the most impossible reason to cause this? Disk ,cpu or other things? How can i fix it?
There are many possible ways.
The root cause is that, there are some messages has waited for long time and no worker thread processes them, rocketmq will trigger the fast failure.
So the below is the cause:
Too many thread are working and they are working very slow to process storing message which makes the cache request is timeout.
The jobs it self cost a long time to process for message storing.
This may be because of:
2.1 Storing message is busy, especially when SYNC_FLUSH is used.
2.2 Syncing message to slave takes long when SYNC_MASTER is used.
In
/broker/src/main/java/org/apache/rocketmq/broker/latency/BrokerFastFailure.java you can see:
final long behind = System.currentTimeMillis() - rt.getCreateTimestamp();
if (behind >= this.brokerController.getBrokerConfig().getWaitTimeMillsInSendQueue()) {
if (this.brokerController.getSendThreadPoolQueue().remove(runnable)) {
rt.setStopRun(true);
rt.returnResponse(RemotingSysResponseCode.SYSTEM_BUSY, String.format("[TIMEOUT_CLEAN_QUEUE]broker busy, start flow control for a while, period in queue: %sms, size of queue: %d", behind, this.brokerController.getSendThreadPoolQueue().size()));
}
}
In common/src/main/java/org/apache/rocketmq/common/BrokerConfig.java, getWaitTimeMillsInSendQueue() method returns
public long getWaitTimeMillsInSendQueue() {
return waitTimeMillsInSendQueue;
}
The default value of waitTimeMillsInSendQueue is 200, thus you can just set it bigger to make the queue waiting for longer time. But if you wanna solve the problem completely, you should follow Jaskey's advice and check your code.