Spring rabbitmq task queue concurrency - spring-boot

I have a task queue which is using a proxy to make http requests. The proxy is limited to 10 concurrent threads / connections. I don't have access to the logs of the proxy.
I am using the following code, and this is making requests on two threads named ntContainer#1-1 and container1. This is resulting in many requests that error due to using too many connections to the proxy.
Is the listener only using 1 default thread and the extra container thread or is there more going on behind the scenes with spring/rabbitmq?
Also how can I debug this further?
#Configuration
public class RabbitMQConfig {
public final static String EXCHANGE_NAME = "my-tx";
public final static String MY_PRODUCT_ROUTING_KEY = "my-product-routing-key";
public final static String MY_PRODUCT_QUEUE = "my-product";
#Bean
public TopicExchange topicExchange() {
return new TopicExchange(EXCHANGE_NAME);
}
#Bean
public Queue myProductQueue() {
return new Queue(MY_PRODUCT_QUEUE);
}
#Bean
Binding myProductBinding() {
return BindingBuilder.bind(myProductQueue()).to(topicExchange()).with(MY_PRODUCT_ROUTING_KEY);
}
#Bean
SimpleMessageListenerContainer container(ConnectionFactory connectionFactory, MessageListenerAdapter messageListenerAdapter) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setQueueNames(MY_PRODUCT_QUEUE);
container.setMessageListener(messageListenerAdapter);
container.setPrefetchCount(1);
container.setConcurrentConsumers(1);
return container;
}
#Bean
MessageListenerAdapter messageListenerAdapter(MyListener myListener) {
return new MessageListenerAdapter(myListener, "process");
}
}
// listener
#RabbitListener(queues = RabbitMQConfig.MY_PRODUCT_QUEUE)
public void process(final Message message) {
// something like this
Jsoup.connect(message.getUrl()).proxy().execute()
}

Oops; I was looking at the question on my 'phone; I skipped past the container bean; I thought the container bean was a container factory not a container.
You have 2 listener containers -
#RabbitListener(queues = RabbitMQConfig.MY_PRODUCT_QUEUE)
public void process(final Message message) {
// something like this
Jsoup.connect(message.getUrl()).proxy().execute()
}
The framework will automatically create a container for that listener (it detects the annotation) and you have explicitly declared another container #Bean.
The proxy is limited to 10 concurrent threads / connections.
Even with 2 containers, you'll only get 2 threads, not 10.

Related

Spring RabbitMQ: one #RabbitListener multiple queues with the priority of the specified queue

I have 2 queues - PRIORITY_QUEUE and SIMPLE_QUEUE. The SIMPLE_QUEUE is large, the number of messages in it reaches several thousand, the PRIORITY_QUEUE can be empty and only sometimes receive a few messages. I need to ensure that messages from the SIMPLE_QUEUE are not read while there are messages in the PRIORITY_QUEUE, and if there are messages in the priority queue, reading the SIMPLE_QUEUE is blocked/paused. When the PRIORITY_QUEUE has received a message, we must pause reading from the SIMPLE_QUEUE.
prefetch property: we read messages by one
spring.rabbitmq.listener.simple.prefetch=1
listener example
#RabbitListener(queues = {priorityQueue , simpleQueue})
public void processMyQueue(String message) {
// if priorityQueue is not empty we shouldn't consume messages from simpleQueue
}
rabbit configuration
#Slf4j
#Configuration
#PropertySource({"...,..."})
#RequiredArgsConstructor
public class RabbitConfiguration {
#Value("${spring.rabbitmq.host}")
private String queueHost;
#Value("${spring.rabbitmq.username}")
private String username;
#Value("${spring.rabbitmq.password}")
private String password;
#Bean
public ConnectionFactory connectionFactory() {
CachingConnectionFactory cachingConnectionFactory = new
CachingConnectionFactory(queueHost);
cachingConnectionFactory.setUsername(username);
cachingConnectionFactory.setPassword(password);
return cachingConnectionFactory;
}
#Bean
public AmqpAdmin amqpAdmin() {
return new RabbitAdmin(connectionFactory());
}
#Bean
public RabbitTemplate rabbitTemplate() {
return new RabbitTemplate(connectionFactory());
}
#Bean(value = "simpleQueue")
public Queue simpleQueue() {
return new Queue("simpleQueue");
}
#Bean(name = "priorityQueue")
public Queue priorityQueue() {
return new Queue("priorityQueue");
}
}
You can't do it with a single listener; use two listeners and stop the simple listener container when the priority listener receives any message; then start the simple queue listener when the priority listener goes idle.
You can use a ListenerContainerIdleEvent to detect an idle listener.
https://docs.spring.io/spring-amqp/docs/current/reference/html/#consumer-events
Use the RabbitListenerEndpointRegistry to stop/start containers.

Spring RabbitMQ - add bindings during runtime

I've been following the guide https://spring.io/guides/gs/messaging-rabbitmq/#initial for setting up rabbitMQ communication via Spring. Queues, Exchanges, Bindings are declared and initialised on Application startup. The code, pretty much like the tutorial:
public static final String SPRINGQUEUE = "springqueue";
public static final String SPRINGEXCHANGE = "springexchange";
#Bean
Queue queue() {
return new Queue(SPRINGQUEUE, false);
}
#Bean
TopicExchange exchange() {
return new TopicExchange(SPRINGEXCHANGE);
}
#Bean
Binding binding(Queue queue, TopicExchange exchange) {
return BindingBuilder.bind(queue).to(exchange).with("routingkey1");
}
#Bean
SimpleMessageListenerContainer container(ConnectionFactory connectionFactory, MessageListener listener) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setQueueNames(SPRINGQUEUE);
container.setMessageListener(listener);
return container;
}
#Bean
MessageListener createListener(Queue queue, TopicExchange exchange) {
return new Listener(queue, exchange);
}
This listens to "routingkey1" just as expected, however, I would like to change the keys to listen for dynamically during runtime. I know this is possible using the RabbitMQ Java client, by simply binding a channel multiple times. Is there any way to do this through spring AMQP as well?
See RabbitAdmin API:
/**
* Declare a binding of a queue to an exchange.
* #param binding a description of the binding to declare.
*/
void declareBinding(Binding binding);
for example.
You don't listen to the routing key: you listen to the queue. The routing key is a part of publisher logic: you publish a message to the exchange with some routing key. The RabbitMQ broker already decides to what queue place this message to. Therefore see if your expectations with dynamic bindings and listening is what is possible with an AQMP protocol.

Spring Integration: connection to multiple MQ servers by config

I do have a Spring Boot 5 application and I also have it running against one IBM MQ server.
Now we want it to connect to three or more MQ servers. My intention is now to just add XY connection infos to the environment and then I get XY MQConnectionFactory beans and al the other beans that are needed for processing.
At the moment this is what I have:
#Bean
#Qualifier(value="MQConnection")
public MQConnectionFactory getIbmConnectionFactory() throws JMSException {
MQConnectionFactory factory = new MQConnectionFactory();
// seeting all the parameters here
return factory;
}
But this is quite static. Is there an elegant way of doing this?
I stumbled about IntegrationFlow. Is this a possibly working solution?
Thanks for all your tipps!
KR
Solution
Based on Artem Bilan's response I built this class.
#Configuration
public class ConnectionWithIntegrationFlowMulti {
protected static final Logger LOG = Logger.create();
#Value("${mq.queue.jms.sources.queue.queue-manager}")
private String queueManager;
#Autowired
private ConnectionConfig connectionConfig;
#Autowired
private SSLSocketFactory sslSocketFactory;
#Bean
public MessageChannel queureader() {
return new DirectChannel();
}
#Autowired
private IntegrationFlowContext flowContext;
#PostConstruct
public void processBeanDefinitionRegistry() throws BeansException {
Assert.notEmpty(connectionConfig.getTab().getLocations(), "At least one CCDT file locations must be provided.");
for (String tabLocation : connectionConfig.getTab().getLocations()) {
try {
IntegrationFlowRegistration theFlow = this.flowContext.registration(createFlow(tabLocation)).register();
LOG.info("Registered bean flow for %s with id = %s", queueManager, theFlow.getId());
} catch (JMSException e) {
LOG.error(e);
}
}
}
public IntegrationFlow createFlow(String tabLocation) throws JMSException {
LOG.info("creating ibmInbound");
return IntegrationFlows.from(Jms.messageDrivenChannelAdapter(getConnection(tabLocation)).destination(createDestinationBean()))
.handle(m -> LOG.info("received payload: " + m.getPayload().toString()))
.get();
}
public MQConnectionFactory getConnection(String tabLocation) throws JMSException {
MQConnectionFactory factory = new MQConnectionFactory();
// doing stuff
return factory;
}
#Bean
public MQQueue createDestinationBean() {
LOG.info("creating destination bean");
MQQueue queue = new MQQueue();
try {
queue.setBaseQueueManagerName(queueManager);
queue.setBaseQueueName(queueName);
} catch (Exception e) {
LOG.error(e, "destination bean: Error for integration flow");
}
return queue;
}
}
With Spring Integration you can create IntegrationFlow instances dynamically at runtime. For that purpose there is an IntegrationFlowContext with its registration() API. The returned IntegrationFlowRegistrationBuilder as a callback like:
/**
* Add an object which will be registered as an {#link IntegrationFlow} dependant bean in the
* application context. Usually it is some support component, which needs an application context.
* For example dynamically created connection factories or header mappers for AMQP, JMS, TCP etc.
* #param bean an additional arbitrary bean to register into the application context.
* #return the current builder instance
*/
IntegrationFlowRegistrationBuilder addBean(Object bean);
So, your MQConnectionFactory instances can be populated alongside with the other flow, used as references in the particular JMS components and registered as beans, too.
See more info in docs: https://docs.spring.io/spring-integration/docs/5.2.3.RELEASE/reference/html/dsl.html#java-dsl-runtime-flows
If you are fine with creating them statically, you can create the beans as you are now (each having a unique qualifier), but you can access them all dynamically in your services / components by having an #Autowired List<MQConnectionFactory> field or #Autowired Map<String, MQConnectionFactory> field. Spring will automatically populate the fields with all of the beans of type MQConnectionFactory
In the the Map implementation, the String will be the qualifier value.
If you also want to create the beans dynamically based on some properties, etc, it gets a little more complicated. You will need to look into something along the lines of instantiating beans at runtime

AWS SQS (queue) with Spring Boot - performance issues

I have a service that reads all messages from AWS SQS.
#Slf4j
#Configuration
#EnableJms
public class JmsConfig {
private SQSConnectionFactory connectionFactory;
public JmsConfig(
#Value("${amazon.sqs.accessKey}") String awsAccessKey,
#Value("${amazon.sqs.secretKey}") String awsSecretKey,
#Value("${amazon.sqs.region}") String awsRegion,
#Value("${amazon.sqs.endpoint}") String awsEndpoint) {
connectionFactory = new SQSConnectionFactory(
new ProviderConfiguration(),
AmazonSQSClientBuilder.standard()
.withCredentials(new AWSStaticCredentialsProvider(
new BasicAWSCredentials(awsAccessKey, awsSecretKey)))
.withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(awsEndpoint, awsRegion))
.build());
}
#Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory() {
DefaultJmsListenerContainerFactory factory =
new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(this.connectionFactory);
factory.setDestinationResolver(new DynamicDestinationResolver());
factory.setConcurrency("3-10");
factory.setSessionAcknowledgeMode(Session.CLIENT_ACKNOWLEDGE);
factory.setReceiveTimeout(2000L); //??????????
return factory;
}
#Bean
public JmsTemplate defaultJmsTemplate() {
return new JmsTemplate(this.connectionFactory);
}
I've heard about long polling so I wonder how I could use it in my case. I wonder how this listener works - I do not want to create unnecessary calls to the AWS SQS.
My listener that reads messages and converts them to the Object and saves on Redis db:
#JmsListener(destination = "${amazon.sqs.destination}")
public void receive(String requestJSON) throws JMSException {
log.info("Received");
try {
Trace trace = Trace.fromJSON(requestJSON);
traceRepository.save(trace);
(...)
I'd like to know your opinions - what is the best approach to minimalize unnecessary calls to SQS to get messages.
Maybe shoud I use for example
factory.setReceiveTimeout(2000L);
Unfortunately there is too little information in Internet about it
Thanks,
Matthew

Spring Boot JMS AutoStartup

I am trying to start/stop manually JMS listeners in my Spring Boot App. I am currently using the following configuration to my container factory:
#EnableJms
public class ConfigJms {
...
#Bean(name = "queueContainerFactory")
public JmsListenerContainerFactory<?> queueContainerFactory(ConnectionFactory cf) {
ActiveMQConnectionFactory amqCf = (ActiveMQConnectionFactory) cf;
amqCf.setTrustAllPackages(true);
SimpleJmsListenerContainerFactory factory = new SimpleJmsListenerContainerFactory();
factory.setConnectionFactory(amqCf);
**factory.setAutoStartup(false);**
return factory;
}
...
}
After testing factory.setAutoStartup(false); I am quite confused because even indicating to do not start any listener for this factory container, the listeners are already registered and started when the context starts.
I tested this situation by using a jmsListenerEndpointRegistry.
jmsListenerEndpointRegistry.isAutoStartup() is true and
jmsListenerEndpointRegistry. isRunning () is true before execute jmsListenerEndpointRegistry.start();
Is it necessary to configure anything else? Maybe I am omitting to override some auto-configuration.
EDIT 1: Invalid status of JmsListenerEndpointRegistry listeners
I detected a couple of inconsistences in my beans:
jmsListenerEndpointRegistry.getListenerContainerIds().size() is always 0.
jmsListenerEndpointRegistry.isAutoStartup() is just a return true method.
Even if I register a couple of listeners with annotations like this:
#JmsListener(containerFactory="queueContainerFactory", destination = "${dest}")
jmsListenerEndpointRegistry does not show information about these listeners status but they are connected to ActiveMQ on startup. (Checking the ActiveMQ admin console)
EDIT 2: #JmsListener starts even auto-startup is set to false
I checked the jmsListenerEndpointRegistry for each container and I do not know if this is a bug or I am not correctly defining the configuration. However, I am just defining the container factory as explained before with AUTO-START set to false and the both listeners are started and consuming messages (running).
From my Log file:
jmsListenerEndpointRegistry ID <org.springframework.jms.JmsListenerEndpointContainer#1>, Auto-Startup <false>, Running <true>
jmsListenerEndpointRegistry ID <org.springframework.jms.JmsListenerEndpointContainer#0>, Auto-Startup <false>, Running <true>
You must have something else going on - I just wrote a quick boot app (1.4.1) and the container is not started...
#SpringBootApplication
public class So39654027Application {
public static void main(String[] args) {
ConfigurableApplicationContext context = SpringApplication.run(So39654027Application.class, args);
JmsListenerEndpointRegistry reg = context.getBean(JmsListenerEndpointRegistry.class);
MessageListenerContainer listenerContainer = reg.getListenerContainer("foo");
System.out.println(listenerContainer.isRunning());
}
#Bean(name = "queueContainerFactory")
public JmsListenerContainerFactory<?> queueContainerFactory(ConnectionFactory cf) {
ActiveMQConnectionFactory amqCf = (ActiveMQConnectionFactory) cf;
amqCf.setTrustAllPackages(true);
SimpleJmsListenerContainerFactory factory = new SimpleJmsListenerContainerFactory();
factory.setConnectionFactory(amqCf);
factory.setAutoStartup(false);
return factory;
}
#JmsListener(id="foo", destination = "so39654027", containerFactory = "queueContainerFactory")
public void listen(String foo) {
System.out.println(foo);
}
}
and...
2016-09-23 09:24:33.428 INFO 97907 --- [ main] com.example.So39654027Application : Started So39654027Application in 1.193 seconds (JVM running for 2.012)
false
I suggest you use a debugger in the container's start() method to see why it's being started.
Order is important, factory.setAutoStartup(autoStartup) after configure.
#Bean
public JmsListenerContainerFactory<?> ShipmentListenerFactory(#Qualifier("GSUBCachingConnectionFactory") CachingConnectionFactory connectionFactory,
DefaultJmsListenerContainerFactoryConfigurer configurer) {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
// This provides all boot's default to this factory, including the message converter
// Added ability to disable not start listener
boolean autoStartup = env.getProperty("app-env.CKPT_QUEUE_AUTO_START",Boolean.class,true);
log.info("[MQ] CKPT_QUEUE_AUTO_START:{}",autoStartup);
configurer.configure(factory, connectionFactory);
factory.setAutoStartup(autoStartup);
// You could still override some of Boot's default if necessary.
return factory;
}

Resources