I'm using 2.1.0.RELEASE version of Spring boot with AMQP. Unfortunetly I need to connect to several different RabbitMQ servers. I had to exclude RabbitAutoConfiguration.class besause due to changes in above version of spring it's impossible to start without one of the ConnectionFactory beans as primary, but even if I set one of them as #Primary, obviously it doesn't work, because how would amqp/spring-boot know which queue to create on which server...
so, is it possible to automatically create queues on different servers with auto configuration disabled?
Yes, you need a RabbitAdmin for each connection factory.
By default all components will be declared on all brokers, but you can add conditions. See Conditional Declaration.
By default, all queues, exchanges, and bindings are declared by all RabbitAdmin instances (assuming they have auto-startup="true") in the application context.
#Bean
public Queue queue1() {
Queue queue = new Queue("foo");
queue.setAdminsThatShouldDeclare(admin1());
return queue;
}
Related
Auto deletion of queues should be turned off,
1)If set "ems_value=true" in application.property file
#ConditionalOnProperty(name = "ems_value")
public class JmsConfiguration {
// Here get JMS connection
In this case what are mentioned queue names in application property file Its created by spring.
2)If set "ems_value=false" in application.property file
Queues deleted by spring class.
How can i disable or stop to delete queues.
It will only work on startup anyway, but you can always use 2 different beans that will do required job for you. Have once component with
#ConditionalOnProperty(name = "ems_value", havingValue="true")
that will do whatever you require, and have another component with
#ConditionalOnProperty(name = "ems_value", havingValue="false")
that will do cleanup operations only.
We use spring kafka configuration to receive messages from upstream systems.
We have java configuration for topic configuration
#Bean(id="firstcontainer")
protected ConcurrentMessageListenerContainer createContainerInstance(...) {
//topics addition
}
#Bean(id="secondcontainer")
protected ConcurrentMessageListenerContainer createContainerInstance(...) {
//topics addition
}
#KafkaListener(firstcontainer)
public void listenerFirst(){
}
#KafkaListener(secondcontainer)
public void listenerSecond(){
}
This code works perfectly fine as we have seperate containerfactory.
Now we have requirement to spin up mulitple instances of this application where one instance will listen to firstContainer and secondContainer will be disabled
And For second instance, it will only enable secondContainer and disable firstContainer.
Can someone help to understand if it is possible to disable listening from a topic(list of topics)?
Your two instances (or many) can be identical and accept topic list from the external configuration. The #KafkaListener allows to do that.
There is Spring #Profile functionality, if you still want to keep several beans in your application. This way you should sever your #KafkaListener method to different classes and mark their component with an appropriate #Profile, which, again, can be activated externally.
The Apache Kafka has a concept as Consumer Group meaning that all consumers in the same group are joining to the broker, but only one of them will consume records from single partition in the topic. This way independently of the number of instances of your application you still will have a consistency because there is nothing to worry about duplicates in case of proper Kafka groups usage.
In my Spring Boot application I have to implement an import service. Users can submit a bunch of JSON files and application will try to import the data from these files. Depending on the data amount at JSON files the single import process can take a 1 or 2 hours.
I do not want to block the users during the import process so I plan to accept the task for importing and notify user that this data is scheduled for processing. I'll put the data into the queue and a free queue-consumer on the other end will start the import process. Also, I need to have a possibility to monitor a jobs in the queue, terminate them if needed.
Right now I'm thinking to use Embedded Apache ActiveMQ in order to introduce message producer and consumer logic but before this I'd like to ask - from the architecture point of view - is it a good choice for the described task or it can be implemented with a more appropriate tools.. like for example plain Spring #Async and so on ?
It is possible to treat files concurrently with Camel like this
from("file://incoming?maxMessagesPerPoll=1&idempotent=true&moveFailed=failed&move=processed&readLock=none").threads(5).process()
Take a look at http://camel.apache.org/file2.html
But i think that it is better for your requirements to use a standalone ActiveMQ, a standalone service to move files to ActiveMQ and standalone consumer to be capable to kill or restart each one independently.
It is better to use ActiveMQ as you said and you can easily create a service to move messages to a queue with Camel like this :
CamelContext context = new DefaultCamelContext();
ConnectionFactory connectionFactory = new ActiveMQConnectionFactory("vm://localhost?broker.persistent=true");
context.addComponent("test-jms", JmsComponent.jmsComponentAutoAcknowledge(connectionFactory));
context.addRoutes(new RouteBuilder() {
public void configure() {
// convertBodyTo to use TextMessage or maybe send them as file to the Queue from("file://testFolderPath").convertBodyTo(String.class).to("test-jms:queue:test.queue");
}
});
context.start();
Here some examples
http://www.programcreek.com/java-api-examples/index.php?api=org.apache.camel.component.jms.JmsComponent
https://skills421.wordpress.com/2014/02/08/sending-local-files-to-a-jms-queue/
https://github.com/apache/camel/blob/master/examples/camel-example-jms-file/src/main/java/org/apache/camel/example/jmstofile/CamelJmsToFileExample.java
https://github.com/apache/camel/tree/master/examples
To monitor and manage you can use jmx with VisualVM or Hawtio http://hawt.io/getstarted/index.html
http://camel.apache.org/camel-jmx.html
To consume you can use DefaultMessageListenerContainer with concurrent consumers on the queue and for this you need to change the prefetchPolicy on the ConnectionFactory of the DefaultMessageListenerContainer , Multithreaded JMS client ActiveMQ
I have a (legacy) TCP service that has multiple processes. Each process runs on the same host, but on a different port. The service is single threaded, so the way to increase throughput is to round-robin each request across each of the ports.
I am providing an AMQP exposure to this legacy application. Its very simple - take a string off the AMQP queue, pass it to the application, and return the response string to the AMQP reply queue.
This works great on a single port. However, i'd like to fan out the requests across all the ports.
Spring Integration seems to only provide AbstractClientConnectionFactory implementations that either connect directly to a single host/port (TcpNetClientConnectionFactory) or maintain a pool of connections to a single host/port (CachingClientConnectionFactory). There arent any that pool connections between a single host and multiple ports.
I have attempted to write my own AbstractClientConnectionFactory that maintains a pool of AbstractClientConnectionFactory objects and round-robins between them. However, I have struck several issues to do with handing the TCP connections when the target service goes away or the network is interrupted that I have not been able to solve.
There is also the approach taken by this question: Spring Integration 4 - configuring a LoadBalancingStrategy in Java DSL but the solution to that was to hardcode the number of endpoints. In my case, the number of endpoints is only known at runtime and is a user-configurable setting.
So, basically I need to create a TcpOutboundGateway per port dynamically at runtime and somehow register it in my IntegrationFlow. I have attempted the following:
#Bean
public IntegrationFlow xmlQueryWorkerIntegrationFlow() {
SimpleMessageListenerContainer inboundQueue = getMessageListenerContainer();
DirectChannel rabbitReplyChannel = MessageChannels.direct().get();
IntegrationFlowBuilder builder = IntegrationFlows
.from(Amqp.inboundGateway(inboundQueue)
.replyChannel(rabbitReplyChannel))
/* SOMEHOW DO THE ROUND ROBIN HERE */
//I have tried:
.channel(handlerChannel()) //doesnt work, the gateways dont get started and the message doesnt get sent to the gateway
//and I have also tried:
.handle(gateway1)
.handle(gateway2) //doesnt work, it chains the handlers instead of round-robining between them
//
.transform(new ObjectToStringTransformer())
.channel(rabbitReplyChannel);
return builder.get();
}
#Bean
//my attempt at dynamically adding handlers to the same channel and load balancing between them
public DirectChannel handlerChannel() {
DirectChannel channel = MessageChannels.direct().loadBalancer(new RoundRobinLoadBalancingStrategy()).get();
for (AbstractClientConnectionFactory factory : generateConnections()) {
channel.subscribe(generateTcpOutboundGateway(factory));
}
return channel;
}
Does anyone know how I can solve this problem?
See the dynamic ftp sample - in essence each outbound gateway goes in its own application context and the dynamic router routes to the appropriate channel (for which the outbound adapter is created on demand if necessary).
Although the sample uses XML, you can do the same thing with java configuration, or even with the Java DSL.
See my answer to a similar question for multiple IMAP mail adapters using Java configuration and then a follow-up question.
I am working on a brand new project in which I need to have listeners that will consume messages from several queues (no need to have producer for now).
Starting from scratch, I am using the last Spring JMS version (4.1.2).
Here is an extract of my configuration file:
<bean id="cachedConnectionFactory"
class="org.springframework.jms.connection.CachingConnectionFactory"
p:targetConnectionFactory-ref="jmsConnectionFactory"
p:sessionCacheSize="3" />
<bean id="jmsListenerContainerFactory"
class="org.springframework.jms.config.DefaultJmsListenerContainerFactory"
p:connectionFactory-ref="cachedConnectionFactory"
p:destinationResolver-ref="jndiDestinationResolver"
p:concurrency="3-5"
p:receiveTimeout="5000" />
But I think I may be wrong since DefaultJmsListenerContainerFactory will build regular DefaultMessageListenerContainerS. And, as stated in the doc, CachingConnectionFactory should not be used with a message listener container...
Even if I am using the new Spring 4.1 DefaultJmsListenerContainerFactory class the answer from post is still valid (cacheConsumers = true can be an issue + don't need to cache sessions for listener containers because the sessions are long lived) , right?
Instead of using the CachingConnectionFactory, I should use the SingleConnectionFactory (and not directly the broker implementation one)?
If the SingleConnectionFactory class should indeed be used, is the "reconnectOnException" property should be set to true (as it is done in the CachingConnectionFactory) or does the new "setBackOff" method (from DefaultJmsListenerContainerFactory) deals with the same kind of issues?
Thanks for any tips
Correct.
There's not really much benefit in using a SingleConnectionFactory unless you want to share a single connection across multiple containers; the DMLC will use a single connection from the vendor factory by default for all consumer threads (cacheLevel >= CACHE_CONNECTION), unless a TransactionManager is configured.
The container(s) will handle reconnection - even before the 'new' backOff property - backOff just adds more sophistication to the reconnection algorithm - it used to just retry every n seconds (5 by default).
As stated in the answer you cited, it's ok to use a CCF as long as you disable consumer caching.
Correction: Yes, when using the SingleConnectionFactory, you do need to set reconnectOnException to true in order for the container to properly recover its connection. Otherwise, it simply hands out the stale connection.