Referring to Spring kafka doc, I am trying to implement a spring boot application which listens to a kafka topic and acks manually.
#SpringBootApplication
public class TestApplication {
public static void main(String[] args) {
SpringApplication.run(TestApplication.class, args);
}
#KafkaListener(topics = "sample_log", containerFactory = "kafkaManualAckListenerContainerFactory")
public void listen(ConsumerRecord<?, ?> cr, Acknowledgment ack) throws Exception {
process(cr);
ack.acknowledge();
}
}
But when I ran the application, it shows:
***************************
APPLICATION FAILED TO START
***************************
Description:
A component required a bean named 'kafkaManualAckListenerContainerFactory' that could not be found.
Action:
Consider defining a bean named 'kafkaManualAckListenerContainerFactory' in your configuration.
As far as I'm concerned, Spring Kafka may have defined kafkaManualAckListenerContainerFactory bean since it is a provided feature. How could I fix the error WITHOUT manually define the bean?
You don’t need that containerFactory = "kafkaManualAckListenerContainerFactory" configuration. Just remove it altogether and rely on the provided by Spring Boot auto-configuration for Kafka: https://docs.spring.io/spring-boot/docs/2.0.1.RELEASE/reference/htmlsingle/#boot-features-kafka
Related
I have the java configuration for the Spring Data Elaticsearch(using Transport Client) and ESTemplate.
Here some except:
#Configuration
#EnableElasticsearchRepositories(basePackages = "subpackage-in-this-project")
#PropertySource("file:path-to-file")
public class ESConfig {
#Bean
ElasticsearchTemplate elasticsearchTemplate(Client client) {
return new ElasticsearchTemplate(client);
}
#Bean
Client client() {
// configuration of the ES client
}
}
And I have a config that extends the one above in the different project.
#Configuration
#ComponentScan("package-prefix-that-matches-packages-in-both-projects")
#EnableElasticsearchRepositories(basePackages = "subpackage-in-this-project")
#PropertySource("file:same-path-to-file-as-in-the-config-above")
public class ExtendedESConfig extends ESConfig {
#Value("index-name")
private String indexName;
#Bean
public String indexName() {
return indexName;
}
}
Upon executing a third Spring Boot application, which uses the dependency on the project with ExtendedESConfig, I get this and I can't quite understand why does it happen, started to happen after upgrading to 2.2.9.RELEASE from 2.0.5.RELEASE Spring Boot version.
***************************
APPLICATION FAILED TO START
***************************
Description:
The bean 'elasticsearchTemplate', defined in class path resource [my/package/ESConfig.class], could not be registered. A bean with that name has already been defined in class path resource [my/other/package/ExtendedESConfig.class] and overriding is disabled.
Action:
Consider renaming one of the beans or enabling overriding by setting spring.main.allow-bean-definition-overriding=true
2020-08-30 16:49:46 ERROR [main] org.springframework.boot.diagnostics.LoggingFailureAnalysisReporter:40 -
Important remark from my comment:
... sadly, I am not the one who wrote this ES config and built the whole infrastructure around it. ...
At the time of this question, I don't own ExtendedESConfig nor can change it.
Or you can add the next property to your application.properties :
spring.main.allow-bean-definition-overriding=true
The default behaviour of overriding bean has been disabled in Spring Boot 2.1. Spring Boot 2.1 Release Notes
Since you don't own / or don't want to modify both configuration classes. You can exclude parent configuration form your SpringBootApplication class using #ComponentScan
#SpringBootApplication
#ComponentScan(excludeFilters =
{#ComponentScan.Filter(type = FilterType.ASSIGNABLE_TYPE, classes = ESConfig.class)})
public class DemoApplication {
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
}
I had a similar problem with my custom springSecurityFilterChain method, in my #Configuration class. The application told me it couldn't create a Bean named springSecurityFilterChain since it was defined elsewhere (in my case, in a Spring Security package, which, as is your case, I couldn't modify).
I found the solution here and it amounted to simply changing my custom method's name; I chose customFilterChain. So it went from
#Bean
public SecurityFilterChain springSecurityFilterChain(HttpSecurity http) throws Exception {
return http
.csrf().disable()
// etc
}
to:
#Bean
public SecurityFilterChain customFilterChain(HttpSecurity http) throws Exception {
return http
.csrf().disable()
// etc
}
Amazingly it worked. Like the article says, Spring builds the bean using the method's name by default. Hope it helps.
find in your modul: resources/application.properties and write:
spring.main.allow-bean-definition-overriding=true
it help you, you need to enable the beans override mechanism.
I want to make my spring boot application fail on start if it cannot connect to the kafka broker. My application is only publishing messages to topics. I added this line to my properties file but no luck so far spring.kafka.admin.fail-fast=true.
Addition on how to speed up fail-fast
TL;DR Spring Boot 2.4.5 auto configuration does not let you speed up fail-fast using env parameters. Add this to your #Configuration to get 10 seconds timeout:
#Bean
public KafkaAdmin kafkaAdmin(#Autowired KafkaProperties properties) {
KafkaAdmin kafkaAdmin = new KafkaAdmin(properties.buildAdminProperties());
kafkaAdmin.setFatalIfBrokerNotAvailable(properties.getAdmin().isFailFast());
/* speed up fail fast */
kafkaAdmin.setOperationTimeout(5);
kafkaAdmin.setCloseTimeout(5);
return kafkaAdmin;
}
More detailed answer
Fail-fast occurs when initialize() method of class org.springframework.kafka.core.KafkaAdmin is executed. This method may block:
If new topics are found, it blocks on topic creation for up to operationTimeout
If topic creation failed, it blocks on releasing resources for up to closeTimeout
By default those values are 30 and 10 seconds accordingly (hardcoded in the class mentioned above). You can redefine them using set methods: setOperationTimeout(int sec), setCloseTimeout(int sec).
What about Spring Boot? KafkaAdmin bean is created in class org.springframework.boot.autoconfigure.kafka.KafkaAutoConfiguration. As you can guess, Spring Boot simply does not set timeouts:
#Bean
#ConditionalOnMissingBean
public KafkaAdmin kafkaAdmin() {
KafkaAdmin kafkaAdmin = new KafkaAdmin(this.properties.buildAdminProperties());
kafkaAdmin.setFatalIfBrokerNotAvailable(this.properties.getAdmin().isFailFast());
return kafkaAdmin;
}
fail-fast will only work if there is at least one NewTopic bean in the context (so the admin will try to check if the topic exists and create it if not).
#SpringBootApplication
public class So55177700Application {
public static void main(String[] args) {
SpringApplication.run(So55177700Application.class, args);
}
#Bean
public NewTopic topic() {
return new NewTopic("so55177700", 1, (short) 1);
}
}
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2019-03-15 09:42:49.555 ERROR 41793 --- [ main] o.s.boot.SpringApplication : Application run failed
java.lang.IllegalStateException: Could not configure topics
I am learning JMS with spring boot and nice to know that spring boot comes with embed Active MQ JMS broker.
I started from spring page on how to achieve this and it works like charm. Now i went little further and create two separate spring boot application one containing jms sender code and another containing receiver code.
I tried starting and application failed as both application are using same port for JMS. I fixed this by including this on one application
#Bean
public BrokerService broker() throws Exception {
final BrokerService broker = new BrokerService();
broker.addConnector("tcp://localhost:61616");
broker.addConnector("vm://localhost");
broker.setPersistent(false);
return broker;
}
But now sender is sending message successfully but receiver is doing nothing. I search on stackoverflow and look at this and this. And they are saying:
If you want to use JMS in production, it would be much wiser to avoid using Spring Boot embedded JMS brokers and host it separately. So 3 node setup would be preferred for PROD.
So my questions are:
1. What is the purpose of putting both jms sender and receiver on same application? Is there any practical example
2. Is it really not possible to use spring boot embedded JMS to communicate two separate application.
You might have sender and receiver in the same application if requests arrive in bursts and you want to save them somewhere before they are processed, in case of a server crash. You typically still wouldn't use an embedded broker for that.
Embedded brokers are usually used for testing only.
You can, however, run an embedded broker that is accessible externally; simply fire up a BrokerService as you have, but the other app needs to connect with the tcp://... address, not the vm://....
EDIT
App1:
#SpringBootApplication
#RestController
public class So52654109Application {
public static void main(String[] args) {
SpringApplication.run(So52654109Application.class, args);
}
#Bean
public BrokerService broker() throws Exception {
final BrokerService broker = new BrokerService();
broker.addConnector("tcp://localhost:61616");
broker.setPersistent(false);
broker.start();
return broker;
}
#Autowired
private JmsTemplate template;
#RequestMapping(path = "/foo/{id}")
public String foo(#PathVariable String id) {
template.convertAndSend("someQueue", id);
return id + ": thank you for your request, we'll send an email to the address on file when complete";
}
}
App2:
application.properties
spring.activemq.broker-url=tcp://localhost:61616
and
#SpringBootApplication
public class So526541091Application {
public static void main(String[] args) {
SpringApplication.run(So526541091Application.class, args);
}
#JmsListener(destination = "someQueue")
public void process(String id) {
System.out.println("Processing request for id");
}
}
Clearly, for a simple app like this you might just run the listener in the first app.
However, since there is no persistence of messages with this configuration, you would likely use an external broker for a production app (or enable persistence).
Spring boot non-web application, when start it has below error
Caused by: org.springframework.context.ApplicationContextException: Unable to start EmbeddedWebApplicationContext due to missing EmbeddedServletContainerFactory bean.
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.getEmbeddedServletContainerFactory(EmbeddedWebApplicationContext.java:185) ~[spring-boot-1.3.5.RELEASE.jar:1.3.5.RELEASE]
Then I tried below manner
new SpringApplication().setWebEnvironment(false);
then start it still have above error.
Then tried
#SpringBootApplication(exclude={SpringDataWebAutoConfiguration.class})
but still have the same error.
At last I tried add below configuration in application.properties
spring.main.web-environment=false
this time it works.
Why the first two manner cannot work?
Starting from Spring Boot 2.0
-web(false)/setWebEnvironment(false) is deprecated and instead Web-Application-Type can be used to specify
Application Properties
spring.main.web-application-type=NONE
# REACTIVE, SERVLET
or SpringApplicationBuilder
#SpringBootApplication
public class SpringBootDisableWebEnvironmentApplication {
public static void main(String[] args) {
new SpringApplicationBuilder(SpringBootDisableWebEnvironmentApplication.class)
.web(WebApplicationType.NONE) // .REACTIVE, .SERVLET
.run(args);
}
}
Where WebApplicationType:
NONE - The application should not run as a web application and should not start an embedded web server.
REACTIVE - The application should run as a reactive web application and should start an embedded reactive web server.
SERVLET - The application should run as a servlet-based web application and should start an embedded servlet web server.
Courtesy: Another SO Answer
This answer is obsolete. Please note the other answer for Spring Boot 2.0
Original answer for Spring Boot 1.x:
The reason this config is not working because these are two different instances:
new SpringApplication().setWebEnvironment(false);
SpringApplication.run(SpringBootDisableWebEnvironmentApplication.class, args);
You are disabling setWebEnvironment(false) in new SpringApplication() object and calling static method run() on SpringApplication.run(...) which is different one.
I figured out 3 ways to do this:
#SpringBootApplication
public class SpringBootDisableWebEnvironmentApplication implements CommandLineRunner{
public static void main(String[] args) throws Exception {
// Method#1: Using SpringApplicationBuilder.
SpringApplication springApplication =
new SpringApplicationBuilder()
.sources(SpringBootDisableWebEnvironmentApplication.class)
.web(false)
.build();
springApplication.run(args);
//--------------------------------------------------------
// Method#2: Using SpringBootDisableWebEnvironmentApplication.
// SpringBootDisableWebEnvironmentApplication springBootDisableWebEnvironmentApplication =
// new SpringBootDisableWebEnvironmentApplication();
// springBootDisableWebEnvironmentApplication.run(args);
//--------------------------------------------------------
// Method#3: Using SpringApplication().
// SpringApplication springApplication = new SpringApplication();
// springApplication.setWebEnvironment(false);
//
// Set<Object> sources = new HashSet<>();
// sources.add(SpringBootDisableWebEnvironmentApplication.class);
// springApplication.setSources(sources);
// springApplication.run(args);
//--------------------------------------------------------
}
#Override
public void run(String... arg0) throws Exception {
System.out.println("Hello, Spring Boot gives many options ;)");
}
}
Here is the complete working Project.
And you don't need to exclude below config:
#SpringBootApplication(exclude = {EmbeddedServletContainerAutoConfiguration.class,
WebMvcAutoConfiguration.class})
Because you don't have spring-boot-starter-web dependency in your pom.xml
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
As already noted in other answers the simplest solution is to add a property:
spring.main.web-application-type=NONE for Spring-boot 2.x
spring.main.web-environment=false for Spring-boot 1.x
But the simplest solution is NOT the best one, it's just quick&dirty.
Spring-boot has a lot of autoconfigurations that are triggered by the content of your classpath, so you probably have some unnecessary web-related dependency in your app.
I had a Spring-batch application that was giving
Unable to start ServletWebServerApplicationContext due to missing ServletWebServerFactory bean.
It was caused by the presence of javax.servlet-api in my POM. I removed it and the problem disappeared.
public static void main(String[] args) {
SpringApplication(Application.class)
.setWebApplicationType(WebApplicationType.NONE)
.run(args);
}
is another way instead of using the SpringApplicationBuilder.
I have a simple Spring Integration application which runs on Tomcat (v7.0.x) and consumes messages off a Websphere MQ Queue. When I un-deploy the WAR from the Tomcat server, the WAR un-deploys okay but, a JMS listener thread is left running on the Tomcat server which will still consume messages off the Websphere MQ Queue. I am therefore assuming that I am not handling the JMS listener clean up part of the application properly?
Here is the stack I am using:
Java 8
Tomcat 7.0.55
Spring Integration 4.0.4
Spring Integration Java Dsl 1.0.0.M3
In terms of my SI application's configurations, I have a JmsConfig class:
#Configuration
#ComponentScan
public class JmsConfig {
#Autowired
private Properties jndiProperties;
private ConnectionFactory mqConnectionFactory() throws NamingException {
Context ctx = new InitialContext(jndiProperties);
try {
MQQueueConnectionFactory connectionFactory = (MQQueueConnectionFactory)
ctx.lookup("jms/service/SERVICE_QCF");
return connectionFactory;
} finally {
ctx.close();
}
}
#Bean
public ConnectionFactory cachingConnectionFactory() throws NamingException {
CachingConnectionFactory connectionFactory = new CachingConnectionFactory();
connectionFactory.setTargetConnectionFactory(mqConnectionFactory());
connectionFactory.setSessionCacheSize(10);
return connectionFactory;
}
}
I have an Integration config class:
#Configuration
#EnableIntegration
public class IntegrationConfig {
#Autowired
private ConnectionFactory cachingConnectionFactory;
#Bean
public IntegrationFlow requestFlow() {
return IntegrationFlows
.from(Jms.inboundAdapter(cachingConnectionFactory).destination(
"SERVICE_QUEUE_NAME"), c -> {
c.poller(Pollers.fixedRate(100));
})
.channel("request.service.ch").get();
}
}
Web Initialiser config class:
#Configuration
public class WebInitialiser implements WebApplicationInitializer {
public void onStartup(ServletContext servletContext)
throws ServletException {
AnnotationConfigWebApplicationContext rootContext =
new AnnotationConfigWebApplicationContext();
rootContext.register(ApplicationConfig.class, JmsConfig.class,
IntegrationConfig.class, DatabaseConfig.class);
servletContext.addListener(new ContextLoaderListener(rootContext));
}
}
During the un-deploy stage I see the following in the catalina logs which may or may not be related:
SEVERE: The web application [/service-a] appears to have started a thread named [Thread-7] but has failed to stop it. This is very likely to create a memory leak.
Is there anything that I have yet NOT set or configured or annotated in order to ensure that the deployment's JMS listener thread is cleaned up from Tomcat's JVM during the WAR's un-deploy stage?
Thanks in advance,
PM.
To ensure that JMS listener threads are cleared up upon the application's un-deploy stage, I simply created a CachingConnectionFactory bean with its targetConnectionFactory being that of the MQConnectionFactory. Then, in the Spring Integration flows, I simply pass in the cachingConnectionFactory bean to the JMS adapters instead. I've updated the configs in this post to show this. Cheers, PM.