Enforce Spring Boot Application to Fail When Cannot Connect to Kafka - spring

I want to make my spring boot application fail on start if it cannot connect to the kafka broker. My application is only publishing messages to topics. I added this line to my properties file but no luck so far spring.kafka.admin.fail-fast=true.

Addition on how to speed up fail-fast
TL;DR Spring Boot 2.4.5 auto configuration does not let you speed up fail-fast using env parameters. Add this to your #Configuration to get 10 seconds timeout:
#Bean
public KafkaAdmin kafkaAdmin(#Autowired KafkaProperties properties) {
KafkaAdmin kafkaAdmin = new KafkaAdmin(properties.buildAdminProperties());
kafkaAdmin.setFatalIfBrokerNotAvailable(properties.getAdmin().isFailFast());
/* speed up fail fast */
kafkaAdmin.setOperationTimeout(5);
kafkaAdmin.setCloseTimeout(5);
return kafkaAdmin;
}
More detailed answer
Fail-fast occurs when initialize() method of class org.springframework.kafka.core.KafkaAdmin is executed. This method may block:
If new topics are found, it blocks on topic creation for up to operationTimeout
If topic creation failed, it blocks on releasing resources for up to closeTimeout
By default those values are 30 and 10 seconds accordingly (hardcoded in the class mentioned above). You can redefine them using set methods: setOperationTimeout(int sec), setCloseTimeout(int sec).
What about Spring Boot? KafkaAdmin bean is created in class org.springframework.boot.autoconfigure.kafka.KafkaAutoConfiguration. As you can guess, Spring Boot simply does not set timeouts:
#Bean
#ConditionalOnMissingBean
public KafkaAdmin kafkaAdmin() {
KafkaAdmin kafkaAdmin = new KafkaAdmin(this.properties.buildAdminProperties());
kafkaAdmin.setFatalIfBrokerNotAvailable(this.properties.getAdmin().isFailFast());
return kafkaAdmin;
}

fail-fast will only work if there is at least one NewTopic bean in the context (so the admin will try to check if the topic exists and create it if not).
#SpringBootApplication
public class So55177700Application {
public static void main(String[] args) {
SpringApplication.run(So55177700Application.class, args);
}
#Bean
public NewTopic topic() {
return new NewTopic("so55177700", 1, (short) 1);
}
}
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2019-03-15 09:42:49.555 ERROR 41793 --- [ main] o.s.boot.SpringApplication : Application run failed
java.lang.IllegalStateException: Could not configure topics

Related

springboot spring datasource tomcat properties not working

I am working on a springboot application with spring jpa with spring starter version <version>2.2.4.RELEASE</version>
I have defined below properties for tomcat and also excluded HikariCP NOTE: HikariCP is also not working
application.properties
spring.datasource.type=org.apache.tomcat.jdbc.pool.DataSource
spring.datasource.tomcat.initial-size=30
spring.datasource.tomcat.max-wait=60000
spring.datasource.tomcat.max-active=300
spring.datasource.tomcat.min-idle=30
spring.datasource.tomcat.default-auto-commit=true
I've tried all combinations and also used default but I am getting below error after 2-3 API calls .
o.h.engine.jdbc.spi.SqlExceptionHelper : [http-nio-8080-exec-5] Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:4; busy:
4; idle:0; lastwait:30000].
The problem is with the deployment. I am deploying the app to cloudfoundry, and it by default adds profile called cloud. So, I created a bean of DataSource for "cloud" profile like below:
#Configuration
#Profile("cloud")
public class CloudConfig extends AbstractCloudConfig {
#Bean
public DataSource dataSource() {
PooledServiceConnectorConfig.PoolConfig poolConfig = new PooledServiceConnectorConfig.PoolConfig(20, 300, 30000);
DataSourceConfig dbConfig = new DataSourceConfig(poolConfig, null);
return connectionFactory().dataSource(dbConfig);
}
}

Spring Boot Kafka with manual ack

Referring to Spring kafka doc, I am trying to implement a spring boot application which listens to a kafka topic and acks manually.
#SpringBootApplication
public class TestApplication {
public static void main(String[] args) {
SpringApplication.run(TestApplication.class, args);
}
#KafkaListener(topics = "sample_log", containerFactory = "kafkaManualAckListenerContainerFactory")
public void listen(ConsumerRecord<?, ?> cr, Acknowledgment ack) throws Exception {
process(cr);
ack.acknowledge();
}
}
But when I ran the application, it shows:
***************************
APPLICATION FAILED TO START
***************************
Description:
A component required a bean named 'kafkaManualAckListenerContainerFactory' that could not be found.
Action:
Consider defining a bean named 'kafkaManualAckListenerContainerFactory' in your configuration.
As far as I'm concerned, Spring Kafka may have defined kafkaManualAckListenerContainerFactory bean since it is a provided feature. How could I fix the error WITHOUT manually define the bean?
You don’t need that containerFactory = "kafkaManualAckListenerContainerFactory" configuration. Just remove it altogether and rely on the provided by Spring Boot auto-configuration for Kafka: https://docs.spring.io/spring-boot/docs/2.0.1.RELEASE/reference/htmlsingle/#boot-features-kafka

Print all the Spring beans that are loaded - Spring Boot

How can I get to know names of all the beans that are loaded as part of my spring boot app? I would like have some code in main method to print the details of beans that are loaded once the server is started up.
As shown in the getting started guide of spring-boot: https://spring.io/guides/gs/spring-boot/
#SpringBootApplication
public class Application {
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
#Bean
public CommandLineRunner commandLineRunner(ApplicationContext ctx) {
return args -> {
System.out.println("Let's inspect the beans provided by Spring Boot:");
String[] beanNames = ctx.getBeanDefinitionNames();
Arrays.sort(beanNames);
for (String beanName : beanNames) {
System.out.println(beanName);
}
};
}
}
As #Velu mentioned in the comments, this will not list manually registered beans.
In case you want to do so, you can use getSingletonNames(). But be careful. This method only returns already instantiated beans. If a bean isn't already instantiated, it will not be returned by getSingletonNames().
May I suggest using Actuator? it provides several endpoints including /beans which lists all beans in the application. You say "once the server is started" so this is an option for web applications.
To set up actuator
https://spring.io/guides/gs/actuator-service/
List of endpoints in actuator
http://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-endpoints.html
Well, Although, this question is already answered, I still want to provide an answer which is a Java 8 variant :)
Arrays.asList(context.getBeanDefinitionNames()).stream().sorted().forEach(System.out::println);
Lets do Java 8 !!!
Actually I would recommend to create this class aside from modifying your #SpringBootApplication.
#Component
public class ContextTeller implements CommandLineRunner {
#Autowired
ApplicationContext applicationContext;
#Override
public void run(String... args) throws Exception {
System.out.println("-------------> just checking!");
System.out.println(Arrays.asList(applicationContext.getBeanDefinitionNames()));
}}
This way Spring Boot will load this class and execute just after loading context. Then you just can remove the file, and everything is clear.
applicationContext.getBeanDefinitionNames() does not show the beans which are registered without BeanDefinition instance.
For spring boot web applications, all the beans can be listed using the below endpoint.
#RestController
#RequestMapping("/list")
class ExportController {
#Autowired
private ApplicationContext applicationContext;
#GetMapping("/beans")
#ResponseStatus(value = HttpStatus.OK)
String[] registeredBeans() {
return printBeans();
}
private String[] printBeans() {
AutowireCapableBeanFactory autowireCapableBeanFactory = applicationContext.getAutowireCapableBeanFactory();
if (autowireCapableBeanFactory instanceof SingletonBeanRegistry) {
String[] singletonNames = ((SingletonBeanRegistry) autowireCapableBeanFactory).getSingletonNames();
for (String singleton : singletonNames) {
System.out.println(singleton);
}
return singletonNames;
}
return null;
}
}
[
"autoConfigurationReport",
"springApplicationArguments",
"springBootBanner",
"springBootLoggingSystem",
"environment",
"systemProperties",
"systemEnvironment",
"org.springframework.context.annotation.internalConfigurationAnnotationProcessor",
"org.springframework.boot.autoconfigure.internalCachingMetadataReaderFactory",
"org.springframework.boot.autoconfigure.condition.BeanTypeRegistry",
"org.springframework.context.annotation.ConfigurationClassPostProcessor.importRegistry",
"propertySourcesPlaceholderConfigurer",
"org.springframework.boot.context.properties.ConfigurationPropertiesBindingPostProcessor.store",
"preserveErrorControllerTargetClassPostProcessor",
"org.springframework.context.annotation.internalAutowiredAnnotationProcessor",
"org.springframework.context.annotation.internalRequiredAnnotationProcessor",
"org.springframework.context.annotation.internalCommonAnnotationProcessor",
"org.springframework.boot.context.properties.ConfigurationPropertiesBindingPostProcessor",
"org.springframework.scheduling.annotation.ProxyAsyncConfiguration",
"org.springframework.context.annotation.internalAsyncAnnotationProcessor",
"methodValidationPostProcessor",
"embeddedServletContainerCustomizerBeanPostProcessor",
"errorPageRegistrarBeanPostProcessor",
"messageSource",
"applicationEventMulticaster",
"org.springframework.boot.autoconfigure.web.EmbeddedServletContainerAutoConfiguration$EmbeddedTomcat",
"tomcatEmbeddedServletContainerFactory",
"org.springframework.boot.autoconfigure.websocket.WebSocketAutoConfiguration$TomcatWebSocketConfiguration",
"websocketContainerCustomizer",
"spring.http.encoding-org.springframework.boot.autoconfigure.web.HttpEncodingProperties",
"org.springframework.boot.autoconfigure.web.HttpEncodingAutoConfiguration",
"localeCharsetMappingsCustomizer",
"org.springframework.boot.autoconfigure.web.ServerPropertiesAutoConfiguration",
"serverProperties",
"duplicateServerPropertiesDetector",
"spring.resources-org.springframework.boot.autoconfigure.web.ResourceProperties",
"org.springframework.boot.autoconfigure.web.ErrorMvcAutoConfiguration$DefaultErrorViewResolverConfiguration",
"conventionErrorViewResolver",
"org.springframework.boot.autoconfigure.web.ErrorMvcAutoConfiguration",
"errorPageCustomizer",
"servletContext",
"contextParameters",
"contextAttributes",
"spring.mvc-org.springframework.boot.autoconfigure.web.WebMvcProperties",
"spring.http.multipart-org.springframework.boot.autoconfigure.web.MultipartProperties",
"org.springframework.boot.autoconfigure.web.MultipartAutoConfiguration",
"multipartConfigElement",
"org.springframework.boot.autoconfigure.web.DispatcherServletAutoConfiguration$DispatcherServletRegistrationConfiguration",
"org.springframework.boot.autoconfigure.web.DispatcherServletAutoConfiguration$DispatcherServletConfiguration",
"dispatcherServlet",
"dispatcherServletRegistration",
"requestContextFilter",
"org.springframework.boot.autoconfigure.web.WebMvcAutoConfiguration",
"hiddenHttpMethodFilter",
"httpPutFormContentFilter",
"characterEncodingFilter",
"org.springframework.context.event.internalEventListenerProcessor",
"org.springframework.context.event.internalEventListenerFactory",
"reportGeneratorApplication",
"exportController",
"exportService",
"org.springframework.boot.autoconfigure.AutoConfigurationPackages",
"org.springframework.boot.autoconfigure.context.PropertyPlaceholderAutoConfiguration",
"org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$Jackson2ObjectMapperBuilderCustomizerConfiguration",
"spring.jackson-org.springframework.boot.autoconfigure.jackson.JacksonProperties",
"standardJacksonObjectMapperBuilderCustomizer",
"org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$JacksonObjectMapperBuilderConfiguration",
"org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration",
"jsonComponentModule",
"jacksonObjectMapperBuilder",
"org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$JacksonObjectMapperConfiguration",
"jacksonObjectMapper",
"org.springframework.boot.autoconfigure.websocket.WebSocketAutoConfiguration",
"org.springframework.boot.autoconfigure.web.EmbeddedServletContainerAutoConfiguration",
"org.springframework.boot.autoconfigure.web.DispatcherServletAutoConfiguration",
"org.springframework.boot.autoconfigure.validation.ValidationAutoConfiguration",
"defaultValidator",
"org.springframework.boot.autoconfigure.web.ErrorMvcAutoConfiguration$WhitelabelErrorViewConfiguration",
"error",
"beanNameViewResolver",
"errorAttributes",
"basicErrorController",
"org.springframework.boot.autoconfigure.web.WebMvcAutoConfiguration$EnableWebMvcConfiguration",
"org.springframework.boot.autoconfigure.web.WebMvcAutoConfiguration$WebMvcAutoConfigurationAdapter",
"mvcContentNegotiationManager",
"org.springframework.boot.autoconfigure.web.HttpMessageConvertersAutoConfiguration$StringHttpMessageConverterConfiguration",
"stringHttpMessageConverter",
"org.springframework.boot.autoconfigure.web.JacksonHttpMessageConvertersConfiguration$MappingJackson2HttpMessageConverterConfiguration",
"mappingJackson2HttpMessageConverter",
"org.springframework.boot.autoconfigure.web.HttpMessageConvertersAutoConfiguration",
"messageConverters",
"mvcConversionService",
"mvcValidator",
"requestMappingHandlerAdapter",
"mvcResourceUrlProvider",
"requestMappingHandlerMapping",
"mvcPathMatcher",
"mvcUrlPathHelper",
"viewControllerHandlerMapping",
"beanNameHandlerMapping",
"resourceHandlerMapping",
"defaultServletHandlerMapping",
"mvcUriComponentsContributor",
"httpRequestHandlerAdapter",
"simpleControllerHandlerAdapter",
"handlerExceptionResolver",
"mvcViewResolver",
"org.springframework.boot.autoconfigure.web.WebMvcAutoConfiguration$WebMvcAutoConfigurationAdapter$FaviconConfiguration",
"faviconRequestHandler",
"faviconHandlerMapping",
"defaultViewResolver",
"viewResolver",
"welcomePageHandlerMapping",
"org.springframework.boot.autoconfigure.jmx.JmxAutoConfiguration",
"objectNamingStrategy",
"mbeanServer",
"mbeanExporter",
"org.springframework.boot.autoconfigure.admin.SpringApplicationAdminJmxAutoConfiguration",
"springApplicationAdminRegistrar",
"org.springframework.boot.autoconfigure.context.ConfigurationPropertiesAutoConfiguration",
"org.springframework.boot.autoconfigure.web.JacksonHttpMessageConvertersConfiguration",
"spring.info-org.springframework.boot.autoconfigure.info.ProjectInfoProperties",
"org.springframework.boot.autoconfigure.info.ProjectInfoAutoConfiguration",
"multipartResolver",
"org.springframework.boot.autoconfigure.web.WebClientAutoConfiguration$RestTemplateConfiguration",
"restTemplateBuilder",
"org.springframework.boot.autoconfigure.web.WebClientAutoConfiguration",
"spring.devtools-org.springframework.boot.devtools.autoconfigure.DevToolsProperties",
"org.springframework.boot.devtools.autoconfigure.LocalDevToolsAutoConfiguration$RestartConfiguration",
"fileSystemWatcherFactory",
"classPathRestartStrategy",
"classPathFileSystemWatcher",
"hateoasObjenesisCacheDisabler",
"org.springframework.boot.devtools.autoconfigure.LocalDevToolsAutoConfiguration$LiveReloadConfiguration$LiveReloadServerConfiguration",
"org.springframework.boot.devtools.autoconfigure.LocalDevToolsAutoConfiguration$LiveReloadConfiguration",
"optionalLiveReloadServer",
"org.springframework.boot.devtools.autoconfigure.LocalDevToolsAutoConfiguration",
"lifecycleProcessor"
]
As you can see in the output, environment, systemProperties, systemEnvironment beans will not be shown using context.getBeanDefinitionNames() method.
#Component
public class ContextTeller implements CommandLineRunner {
#Autowired
public ApplicationContext applicationContext;
#Override
public void run(String... args) throws Exception {
System.out.println("<------------- Beans loaded --------------->");
Arrays.asList(applicationContext.getBeanDefinitionNames()).stream().forEach(System.out::println);
}
}
As mentioned by #Zergleb, using Actuator is appropriate too, however, based on the reference documentation this endpoint is no longer exposed via "web" by default. Thus you need to consider the following steps to access the endpoint
Add the below dependency to your pom.xml
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
Add these two propertiers to your application.properties file:
management.endpoints.web.exposure.include=beans
management.endpoint.beans.enabled=true
Access to your application context beans using /actuator/beans http endpoint
I did one small experiment for this requirement and found this solution. I have created SpringBoot while selecting the modules like WEB, Actuator, HAL and Devtools. I have configured the below property in application properties to load all the endpoints present in actuator.
management.endpoints.web.exposure.include=*
You can see actuator information in, http://localhost:8080/actuator. This will show all the application information along with actuator, status, info,etc.. In that, you can find the http://localhost:8080/actuator/beans which will load all the beans internally created by springboot application.
Once you are able to see all the beans information, I think it's not necessary to print again in main class.
Since, I have already configured rest-hal-browser dependency in my application, when I load the URL of http://localhost:8080, will load the UI to search for different endpoints. In the below image I am searching for actuator information.
Simple way to get all beans
ApplicationContext ctx =SpringApplication.run(DemoApplication.class, args);
String [] beans = ctx.getBeanDefinitionNames();
Arrays.sort(beans);
System.out.println("#############");
for(String s:beans) {
System.out.println(s + " of type " + ctx.getBean(s).getClass());
}

Ensuring Spring Integration deployment's JMS listener threads are cleaned up on Tomcat undeploy

I have a simple Spring Integration application which runs on Tomcat (v7.0.x) and consumes messages off a Websphere MQ Queue. When I un-deploy the WAR from the Tomcat server, the WAR un-deploys okay but, a JMS listener thread is left running on the Tomcat server which will still consume messages off the Websphere MQ Queue. I am therefore assuming that I am not handling the JMS listener clean up part of the application properly?
Here is the stack I am using:
Java 8
Tomcat 7.0.55
Spring Integration 4.0.4
Spring Integration Java Dsl 1.0.0.M3
In terms of my SI application's configurations, I have a JmsConfig class:
#Configuration
#ComponentScan
public class JmsConfig {
#Autowired
private Properties jndiProperties;
private ConnectionFactory mqConnectionFactory() throws NamingException {
Context ctx = new InitialContext(jndiProperties);
try {
MQQueueConnectionFactory connectionFactory = (MQQueueConnectionFactory)
ctx.lookup("jms/service/SERVICE_QCF");
return connectionFactory;
} finally {
ctx.close();
}
}
#Bean
public ConnectionFactory cachingConnectionFactory() throws NamingException {
CachingConnectionFactory connectionFactory = new CachingConnectionFactory();
connectionFactory.setTargetConnectionFactory(mqConnectionFactory());
connectionFactory.setSessionCacheSize(10);
return connectionFactory;
}
}
I have an Integration config class:
#Configuration
#EnableIntegration
public class IntegrationConfig {
#Autowired
private ConnectionFactory cachingConnectionFactory;
#Bean
public IntegrationFlow requestFlow() {
return IntegrationFlows
.from(Jms.inboundAdapter(cachingConnectionFactory).destination(
"SERVICE_QUEUE_NAME"), c -> {
c.poller(Pollers.fixedRate(100));
})
.channel("request.service.ch").get();
}
}
Web Initialiser config class:
#Configuration
public class WebInitialiser implements WebApplicationInitializer {
public void onStartup(ServletContext servletContext)
throws ServletException {
AnnotationConfigWebApplicationContext rootContext =
new AnnotationConfigWebApplicationContext();
rootContext.register(ApplicationConfig.class, JmsConfig.class,
IntegrationConfig.class, DatabaseConfig.class);
servletContext.addListener(new ContextLoaderListener(rootContext));
}
}
During the un-deploy stage I see the following in the catalina logs which may or may not be related:
SEVERE: The web application [/service-a] appears to have started a thread named [Thread-7] but has failed to stop it. This is very likely to create a memory leak.
Is there anything that I have yet NOT set or configured or annotated in order to ensure that the deployment's JMS listener thread is cleaned up from Tomcat's JVM during the WAR's un-deploy stage?
Thanks in advance,
PM.
To ensure that JMS listener threads are cleared up upon the application's un-deploy stage, I simply created a CachingConnectionFactory bean with its targetConnectionFactory being that of the MQConnectionFactory. Then, in the Spring Integration flows, I simply pass in the cachingConnectionFactory bean to the JMS adapters instead. I've updated the configs in this post to show this. Cheers, PM.

Using Quartz with Spring Boot - injection order changes based upon return type of method

I am trying to get Quartz working with Spring Boot, and am not managing to get the injection working correctly. I am basing myself on the example shown here
Here is my boot class:
#ComponentScan
#EnableAutoConfiguration
public class MyApp {
#Autowired
private DataSource dataSource;
#Bean
public JobFactory jobFactory() {
return new SpringBeanJobFactory();
}
#Bean
public SchedulerFactoryBean quartz() {
final SchedulerFactoryBean bean = new SchedulerFactoryBean();
bean.setJobFactory(jobFactory());
bean.setDataSource(dataSource);
bean.setConfigLocation(new ClassPathResource("quartz.properties"));
...
return bean;
}
public static void main(String[] args) {
SpringApplication.run(MyApp.class, args);
}
}
When the quartz() method is invoked by Spring, dataSource is null. However, if I change the return type of the quartz() method to Object, dataSource is correctly injected with the datasource created by reading application.properties, the bean is built, everything works and I get a subsequent error saying that Quartz has been unable to retrieve any jobs from the database, which is normal as I haven't put the schema in place yet.
I have tried adding a #DependsOn("dataSource") annotation on the quartz() method but that doesn't make any difference.
This class is the only class annotated with #Configuration.
Here are my dependencies (I'm using Maven but present them like this for space reasons):
org.springframework.boot:spring-boot-starter-actuator:1.0.0.RC4
org.springframework.boot:spring-boot-starter-jdbc:1.0.0.RC4
org.springframework.boot:spring-boot-starter-web:1.0.0.RC4
org.quartz-scheduler:quartz:2.2.1
org.springframework:spring-support:2.0.8
And the parent:
org.springframework.boot:spring-boot-starter-parent:1.0.0.RC4
Finally the content of quartz.properties:
org.quartz.threadPool.threadCount = 3
org.quartz.jobStore.class=org.springframework.scheduling.quartz.LocalDataSourceJobStore
org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.PostgreSQLDelegate
What am I doing wrong?
(I have seen this question, but that question initialises the datasource in the #Configuration class)
Your app starts up (with a schema error, which is expected) if I use "org.springframework:spring-context-support:4.0.2.RELEASE" ("org.springframework:spring-support:2.0.8" if it ever existed must be nearly 10 years old now and certainly isn't compatible with Boot or Quartz 2).

Resources