Hikari After adding vs Pool Stats - spring-boot

I have the spring boot application with HikariCP Pool. In one of my deployments, I found Hikari Stats like the following:
2022-10-17 11:33:41.532 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=4)
2022-10-17 11:34:11.532 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=2)
2022-10-17 11:34:41.533 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=5)
2022-10-17 11:35:11.533 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=1)
2022-10-17 11:35:41.534 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=2)
2022-10-17 11:36:11.534 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=2)
2022-10-17 11:36:41.535 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=3)
2022-10-17 11:37:11.535 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=1)
2022-10-17 11:37:41.536 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=1)
2022-10-17 11:38:11.536 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=1)
2022-10-17 11:38:20.766 DEBUG 8 [onnection adder] c.z.h.p.HikariPool : DBPool - After adding stats (total=10, active=4, idle=6, waiting=0)
2022-10-17 11:38:41.537 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=2)
2022-10-17 11:39:11.537 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=1)
2022-10-17 11:39:39.507 DEBUG 8 [onnection adder] c.z.h.p.HikariPool : DBPool - After adding stats (total=10, active=7, idle=3, waiting=0)
2022-10-17 11:39:41.538 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=2)
2022-10-17 11:40:11.538 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=1)
2022-10-17 11:40:41.539 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=2)
2022-10-17 11:41:11.540 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=1)
2022-10-17 11:41:41.540 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=10)
2022-10-17 11:42:11.541 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=1)
2022-10-17 11:42:41.541 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=1)
2022-10-17 11:43:11.542 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=5)
2022-10-17 11:43:41.542 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=2)
2022-10-17 11:44:11.543 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=1)
2022-10-17 11:44:41.545 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=2)
2022-10-17 11:45:11.545 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=2)
2022-10-17 11:45:20.243 DEBUG 8 [onnection adder] c.z.h.p.HikariPool : DBPool - After adding stats (total=10, active=0, idle=10, waiting=0)
2022-10-17 11:45:41.546 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=2)
2022-10-17 11:46:11.546 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=1)
2022-10-17 11:46:41.547 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=1)
2022-10-17 11:47:11.547 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=1)
2022-10-17 11:47:41.548 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=2)
2022-10-17 11:48:11.548 DEBUG 8 [ool housekeeper] c.z.h.p.HikariPool : DBPool - Pool stats (total=10, active=10, idle=0, waiting=3)
Here Pool Stats is always showing active=10 and after adding stats has varying active connections. I have kept the maximum pool size at 10. My application keeps running without any issues for now.
Questions:
What is the difference between "Pool Stats" and "After Adding Stats"?
What could be the reason that the pool stats active is always 10 and the other one is changing?

Related

HikariPool-1 - Exception during pool initialization with SpringBoot and Hibernate and JPA Repository

HikariPool-1 - Exception during pool initialization when trying to run spring-boot application. While running mvn spring-boot:run command, the logs show repositories initialized and server starts and is in running state but throws exception during hikari pool initialization!
Able to connect to the DB using pgAdmin but not able to connect from the application. The code from application.properties is below.
spring.servlet.multipart.max-file-size=10MB
spring.servlet.multipart.max-request-size=10MB
spring.datasource.url= jdbc:postgresql:/34.93.135.89:5433/mtt_04_dec_22
spring.datasource.username= postgres
spring.datasource.password= justdoit#mtt
spring.jpa.hibernate.ddl-auto= none
spring.datasource.testWhileIdle=true
spring.datasource.test-on-borrow=true
spring.datasource.hikari.connectionTimeout=30000
spring.datasource.hikari.maxLifetime=60000
spring.datasource.hikari.maximum-pool-size=30
spring.jpa.open-in-view=false
And Hikari debug logs
PersistenceUnitInfo [name: default]
2022-12-02 11:30:24.569 INFO 14176 --- [ task-1] org.hibernate.Version : HHH000412: Hibernate ORM core version 5.4.15.Final
2022-12-02 11:30:24.791 INFO 14176 --- [ task-1] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.1.0.Final}
2022-12-02 11:30:24.947 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : HikariPool-1 - configuration:
2022-12-02 11:30:24.950 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : allowPoolSuspension.............false
2022-12-02 11:30:24.950 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : autoCommit......................true
2022-12-02 11:30:24.951 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : catalog.........................none
2022-12-02 11:30:24.951 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : connectionInitSql...............none
2022-12-02 11:30:24.951 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : connectionTestQuery.............none
2022-12-02 11:30:24.952 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : connectionTimeout...............30000
2022-12-02 11:30:24.952 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : dataSource......................none
2022-12-02 11:30:24.952 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : dataSourceClassName.............none
2022-12-02 11:30:24.952 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : dataSourceJNDI..................none
2022-12-02 11:30:24.953 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : dataSourceProperties............{password=<masked>}
2022-12-02 11:30:24.953 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : driverClassName................."org.postgresql.Driver"
2022-12-02 11:30:24.953 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : exceptionOverrideClassName......none
2022-12-02 11:30:24.954 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : healthCheckProperties...........{}
2022-12-02 11:30:24.954 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : healthCheckRegistry.............none
2022-12-02 11:30:24.954 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : idleTimeout.....................600000
2022-12-02 11:30:24.954 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : initializationFailTimeout.......1
2022-12-02 11:30:24.955 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : isolateInternalQueries..........false
2022-12-02 11:30:24.955 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : jdbcUrl.........................jdbc:postgresql:/35.200.211.39:5432/mtt_04_dec_22
2022-12-02 11:30:24.955 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : leakDetectionThreshold..........0
2022-12-02 11:30:24.955 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : maxLifetime.....................30000
2022-12-02 11:30:24.956 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : maximumPoolSize.................30
2022-12-02 11:30:24.956 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : metricRegistry..................none
2022-12-02 11:30:24.956 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : metricsTrackerFactory...........none
2022-12-02 11:30:24.956 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : minimumIdle.....................30
2022-12-02 11:30:24.956 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : password........................<masked>
2022-12-02 11:30:24.957 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : poolName........................"HikariPool-1"
2022-12-02 11:30:24.957 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : readOnly........................false
2022-12-02 11:30:24.957 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : registerMbeans..................false
2022-12-02 11:30:24.957 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : scheduledExecutor...............none
2022-12-02 11:30:24.957 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : schema..........................none
2022-12-02 11:30:24.958 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : threadFactory...................internal
2022-12-02 11:30:24.958 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : transactionIsolation............default
2022-12-02 11:30:24.958 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : username........................"postgres"
2022-12-02 11:30:24.959 DEBUG 14176 --- [ task-1] com.zaxxer.hikari.HikariConfig : validationTimeout...............5000
EXCEPTION ON CONSOLE LOOKS LIKE:
HikariPool-1 - Exception during pool initialization.
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.3.0.RELEASE)
2022-12-02 09:26:51.258 INFO 14888 --- [ main] org.sadisha.backend.service.Main : Starting Main on DESKTOP-991KNCS with PID 14888 (C:\mttbe\mtt-be\target\classes started by Admin in C:\mttbe\mtt-be)
2022-12-02 09:26:51.265 INFO 14888 --- [ main] org.sadisha.backend.service.Main : No active profile set, falling back to default profiles: default
2022-12-02 09:26:52.481 INFO 14888 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFERRED mode.
2022-12-02 09:26:52.715 INFO 14888 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 220ms. Found 11 JPA repository interfaces.
2022-12-02 09:26:53.776 INFO 14888 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2022-12-02 09:26:53.793 INFO 14888 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2022-12-02 09:26:53.794 INFO 14888 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.35]
2022-12-02 09:26:54.042 INFO 14888 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2022-12-02 09:26:54.043 INFO 14888 --- [ main] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 2655 ms
2022-12-02 09:26:54.178 INFO 14888 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
2022-12-02 09:26:57.285 ERROR 14888 --- [ main] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Exception during pool initialization.
org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:303) ~[postgresql-42.2.18.jar:42.2.18]
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51) ~[postgresql-42.2.18.jar:42.2.18]
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:225) ~[postgresql-42.2.18.jar:42.2.18]
at org.postgresql.Driver.makeConnection(Driver.java:465) ~[postgresql-42.2.18.jar:42.2.18]
at org.postgresql.Driver.connect(Driver.java:264) ~[postgresql-42.2.18.jar:42.2.18]
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138) ~[HikariCP-3.4.5.jar:na]
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:358) ~[HikariCP-3.4.5.jar:na]
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206) ~[HikariCP-3.4.5.jar:na]
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:477) ~[HikariCP-3.4.5.jar:na]
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:560) ~[HikariCP-3.4.5.jar:na]
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115) ~[HikariCP-3.4.5.jar:na]
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112) ~[HikariCP-3.4.5.jar:na]
at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:158) ~[spring-jdbc-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:116) ~[spring-jdbc-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79) ~[spring-jdbc-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:324) ~[spring-jdbc-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.boot.jdbc.EmbeddedDatabaseConnection.isEmbedded(EmbeddedDatabaseConnection.java:120) ~[spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
at org.springframework.boot.autoconfigure.jdbc.DataSourceInitializer.isEmbedded(DataSourceInitializer.java:137) ~[spring-boot-autoconfigure-2.3.0.RELEASE.jar:2.3.0.RELEASE]
at org.springframework.boot.autoconfigure.jdbc.DataSourceInitializer.isEnabled(DataSourceInitializer.java:129) ~[spring-boot-autoconfigure-2.3.0.RELEASE.jar:2.3.0.RELEASE]
at org.springframework.boot.autoconfigure.jdbc.DataSourceInitializer.createSchema(DataSourceInitializer.java:96) ~[spring-boot-autoconfigure-2.3.0.RELEASE.jar:2.3.0.RELEASE]

in your application.properties
change this :
spring.datasource.url=jdbc:postgresql:/34.93.135.89:5433/mtt_04_dec_22
postgresql defaulty adress is 5432 not 5433. probably this is the reason of error
here is you wrote: 34.93.135.89:5433

Spring boot docker running on 8080 only?

I have deployed my spring boot application on docker but when I am trying to run it, it is only running on 8080 port not any other port i.e 3000 from logs I am able to deduce that it has something to do with internal tomcat listens to 8080 port. can anyone please tell me what is happening inside?
below are the logs when I am running on 8080
2021-02-14 05:53:36.156 INFO 1 --- [ main] com.ebi.uk.EbiProjectJavaApplication : Starting EbiProjectJavaApplication v0.0.1-SNAPSHOT using Java 1.8.0_212 on 8a710deb6884 with PID 1 (/ebiProjectJava.jar started by root in /)
2021-02-14 05:53:36.160 INFO 1 --- [ main] com.ebi.uk.EbiProjectJavaApplication : No active profile set, falling back to default profiles: default
2021-02-14 05:53:37.426 INFO 1 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
2021-02-14 05:53:37.663 INFO 1 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 224 ms. Found 1 JPA repository interfaces.
2021-02-14 05:53:38.750 INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2021-02-14 05:53:38.770 INFO 1 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2021-02-14 05:53:38.771 INFO 1 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.41]
2021-02-14 05:53:38.902 INFO 1 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2021-02-14 05:53:38.903 INFO 1 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2621 ms
2021-02-14 05:53:39.139 INFO 1 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
2021-02-14 05:53:39.648 INFO 1 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
2021-02-14 05:53:39.732 INFO 1 --- [ main] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default]
2021-02-14 05:53:39.817 INFO 1 --- [ main] org.hibernate.Version : HHH000412: Hibernate ORM core version 5.4.27.Final
2021-02-14 05:53:39.990 INFO 1 --- [ main] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.1.2.Final}
2021-02-14 05:53:40.149 INFO 1 --- [ main] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.H2Dialect
2021-02-14 05:53:40.898 INFO 1 --- [ main] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform]
2021-02-14 05:53:40.909 INFO 1 --- [ main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'
2021-02-14 05:53:41.299 WARN 1 --- [ main] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning
2021-02-14 05:53:41.595 INFO 1 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'applicationTaskExecutor'
2021-02-14 05:53:42.091 INFO 1 --- [ main] o.s.s.web.DefaultSecurityFilterChain : Will secure any request with [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter#7b94089b, org.springframework.security.web.context.SecurityContextPersistenceFilter#ee86bcb, org.springframework.security.web.header.HeaderWriterFilter#1f010bf0, org.springframework.security.web.authentication.logout.LogoutFilter#bcef303, org.springframework.security.web.authentication.www.BasicAuthenticationFilter#33308786, org.springframework.security.web.savedrequest.RequestCacheAwareFilter#7f132176, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter#6ed3f258, org.springframework.security.web.authentication.AnonymousAuthenticationFilter#7ca20101, org.springframework.security.web.session.SessionManagementFilter#10cf09e8, org.springframework.security.web.access.ExceptionTranslationFilter#19b93fa8, org.springframework.security.web.access.intercept.FilterSecurityInterceptor#71812481]
2021-02-14 05:53:42.410 INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2021-02-14 05:53:42.424 INFO 1 --- [ main] com.ebi.uk.EbiProjectJavaApplication : Started EbiProjectJavaApplication in 6.864 seconds (JVM running for 7.751)
2021-02-14 05:54:13.163 INFO 1 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring DispatcherServlet 'dispatcherServlet'
2021-02-14 05:54:13.164 INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : Initializing Servlet 'dispatcherServlet'
2021-02-14 05:54:13.167 INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : Completed initialization in 3 ms
below are the logs when I am trying to run on 3000
2021-02-14 05:50:58.993 INFO 1 --- [ main] com.ebi.uk.EbiProjectJavaApplication : Starting EbiProjectJavaApplication v0.0.1-SNAPSHOT using Java 1.8.0_212 on b0f62c63fde2 with PID 1 (/ebiProjectJava.jar started by root in /)
2021-02-14 05:50:58.997 INFO 1 --- [ main] com.ebi.uk.EbiProjectJavaApplication : No active profile set, falling back to default profiles: default
2021-02-14 05:51:00.304 INFO 1 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
2021-02-14 05:51:00.566 INFO 1 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 243 ms. Found 1 JPA repository interfaces.
2021-02-14 05:51:01.714 INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2021-02-14 05:51:01.734 INFO 1 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2021-02-14 05:51:01.734 INFO 1 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.41]
2021-02-14 05:51:01.816 INFO 1 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2021-02-14 05:51:01.817 INFO 1 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2706 ms
2021-02-14 05:51:02.060 INFO 1 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
2021-02-14 05:51:02.534 INFO 1 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
2021-02-14 05:51:02.632 INFO 1 --- [ main] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default]
2021-02-14 05:51:02.723 INFO 1 --- [ main] org.hibernate.Version : HHH000412: Hibernate ORM core version 5.4.27.Final
2021-02-14 05:51:02.909 INFO 1 --- [ main] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.1.2.Final}
2021-02-14 05:51:03.163 INFO 1 --- [ main] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.H2Dialect
2021-02-14 05:51:04.045 INFO 1 --- [ main] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform]
2021-02-14 05:51:04.055 INFO 1 --- [ main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'
2021-02-14 05:51:04.480 WARN 1 --- [ main] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning
2021-02-14 05:51:04.799 INFO 1 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'applicationTaskExecutor'
2021-02-14 05:51:05.330 INFO 1 --- [ main] o.s.s.web.DefaultSecurityFilterChain : Will secure any request with [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter#7ca20101, org.springframework.security.web.context.SecurityContextPersistenceFilter#177bea38, org.springframework.security.web.header.HeaderWriterFilter#40db2a24, org.springframework.security.web.authentication.logout.LogoutFilter#41709512, org.springframework.security.web.authentication.www.BasicAuthenticationFilter#4f9a2c08, org.springframework.security.web.savedrequest.RequestCacheAwareFilter#6bca7e0d, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter#8ad6665, org.springframework.security.web.authentication.AnonymousAuthenticationFilter#47f9738, org.springframework.security.web.session.SessionManagementFilter#1921ad94, org.springframework.security.web.access.ExceptionTranslationFilter#7e6ef134, org.springframework.security.web.access.intercept.FilterSecurityInterceptor#16ce702d]
2021-02-14 05:51:05.709 INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2021-02-14 05:51:05.726 INFO 1 --- [ main] com.ebi.uk.EbiProjectJavaApplication : Started EbiProjectJavaApplication in 7.436 seconds (JVM running for 8.28)
below is my pom.xml
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-webflux</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<!-- <dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<scope>runtime</scope>
</dependency> -->
<!-- <dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
<scope>provided</scope>
</dependency> -->
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
</dependency>
<!-- spring security test -->
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>junit-jupiter</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>mysql</artifactId>
<scope>test</scope>
</dependency>
below is the command I am trying to run
docker run -p 3000:3000 ebiproject

I think you are missing important concept here. 8080 is default port for spring boot application. But, in your docker network configuration you mapped 3000 -> 3000 which is wrong.
Instead, you should configure like below:
docker run -p 3000:8080 ebiproject

Spring boot (2.2.4.RELEASE) on docker hangs on boot unless network_mode: host

I have a dockerized springboot application, if I run the image on my machine (ubuntu) all works fine with docker network mode default, as soon I run the image on an enterprise server (VPS) with the default, it hangs on startup and stays there forever.
2020-03-24 08:26:47.590 INFO 1 --- [ main] org.hibernate.Version : HHH000412: Hibernate Core {5.4.10.Final}
2020-03-24 08:26:47.821 INFO 1 --- [ main] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.1.0.Final}
2020-03-24 08:26:48.015 INFO 1 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
2020-03-24 08:26:48.125 INFO 1 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
2020-03-24 08:26:48.151 INFO 1 --- [ main] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.PostgreSQL95Dialect
or in debug
2020-03-24 08:19:43.856 DEBUG 1 --- [onnection adder] o.p.core.v3.ConnectionFactoryImpl : Send Buffer Size is 43,520
2020-03-24 08:19:43.935 DEBUG 1 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection#7b248da4
2020-03-24 08:19:43.935 DEBUG 1 --- [onnection adder] org.postgresql.Driver : Connecting with URL: jdbc:postgresql://157.46.186.128:62013/db
2020-03-24 08:19:43.935 DEBUG 1 --- [onnection adder] org.postgresql.jdbc.PgConnection : PostgreSQL JDBC Driver 42.2.9
2020-03-24 08:19:43.935 DEBUG 1 --- [onnection adder] org.postgresql.jdbc.PgConnection : setDefaultFetchSize = 0
2020-03-24 08:19:43.935 DEBUG 1 --- [onnection adder] org.postgresql.jdbc.PgConnection : setPrepareThreshold = 5
2020-03-24 08:19:43.935 DEBUG 1 --- [onnection adder] o.p.core.v3.ConnectionFactoryImpl : Trying to establish a protocol version 3 connection to 160.46.186.128:62013
2020-03-24 08:19:43.936 DEBUG 1 --- [onnection adder] o.p.core.v3.ConnectionFactoryImpl : Receive Buffer Size is 186,240
2020-03-24 08:19:43.937 DEBUG 1 --- [onnection adder] o.p.core.v3.ConnectionFactoryImpl : Send Buffer Size is 43,520
2020-03-24 08:19:44.007 DEBUG 1 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection#205272d5
2020-03-24 08:19:44.008 DEBUG 1 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - After adding stats (total=10, active=1, idle=9, waiting=0)
2020-03-24 08:20:10.592 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
2020-03-24 08:20:39.272 DEBUG 1 --- [alina-utility-2] org.apache.catalina.session.ManagerBase : Start expire sessions StandardManager at 1585038039271 sessioncount 0
2020-03-24 08:20:39.272 DEBUG 1 --- [alina-utility-2] org.apache.catalina.session.ManagerBase : End expire sessions StandardManager processingTime 1 expired sessions: 0
2020-03-24 08:20:40.592 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
2020-03-24 08:21:10.593 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
2020-03-24 08:21:39.274 DEBUG 1 --- [alina-utility-2] org.apache.catalina.session.ManagerBase : Start expire sessions StandardManager at 1585038099274 sessioncount 0
2020-03-24 08:21:39.274 DEBUG 1 --- [alina-utility-2] org.apache.catalina.session.ManagerBase : End expire sessions StandardManager processingTime 0 expired sessions: 0
2020-03-24 08:21:40.593 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
2020-03-24 08:22:10.594 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
2020-03-24 08:22:39.277 DEBUG 1 --- [alina-utility-1] org.apache.catalina.session.ManagerBase : Start expire sessions StandardManager at 1585038159277 sessioncount 0
2020-03-24 08:22:39.277 DEBUG 1 --- [alina-utility-1] org.apache.catalina.session.ManagerBase : End expire sessions StandardManager processingTime 0 expired sessions: 0
2020-03-24 08:22:40.594 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
2020-03-24 08:23:10.595 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
I have tried to configure my custom network on docker level, use ip, hostnames and no luck.
has anyone experienced same behavior?
Spring Config:
spring:
flyway:
url: jdbc:postgresql://157.46.186.128:62013/db
user: user
password: xxxxx
locations: classpath:db/migration
schemas: db1
datasource:
db:
jdbcUrl: "jdbc:postgresql://157.46.186.128:62013/db"
username: "user"
password: "xxxxx"
driver-class-name: "org.postgresql.Driver"
dialect: "org.hibernate.dialect.PostgreSQL95Dialect"
ci:
jdbcUrl: "jdbc:postgresql://157.46.186.128:62013/db"
username: "user"
password: "xxxxx"
driver-class-name: "org.postgresql.Driver"
dialect: "org.hibernate.dialect.PostgreSQL95Dialect"
schema: ci
docker network settings:
with host mode
"Bridge": "",
"SandboxID": "49fa5d5e812c816a84efe92156e825151b2dc75a36bb68c399b943e06c26a6f7",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {},
"SandboxKey": "/var/run/docker/netns/default",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"host": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "d6ecd673f06e15560a16dc5bb38ea4be99e751bfb729d819ab76c50320836443",
"EndpointID": "087b1a6fed2cd5b4de46c6700a9ef7e79de6fc3e58a0ecfdd74a8a851429eaa6",
"Gateway": "",
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "",
"DriverOpts": null
}
}
}
and without
"NetworkSettings": {
"Bridge": "",
"SandboxID": "cf8d68a1b1c61d454ed45eb48f4007591d07414f2147410c1381d60e12b93445",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"7979/tcp": null,
"8080/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "8080"
}
]
},
"SandboxKey": "/var/run/docker/netns/cf8d68a1b1c6",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "49993412f56f7bd29edd55091bd77bba66a40ae566daf8111f891be9aa21d4ce",
"Gateway": "172.17.0.1",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "172.17.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"MacAddress": "02:42:ac:11:00:02",
"Networks": {
"bridge": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "bf5b2dfe32339838a40b351409fb7750f632a3810cb8774db195f49eed8a3ed5",
"EndpointID": "49993412f56f7bd29edd55091bd77bba66a40ae566daf8111f891be9aa21d4ce",
"Gateway": "172.17.0.1",
"IPAddress": "172.17.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:11:00:02",
"DriverOpts": null
}
}
}

I had to set docker daemon MTU to be the same as the network card.
ip link show
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> **mtu 1450** qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
edited /etc/docker/daemon.json
"mtu": 1450
}
Problem solved. Tricky one, when I did remote debugging I notice that it was blocking on socket read, from there google was my friend.
https://mlohr.com/docker-mtu/

Unexpected Web-Application closing immediately after starting

These are the current dependencies included in my web application.
dependencies {
implementation("org.springframework.boot:spring-boot-starter-data-jpa")
implementation("org.springframework.boot:spring-boot-starter-data-rest")
implementation("org.springframework.boot:spring-boot-starter-web")
implementation("org.springframework.boot:spring-boot-starter-web-services")
implementation("com.fasterxml.jackson.module:jackson-module-kotlin")
implementation("org.jetbrains.kotlin:kotlin-reflect")
implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8")
implementation("com.google.firebase:firebase-admin:6.10.0")
}
Every time I run the application I get the following error:
2019-09-30 13:32:38.385 INFO 17104 --- [ restartedMain] unito.taas.project.ProjectApplicationKt : Starting ProjectApplicationKt on LAPTOP-K1DHEJQ6 with PID 17104 (C:\Users\beppe\Desktop\project\TAAS_project\project\build\classes\kotlin\main started by beppe in C:\Users\beppe\Desktop\project\TAAS_project)
2019-09-30 13:32:38.388 INFO 17104 --- [ restartedMain] unito.taas.project.ProjectApplicationKt : No active profile set, falling back to default profiles: default
2019-09-30 13:32:38.444 INFO 17104 --- [ restartedMain] .e.DevToolsPropertyDefaultsPostProcessor : Devtools property defaults active! Set 'spring.devtools.add-properties' to 'false' to disable
2019-09-30 13:32:39.177 INFO 17104 --- [ restartedMain] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data repositories in DEFAULT mode.
2019-09-30 13:32:39.263 INFO 17104 --- [ restartedMain] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 77ms. Found 3 repository interfaces.
2019-09-30 13:32:39.785 INFO 17104 --- [ restartedMain] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
2019-09-30 13:32:39.976 INFO 17104 --- [ restartedMain] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
2019-09-30 13:32:40.045 INFO 17104 --- [ restartedMain] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [
name: default
...]
2019-09-30 13:32:40.127 INFO 17104 --- [ restartedMain] org.hibernate.Version : HHH000412: Hibernate Core {5.3.11.Final}
2019-09-30 13:32:40.128 INFO 17104 --- [ restartedMain] org.hibernate.cfg.Environment : HHH000206: hibernate.properties not found
2019-09-30 13:32:40.317 INFO 17104 --- [ restartedMain] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.0.4.Final}
2019-09-30 13:32:40.734 INFO 17104 --- [ restartedMain] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.H2Dialect
2019-09-30 13:32:41.419 INFO 17104 --- [ restartedMain] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'
2019-09-30 13:32:41.439 INFO 17104 --- [ restartedMain] o.s.b.d.a.OptionalLiveReloadServer : LiveReload server is running on port 35729
2019-09-30 13:32:43.049 INFO 17104 --- [ restartedMain] unito.taas.project.ProjectApplicationKt : Started ProjectApplicationKt in 5.053 seconds (JVM running for 5.806)
2019-09-30 13:32:43.063 INFO 17104 --- [ Thread-9] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
2019-09-30 13:32:43.068 INFO 17104 --- [ Thread-9] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...
2019-09-30 13:32:43.073 INFO 17104 --- [ Thread-9] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
How can I avoid the application closing?
EDIT:
The following is my SpringBoot annotated class
package unito.taas.project
import org.springframework.boot.autoconfigure.SpringBootApplication
import org.springframework.boot.runApplication
#SpringBootApplication
class ProjectApplication
fun main(args: Array<String>) {
runApplication<ProjectApplication>(*args)
}

Spring Kafka rebalance clarification

I just want to understand kafka rebalance.This is my listener method. I have configured RetryTemplate with consumer factory to retry 20 times with a backoff delay of 20secs. I'm using spring-kafka 1.2.2(we are planning to upgrade client) and using manual acks.
#KafkaListener(id = "${kafka.listener-id}", topics = "${kafka.topic}")
public void listen(final ConsumerRecord<String, String> consumerRecord,
final Acknowledgment acknowledgment) throws ServiceResponseException {}
if (true){
System.out.println("throwing exception ");
throw new RuntimeException();
}
try {
acknowledgment.acknowledge();
LOGGER.info("Kafka acknowledgment sent for Transaction ID:");
} catch (Exception e) {
LOGGER.info("Exception encountered when acking record with transaction id: {}");
}
}
I have 2 workers that has concurrency 2 each. On kafka I have 3 partitions. I started one worker and 3 partitions are assigned to worker1. And then I sent a message. And a RuntimeException is thrown in listener and this happens for 20 times with 20 sec delay. Then when I started worker2 kafka rebalance triggers but partitions are not yet assigned. worker1 fails with message "Error while processing: ConsumerRecord"(after getContainerProperties().getShutdownTimeout())and and then all consumers join the group. And now the same message is delivered to worker2.
1) And this is working as I needed it to work. But I have a question, when a rebalance triggers why is partition assignment not happening immediately instead it waits for the worker1 to stop completely(waiting for getContainerProperties().getShutdownTimeout()) and then all consumers from worker1 and worker2 joins groups.
2) And during rebalance I observed that consumers stop calling poll(from logs below). Is it true?
Trace logs from worker 1:
2018-09-23 13:52:53.259 TRACE 6384 --- [ listener-2-L-1] essageListenerContainer$ListenerConsumer : No records to process
2018-09-23 13:52:53.259 TRACE 6384 --- [ listener-0-L-1] essageListenerContainer$ListenerConsumer : No records to process
2018-09-23 13:52:53.384 DEBUG 6384 --- [ listener-1-C-1] essageListenerContainer$ListenerConsumer : Received: 0 records
2018-09-23 13:52:53.384 TRACE 6384 --- [ listener-1-C-1] essageListenerContainer$ListenerConsumer : Polling (paused=false)...
2018-09-23 13:52:53.977 DEBUG 6384 --- [ listener-0-C-1] essageListenerContainer$ListenerConsumer : Received: 0 records
2018-09-23 13:52:53.977 TRACE 6384 --- [ listener-0-C-1] essageListenerContainer$ListenerConsumer : Polling (paused=false)...
2018-09-23 13:52:54.008 DEBUG 6384 --- [ listener-2-C-1] essageListenerContainer$ListenerConsumer : Received: 0 records
2018-09-23 13:52:54.008 TRACE 6384 --- [ listener-2-C-1] essageListenerContainer$ListenerConsumer : Polling (paused=false)...
2018-09-23 13:52:54.023 INFO 6384 --- [ listener-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : Revoking previously assigned partitions [messages-0] for group mris-group
2018-09-23 13:52:54.023 TRACE 6384 --- [ listener-0-C-1] essageListenerContainer$ListenerConsumer : Received partition revocation notification, and will stop the invoker.
2018-09-23 13:52:54.023 DEBUG 6384 --- [ listener-0-C-1] essageListenerContainer$ListenerConsumer : Stopping invoker
2018-09-23 13:52:54.081 INFO 6384 --- [ listener-1-C-1] o.a.k.c.c.internals.ConsumerCoordinator : Revoking previously assigned partitions [messages-1] for group mris-group
2018-09-23 13:52:54.081 TRACE 6384 --- [ listener-1-C-1] essageListenerContainer$ListenerConsumer : Received partition revocation notification, and will stop the invoker.
2018-09-23 13:52:54.081 DEBUG 6384 --- [ listener-1-C-1] essageListenerContainer$ListenerConsumer : Stopping invoker
2018-09-23 13:52:54.241 INFO 6384 --- [ listener-2-C-1] o.a.k.c.c.internals.ConsumerCoordinator : Revoking previously assigned partitions [messages-2] for group mris-group
2018-09-23 13:52:54.241 TRACE 6384 --- [ listener-2-C-1] essageListenerContainer$ListenerConsumer : Received partition revocation notification, and will stop the invoker.
2018-09-23 13:52:54.241 DEBUG 6384 --- [ listener-2-C-1] essageListenerContainer$ListenerConsumer : Stopping invoker
2018-09-23 13:52:54.264 DEBUG 6384 --- [ listener-2-C-1] essageListenerContainer$ListenerConsumer : Invoker stopped
2018-09-23 13:52:54.264 DEBUG 6384 --- [ listener-0-C-1] essageListenerContainer$ListenerConsumer : Invoker stopped
2018-09-23 13:52:54.264 INFO 6384 --- [ listener-2-C-1] o.s.k.l.KafkaMessageListenerContainer : partitions revoked:[messages-2]
2018-09-23 13:52:54.264 INFO 6384 --- [ listener-0-C-1] o.s.k.l.KafkaMessageListenerContainer : partitions revoked:[messages-0]
2018-09-23 13:52:54.264 INFO 6384 --- [ listener-2-C-1] o.a.k.c.c.internals.AbstractCoordinator : (Re-)joining group mris-group
2018-09-23 13:52:54.265 INFO 6384 --- [ listener-0-C-1] o.a.k.c.c.internals.AbstractCoordinator : (Re-)joining group mris-group
2018-09-23 13:53:09.355 DEBUG 6384 --- [ listener-1-L-1] .a.RecordMessagingMessageListenerAdapter : Processing [GenericMessage [payload=<removed>]
throwing exception
2018-09-23 13:53:24.083 DEBUG 6384 --- [ listener-1-C-1] essageListenerContainer$ListenerConsumer : Interrupting invoker
2018-09-23 13:53:24.083 DEBUG 6384 --- [ listener-1-C-1] essageListenerContainer$ListenerConsumer : Invoker stopped
2018-09-23 13:53:24.085 INFO 6384 --- [ listener-1-C-1] essageListenerContainer$ListenerConsumer : Invoker timed out while waiting for shutdown and will be canceled.
2018-09-23 13:53:24.085 INFO 6384 --- [ listener-1-C-1] o.s.k.l.KafkaMessageListenerContainer : partitions revoked:[messages-1]
2018-09-23 13:53:24.085 INFO 6384 --- [ listener-1-C-1] o.a.k.c.c.internals.AbstractCoordinator : (Re-)joining group mris-group
2018-09-23 13:53:24.101 ERROR 6384 --- [ listener-1-L-1] o.s.kafka.listener.LoggingErrorHandler : Error while processing: ConsumerRecord(topic = messages, partition = 1, offset = 0, CreateTime = 1537725149052, checksum = 3567644394, serialized key size = 27, serialized value size = 1952, key = test_hotfix1#test.com, value = <removed>])
org.springframework.retry.backoff.BackOffInterruptedException: Thread interrupted while sleeping; nested exception is java.lang.InterruptedException: sleep interrupted
at org.springframework.retry.backoff.FixedBackOffPolicy.doBackOff(FixedBackOffPolicy.java:86) ~[spring-retry-1.2.0.RELEASE.jar:na]
at org.springframework.retry.backoff.StatelessBackOffPolicy.backOff(StatelessBackOffPolicy.java:36) ~[spring-retry-1.2.0.RELEASE.jar:na]
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:305) ~[spring-retry-1.2.0.RELEASE.jar:na]
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:179) ~[spring-retry-1.2.0.RELEASE.jar:na]
at org.springframework.kafka.listener.adapter.RetryingAcknowledgingMessageListenerAdapter.onMessage(RetryingAcknowledgingMessageListenerAdapter.java:73) ~[spring-kafka-1.2.2.RELEASE.jar:na]
at org.springframework.kafka.listener.adapter.RetryingAcknowledgingMessageListenerAdapter.onMessage(RetryingAcknowledgingMessageListenerAdapter.java:39) ~[spring-kafka-1.2.2.RELEASE.jar:na]
at org.springframework.kafka.listener.adapter.FilteringAcknowledgingMessageListenerAdapter.onMessage(FilteringAcknowledgingMessageListenerAdapter.java:55) ~[spring-kafka-1.2.2.RELEASE.jar:na]
at org.springframework.kafka.listener.adapter.FilteringAcknowledgingMessageListenerAdapter.onMessage(FilteringAcknowledgingMessageListenerAdapter.java:34) ~[spring-kafka-1.2.2.RELEASE.jar:na]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:794) [spring-kafka-1.2.2.RELEASE.jar:na]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:738) [spring-kafka-1.2.2.RELEASE.jar:na]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.access$2200(KafkaMessageListenerContainer.java:245) [spring-kafka-1.2.2.RELEASE.jar:na]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer$ListenerInvoker.run(KafkaMessageListenerContainer.java:1031) [spring-kafka-1.2.2.RELEASE.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_162]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_162]
Caused by: java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method) [na:1.8.0_162]
at org.springframework.retry.backoff.ThreadWaitSleeper.sleep(ThreadWaitSleeper.java:30) ~[spring-retry-1.2.0.RELEASE.jar:na]
at org.springframework.retry.backoff.FixedBackOffPolicy.doBackOff(FixedBackOffPolicy.java:83) ~[spring-retry-1.2.0.RELEASE.jar:na]
... 14 common frames omitted
2018-09-23 13:53:24.101 INFO 6384 --- [ listener-1-C-1] o.a.k.c.c.internals.AbstractCoordinator : Successfully joined group mris-group with generation 10
2018-09-23 13:53:24.101 INFO 6384 --- [ listener-2-C-1] o.a.k.c.c.internals.AbstractCoordinator : Successfully joined group mris-group with generation 10
2018-09-23 13:53:24.102 INFO 6384 --- [ listener-0-C-1] o.a.k.c.c.internals.AbstractCoordinator : Successfully joined group mris-group with generation 10
2018-09-23 13:53:24.102 INFO 6384 --- [ listener-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : Setting newly assigned partitions [messages-0] for group mris-group
2018-09-23 13:53:24.102 INFO 6384 --- [ listener-1-C-1] o.a.k.c.c.internals.ConsumerCoordinator : Setting newly assigned partitions [messages-2] for group mris-group
2018-09-23 13:53:24.102 INFO 6384 --- [ listener-2-C-1] o.a.k.c.c.internals.ConsumerCoordinator : Setting newly assigned partitions [] for group mris-group
2018-09-23 13:53:24.103 DEBUG 6384 --- [ listener-2-C-1] essageListenerContainer$ListenerConsumer : Committing: {}
2018-09-23 13:53:24.103 INFO 6384 --- [ listener-2-C-1] o.s.k.l.KafkaMessageListenerContainer : partitions assigned:[]
2018-09-23 13:53:24.103 DEBUG 6384 --- [ listener-0-C-1] essageListenerContainer$ListenerConsumer : Committing: {messages-0=OffsetAndMetadata{offset=0, metadata=''}}
2018-09-23 13:53:24.104 DEBUG 6384 --- [ listener-1-C-1] essageListenerContainer$ListenerConsumer : Committing: {messages-2=OffsetAndMetadata{offset=0, metadata=''}}
2018-09-23 13:53:24.106 INFO 6384 --- [ listener-0-C-1] o.s.k.l.KafkaMessageListenerContainer : partitions assigned:[messages-0]
2018-09-23 13:53:24.107 DEBUG 6384 --- [ listener-0-C-1] essageListenerContainer$ListenerConsumer : Received: 0 records
2018-09-23 13:53:24.107 TRACE 6384 --- [ listener-0-C-1] essageListenerContainer$ListenerConsumer : Polling (paused=false)...
2018-09-23 13:53:24.108 INFO 6384 --- [ listener-1-C-1] o.s.k.l.KafkaMessageListenerContainer : partitions assigned:[messages-2]
2018-09-23 13:53:24.108 DEBUG 6384 --- [ listener-1-C-1] essageListenerContainer$ListenerConsumer : Received: 0 records
2018-09-23 13:53:24.108 TRACE 6384 --- [ listener-1-C-1] essageListenerContainer$ListenerConsumer : Polling (paused=false)...
2018-09-23 13:53:24.207 DEBUG 6384 --- [ listener-2-C-1] essageListenerContainer$ListenerConsumer : Received: 0 records
2018-09-23 13:53:24.207 TRACE 6384 --- [ listener-2-C-1] essageListenerContainer$ListenerConsumer : Polling (paused=false)...
2018-09-23 13:53:25.111 TRACE 6384 --- [ listener-0-L-2] essageListenerContainer$ListenerConsumer : No records to process
Trace logs from worker2:
2018-09-23 13:53:24.102 INFO 6401 --- [ listener-2-C-1] o.a.k.c.c.internals.AbstractCoordinator : Successfully joined group mris-group with generation 10
2018-09-23 13:53:24.104 INFO 6401 --- [ listener-1-C-1] o.a.k.c.c.internals.AbstractCoordinator : Successfully joined group mris-group with generation 10
2018-09-23 13:53:24.105 INFO 6401 --- [ listener-1-C-1] o.a.k.c.c.internals.ConsumerCoordinator : Setting newly assigned partitions [] for group mris-group
2018-09-23 13:53:24.105 INFO 6401 --- [ listener-2-C-1] o.a.k.c.c.internals.ConsumerCoordinator : Setting newly assigned partitions [] for group mris-group
2018-09-23 13:53:24.105 DEBUG 6401 --- [ listener-2-C-1] essageListenerContainer$ListenerConsumer : Committing: {}
2018-09-23 13:53:24.105 DEBUG 6401 --- [ listener-1-C-1] essageListenerContainer$ListenerConsumer : Committing: {}
2018-09-23 13:53:24.105 INFO 6401 --- [ listener-2-C-1] o.s.k.l.KafkaMessageListenerContainer : partitions assigned:[]
2018-09-23 13:53:24.105 INFO 6401 --- [ listener-1-C-1] o.s.k.l.KafkaMessageListenerContainer : partitions assigned:[]
2018-09-23 13:53:24.106 INFO 6401 --- [ listener-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : Setting newly assigned partitions [messages-1] for group mris-group
2018-09-23 13:53:24.111 DEBUG 6401 --- [ listener-0-C-1] essageListenerContainer$ListenerConsumer : Committing: {messages-1=OffsetAndMetadata{offset=0, metadata=''}}
2018-09-23 13:53:24.115 INFO 6401 --- [ listener-0-C-1] o.s.k.l.KafkaMessageListenerContainer : partitions assigned:[messages-1]
2018-09-23 13:53:24.118 DEBUG 6401 --- [ listener-0-C-1] essageListenerContainer$ListenerConsumer : Received: 0 records
2018-09-23 13:53:24.118 TRACE 6401 --- [ listener-0-C-1] essageListenerContainer$ListenerConsumer : Polling (paused=false)...
2018-09-23 13:53:24.189 DEBUG 6401 --- [ listener-0-C-1] essageListenerContainer$ListenerConsumer : Received: 1 records
2018-09-23 13:53:24.189 TRACE 6401 --- [ listener-0-C-1] essageListenerContainer$ListenerConsumer : Polling (paused=false)...
2018-09-23 13:53:24.202 TRACE 6401 --- [ listener-0-L-1] essageListenerContainer$ListenerConsumer : Processing ConsumerRecord(topic = messages, partition = 1, offset = 0, CreateTime = 1537725149052, checksum = 3567644394, serialized key size = 27, serialized value size = 1952, key = test_hotfix1#test.com, value = <removed>)
2018-09-23 13:53:24.209 DEBUG 6401 --- [ listener-1-C-1] essageListenerContainer$ListenerConsumer : Received: 0 records
2018-09-23 13:53:24.209 DEBUG 6401 --- [ listener-2-C-1] essageListenerContainer$ListenerConsumer : Received: 0 records
2018-09-23 13:53:24.209 TRACE 6401 --- [ listener-1-C-1] essageListenerContainer$ListenerConsumer : Polling (paused=false)...
2018-09-23 13:53:24.210 TRACE 6401 --- [ listener-2-C-1] essageListenerContainer$ListenerConsumer : Polling (paused=false)...
2018-09-23 13:53:24.216 DEBUG 6401 --- [ listener-0-L-1] .a.RecordMessagingMessageListenerAdapter : Processing [GenericMessage [payload=<removed>]
throwing exception
2018-09-23 13:53:25.194 DEBUG 6401 --- [ listener-0-C-1] essageListenerContainer$ListenerConsumer : Received: 0 records
2018-09-23 13:53:25.194 TRACE 6401 --- [ listener-0-C-1] essageListenerContainer$ListenerConsumer : Polling (paused=false)...

Versions prior to 1.3 had a very complicated threading model to avoid rebalancing due to a slow listener. KIP-62 enabled us to use a much simpler threading model in 1.3 and later.
1.2.x is no longer supported, and I don't have the time (or inclination) to go back to figure out what happened. Please upgrade to 1.3.7 (or even better, 2.1.10).

Resources