I am unable to understand the reason behind intermittent HikariCP Connection is not available.
From the logs, It doesn't look like a connection leak issue. A bigger problem is I am unable to predictably reproduce the error. Following is a sample log trace where the error starts while this gist contains it till the end.
2017-12-12T19:31:55.958Z DEBUG <> [HikariPool-1 housekeeper] com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
2017-12-12T19:31:57.052Z WARN <> [main] c.zaxxer.hikari.pool.ProxyConnection - HikariPool-1 - Connection org.postgresql.jdbc.PgConnection#1de5f0ef marked as broken because of SQLSTATE(08P01), ErrorCode(0)
org.postgresql.util.PSQLException: Expected command status BEGIN, got EMPTY.
at org.postgresql.core.v3.QueryExecutorImpl$1.handleCommandStatus(QueryExecutorImpl.java:515)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2180)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:288)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:430)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:356)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:168)
at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:116)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:70)
at org.hibernate.id.SequenceGenerator.generateHolder(SequenceGenerator.java:116)
at org.hibernate.id.SequenceHiLoGenerator.generate(SequenceHiLoGenerator.java:62)
at org.hibernate.event.internal.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:101)
at org.hibernate.jpa.event.internal.core.JpaPersistEventListener.saveWithGeneratedId(JpaPersistEventListener.java:67)
... MORE DETAILS IN [GIST][1]
2017-12-12T19:31:57.067Z ERROR <> [main] c.o.r.s.t.RAUTwitterUserService - populateFromPayload: id=128, twitter_handle=non_local, exception=org.springframework.transaction.TransactionSystemException: Could not roll back JPA transaction; nested exception is javax.persistence.PersistenceException: unexpected error when rollbacking
2017-12-12T19:31:57.067Z INFO <> [main] c.o.r.s.t.TwitterCollectorService - RAU from RAU service: RAU(id=129, twitter=non_global)
2017-12-12T19:31:57.089Z DEBUG <> [HikariPool-1 connection adder] com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection#419dcf40
Related
**> I am getting exception after upgrading oracle11g to oracle19C, using hikari as connection pool and hibernate and spring boot. Please
help.
When using IN query with lower number of value list its working but with large numbers its giving error.
Error Recieved:-**
WARN 2022-05-20 11:56:03,250 [http-nio-8080-exec-3] HikariPool-1 - Connection oracle.jdbc.driver.T4CConnection#40896e8e marked as broken because of SQLSTATE(08006), ErrorCode(17002)java.sql.SQLRecoverableException: IO Error: Connection reset
Caused by: java.net.SocketException: Connection reset
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:186)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
at oracle.net.ns.Packet.receive(Packet.java:316)
at oracle.net.ns.DataPacket.receive(DataPacket.java:128)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:257)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:177)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:95)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:132)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:88)
at oracle.jdbc.driver.T4CMAREngineStream.unmarshalUB1(T4CMAREngineStream.java:395)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:410)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:269)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:655)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:270)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:91)
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:807)
... 148 common frames omitted
WARN 2022-05-20 11:56:03,251 [http-nio-8080-exec-3] SQL Error: 17002, SQLState: 08006
ERROR 2022-05-20 11:56:03,251 [http-nio-8080-exec-3] IO Error: Connection reset
DEBUG 2022-05-20 11:56:03,251 [HikariPool-1 connection closer] HikariPool-1 - Closing connection oracle.jdbc.driver.T4CConnection#40896e8e: (connection is broken)
ERROR 2022-05-20 11:56:03,252 [http-nio-8080-exec-3] Application exception overridden by rollback exception
org.springframework.dao.DataAccessResourceFailureException: could not extract ResultSet; nested exception is org.hibernate.exception.JDBCConnectionException: could not extract ResultSet
**When I hit my API endpoints it redirects to the spring default login page although I didn't use any spring security ** How can I solve this problem. And also no matter which port I used always failed to run showing the avobe error.
2021-03-18 22:20:28.987 INFO 1136 --- [ restartedMain] .s.s.UserDetailsServiceAutoConfiguration :
Using generated security password: 10ca6485-f9b1-45fd-8422-7f521974e807
2021-03-18 22:20:29.058 INFO 1136 --- [ restartedMain] o.s.s.web.DefaultSecurityFilterChain : Will secure any request with [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter#5afcaa40, org.springframework.security.web.context.SecurityContextPersistenceFilter#1ecd6c38, org.springframework.security.web.header.HeaderWriterFilter#57c0b97a, org.springframework.security.web.csrf.CsrfFilter#70e744e9, org.springframework.security.web.authentication.logout.LogoutFilter#ddc05c8, org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter#72ab48fb, org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter#77c3d4dd, org.springframework.security.web.authentication.ui.DefaultLogoutPageGeneratingFilter#53c95e38, org.springframework.security.web.authentication.www.BasicAuthenticationFilter#456f30b5, org.springframework.security.web.savedrequest.RequestCacheAwareFilter#723b9c48, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter#217298a9, org.springframework.security.web.authentication.AnonymousAuthenticationFilter#6f9f366c, org.springframework.security.web.session.SessionManagementFilter#3487264d, org.springframework.security.web.access.ExceptionTranslationFilter#bba9566, org.springframework.security.web.access.intercept.FilterSecurityInterceptor#2c1e9611]
2021-03-18 22:20:30.156 INFO 1136 --- [ restartedMain] o.s.cloud.commons.util.InetUtils
2021-03-18 22:20:30.366 INFO 1136 --- [ restartedMain] com.netflix.discovery.DiscoveryClient : Getting all instance registry info from the eureka server
2021-03-18 22:20:31.473 INFO 1136 --- [ restartedMain] c.n.d.s.t.d.RedirectingEurekaHttpClient : Request execution error. endpoint=DefaultEndpoint{ serviceUrl='http://localhost:8761/eureka/}, exception=I/O error on GET request for "http://localhost:8761/eureka/apps/": Connection refused: connect; nested exception is java.net.ConnectException: Connection refused: connect stacktrace=org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost:8761/eureka/apps/": Connection refused: connect; nested exception is java.net.ConnectException: Connection refused: connect
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:785)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:711)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:602)
at org.springframework.cloud.netflix.eureka.http.RestTemplateEurekaHttpClient.getApplicationsInternal(RestTemplateEurekaHttpClient.java:145)
at org.springframework.cloud.netflix.eureka.http.RestTemplateEurekaHttpClient.getApplications(RestTemplateEurekaHttpClient.java:135)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$6.execute(EurekaHttpClientDecorator.java:137)
at com.netflix.discovery.shared.transport.decorator.RedirectingEurekaHttpClient.executeOnNewServer(RedirectingEurekaHttpClient.java:121)
at com.productservice.ProductServiceApplication.main(ProductServiceApplication.java:12)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49)
Caused by: java.net.ConnectException: Connection refused: connect
at java.base/sun.nio.ch.Net.connect0(Native Method)
at java.base/sun.nio.ch.Net.connect(Net.java:574)
2021-03-18 22:20:31.473 WARN 1136 --- [ restartedMain] c.n.d.s.t.d.RetryableEurekaHttpClient : Request execution failed with message: I/O error on GET request for "http://localhost:8761/eureka/apps/": Connection refused: connect; nested exception is java.net.ConnectException: Connection refused: connect
2021-03-18 22:20:31.474 INFO 1136 --- [ restartedMain] com.netflix.discovery.DiscoveryClient : DiscoveryClient_PRODUCT-SERVICE/localhost:product-service:8888 - was unable to refresh its cache! This periodic background refresh will be retried in 30 seconds. status = Cannot execute request on any known server stacktrace = com.netflix.discovery.shared.transport.TransportException: Cannot execute request on any known server
at com.netflix.discovery.shared.transport.decorator.RetryableEurekaHttpClient.execute(RetryableEurekaHttpClient.java:112)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.getApplications(EurekaHttpClientDecorator.java:134)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$6.execute(EurekaHttpClientDecorator.java:137)
at com.netflix.discovery.shared.transport.decorator.SessionedEurekaHttpClient.execute(SessionedEurekaHttpClient.java:77)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.getApplications(EurekaHttpClientDecorator.java:134)
at com.netflix.discovery.DiscoveryClient.getAndStoreFullRegistry(DiscoveryClient.java:1101)
at java.base/java.lang.Thread.run(Thread.java:832) ~[na:na]
2021-03-18 22:20:32.556 INFO 1136 --- [ restartedMain] com.netflix.discovery.DiscoveryClient : Unregistering ...
2021-03-18 22:20:33.561 INFO 1136 --- [ restartedMain] c.n.d.s.t.d.RedirectingEurekaHttpClient : Request execution error. endpoint=DefaultEndpoint{ serviceUrl='http://localhost:8761/eureka/}, exception=I/O error on DELETE request for "http://localhost:8761/eureka/apps/PRODUCT-SERVICE/localhost:product-service:8888": Connection refused: connect; nested exception is java.net.ConnectException: Connection refused: connect stacktrace=org.springframework.web.client.ResourceAccessException: I/O error on DELETE request for "http://localhost:8761/eureka/apps/PRODUCT-SERVICE/localhost:product-service:8888": Connection refused: connect; nested exception is java.net.ConnectException: Connection refused: connect
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:785)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:711)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:602)
at org.springframework.cloud.netflix.eureka.http.RestTemplateEurekaHttpClient.cancel(RestTemplateEurekaHttpClient.java:87)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$2.execute(EurekaHttpClientDecorator.java:74)
at com.netflix.discovery.shared.transport.decorator.RedirectingEurekaHttpClient.executeOnNewServer(RedirectingEurekaHttpClient.java:121)
at com.netflix.discovery.shared.transport.decorator.RedirectingEurekaHttpClient.execute(RedirectingEurekaHttpClient.java:80)
at com.netflix.discovery.sh
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2021-03-18 22:20:33.651 ERROR 1136 --- [ restartedMain] o.s.b.d.LoggingFailureAnalysisReporter :
***************************
APPLICATION FAILED TO START
***************************
Description:
Web server failed to start. Port 8888 was already in use.
Action:
Identify and stop the process that's listening on port 8888 or configure this application to listen on another port.
Process finished with exit code 0
**When I hit my API endpoints it redirects to the spring default login page although I didn't use any spring security ** How can I solve this problem. And also no matter which port I used always failed to run showing the avobe error.
I'm getting error when deployingg project
INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting...
WARN o.s.b.a.j.DataSourceHealthIndicator - DataSource health check failed
java.lang.RuntimeException: Driver oracle.jdbc.OracleDriver claims to not accept jdbcUrl, jdbc-url:...
Connection String
jdbc:oracle:thin:
#(DESCRIPTION=(CONNECT_TIMEOUT=5)(TRANSPORT_CONNECT_TIMEOUT=3)(RETRY_COUNT=1)(ADDRESS=(PROTOCOL=TCP)(HOST=my2.org)(PORT=123))(ADDRESS=(PROTOCOL=TCP)(HOST=my1.org)(PORT=123))(CONNECT_DATA=(SERVICE_NAME=DG)))
I have a Spring boot application in version 2.1.2.RELEASE. Once a month or few weeks application crashes without any specific information in logs.
Server where app is deploy is CentOS ver 7, app is launched by a systemd service.
Tried analysing heap dump, no luck, checked logs of postgres, tomcat etc. nothing there.
Stacktrace:
INFO ;read-5;523575402;2019-08-07 17:27:24,714;o.s.i.e.EventDrivenConsumer; Removing {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
INFO ;read-5;523575418;2019-08-07 17:27:24,730;o.s.i.c.PublishSubscribeChannel; Channel 'application.errorChannel' has 0 subscriber(s).
INFO ;read-5;523575418;2019-08-07 17:27:24,730;o.s.i.e.EventDrivenConsumer; stopped _org.springframework.integration.errorLogger
INFO ;read-5;523575502;2019-08-07 17:27:24,814;o.s.s.c.ThreadPoolTaskScheduler; Shutting down ExecutorService 'taskScheduler'
INFO ;read-5;523575541;2019-08-07 17:27:24,853;o.s.s.c.ThreadPoolTaskExecutor; Shutting down ExecutorService 'applicationTaskExecutor'
INFO ;read-5;523575572;2019-08-07 17:27:24,884;o.s.o.j.LocalContainerEntityManagerFactoryBean; Closing JPA EntityManagerFactory for persistence unit 'default'
INFO ;read-5;523575596;2019-08-07 17:27:24,908;c.z.h.HikariDataSource; HikariPool-1 - Shutdown initiated...
INFO ;read-5;523575631;2019-08-07 17:27:24,943;c.z.h.HikariDataSource; HikariPool-1 - Shutdown completed.
WARN ;exec-8;523575668;2019-08-07 17:27:24,980;c.z.h.p.ProxyConnection; HikariPool-1 - Connection org.postgresql.jdbc.PgConnection#6ff70ea8 marked as broken because of SQLSTATE(08006), ErrorCode(0)
org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:335)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:143)
at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:106)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:60)
at org.hibernate.loader.Loader.getResultSet(Loader.java:2173)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1936)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1898)
WARN ;exec-6;523575677;2019-08-07 17:27:24,989;c.z.h.p.ProxyConnection; HikariPool-1 - Connection org.postgresql.jdbc.PgConnection#71c6647 marked as broken because of SQLSTATE(08006), ErrorCode(0)
org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:335)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:143)
at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:106)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
WARN ;exec-8;523575721;2019-08-07 17:27:25,033;o.h.e.j.s.SqlExceptionHelper; SQL Error: 0, SQLState: 08006
WARN ;exec-6;523575721;2019-08-07 17:27:25,033;o.h.e.j.s.SqlExceptionHelper; SQL Error: 0, SQLState: 08006
ERROR;exec-8;523575721;2019-08-07 17:27:25,033;o.h.e.j.s.SqlExceptionHelper; An I/O error occurred while sending to the backend.
ERROR;exec-6;523575721;2019-08-07 17:27:25,033;o.h.e.j.s.SqlExceptionHelper; An I/O error occurred while sending to the backend.
INFO ;exec-6;523575885;2019-08-07 17:27:25,197;o.h.e.i.DefaultLoadEventListener; HHH000327: Error performing load command : org.hibernate.exception.JDBCConnectionException: could not extract ResultSet
org.hibernate.exception.JDBCConnectionException: could not extract ResultSet
at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:115)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
Has anyone got idea how can I what might be going on? Last crash was 3, 4 weeks before that.
That type of error happens when the database connection has been lost, which in turn can happen for a whole number of reasons, the most simple one being that it's being restarted after system updates or as part of a scheduled operation.
HikariCP should be able to recover according to the following link (provided that your HikariCP version is recent enough):
https://github.com/brettwooldridge/HikariCP/wiki/Rapid-Recovery
You may need to configure some properties in your pool. The full list of properties is available here:
https://github.com/brettwooldridge/HikariCP
I recommend you configure the following in HikariCP:
connectionTestQuery: will test the connection periodically, dropping dead connections. Use it in conjunction with socketTimeout in your PostgreSQL connection string
leakDetectionThreshold: will help with applications having connection leaks
I'm trying to use Hibernate with Spring and PostgreSQL, but I have lots of errors like:
org.postgresql.Driver : Connection error
ERROR 3424 --- [ main] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Exception during pool initialization.
I have put postgreSQL.Driver in my libs but nothing to do error say there.
Edit:
2018-03-19 15:54:53.660 INFO 7640 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
2018-03-19 15:54:54.664 WARN 7640 --- [ main] unknown.jul.logger : ConnectException occurred while connecting to localhost:5432`
`at com.ttmik.back.MainKt.main(main.kt:19) ~[classes/:na]
2018-03-19 15:54:54.675 ERROR 7640 --- [ main] org.postgresql.Driver : Connection error:
org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
Check if your PostgreSQL can be reached by using telnet. If you get something like:
telnet localhost 5432
Connecting To localhost...
Could not open connection to the host, on port 5432: Connection failed
then most likely need to change listen_addresses to * in postgresql.conf file as per this answer on serverfault.