Spring boot app, Embedded DebeziumEngine, MySql connector.
when closing application (ctrl+c), Im closing Debezium Engine (DebeziumEngine.close) within #PreDestroy, after few seconds, connectors throws SQLNonTransientConnectionException.
Log:
2022-09-21 06:24:54,487 DEBUG {SpringApplicationShutdownHook} (DBEventsListener.java:425) DB events listener is closing...
2022-09-21 06:24:54,487 INFO {SpringApplicationShutdownHook} (EmbeddedEngine.java:1047) Stopping the embedded engine
2022-09-21 06:24:54,488 INFO {SpringApplicationShutdownHook} (EmbeddedEngine.java:1055) Waiting for PT5M for connector to stop
2022-09-21 06:24:56,523 DEBUG {modelserviceDBCacheEventsWorker1} (ChangeEventQueue.java:251) checking for more records...
2022-09-21 06:24:56,523 DEBUG {modelserviceDBCacheEventsWorker1} (EmbeddedEngine.java:789) Embedded engine returned from polling task for records
2022-09-21 06:24:56,523 DEBUG {modelserviceDBCacheEventsWorker1} (EmbeddedEngine.java:828) Received no records from the task
2022-09-21 06:24:56,523 INFO {modelserviceDBCacheEventsWorker1} (EmbeddedEngine.java:846) Stopping the task and engine
2022-09-21 06:24:56,524 INFO {modelserviceDBCacheEventsWorker1} (BaseSourceTask.java:238) Stopping down connector
2022-09-21 06:24:56,539 INFO {debezium-mysqlconnector-skybox_mysql_db_server-change-event-source-coordinator} (ChangeEventSourceCoordinator.java:175) Finished streaming
2022-09-21 06:24:56,539 INFO {blc-127.0.0.1:3306} (MySqlStreamingChangeEventSource.java:1205) Stopped reading binlog after 0 events, last recorded offset: {transaction_id=null, ts_sec=1663661228, file=mysqlbin.000001, pos=10879727, server_id=1, event=1}
2022-09-21 06:24:56,555 ERROR {modelserviceDBCacheEventsWorker1} (MySqlConnectorTask.java:224) Exception while closing JDBC connection
java.sql.SQLNonTransientConnectionException: Communications link failure during rollback(). Transaction resolution unknown.
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110) ~[mysql-connector-java-8.0.30.jar!/:8.0.30]
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97) ~[mysql-connector-java-8.0.30.jar!/:8.0.30]
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89) ~[mysql-connector-java-8.0.30.jar!/:8.0.30]
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63) ~[mysql-connector-java-8.0.30.jar!/:8.0.30]
at com.mysql.cj.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:1835) ~[mysql-connector-java-8.0.30.jar!/:8.0.30]
at com.mysql.cj.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:1693) ~[mysql-connector-java-8.0.30.jar!/:8.0.30]
at com.mysql.cj.jdbc.ConnectionImpl.close(ConnectionImpl.java:713) ~[mysql-connector-java-8.0.30.jar!/:8.0.30]
at io.debezium.jdbc.JdbcConnection.lambda$doClose$3(JdbcConnection.java:955) ~[debezium-core-1.9.5.Final.jar!/:1.9.5.Final]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_322]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_322]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_322]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322]
2022-09-21 06:24:56,573 DEBUG {modelserviceDBCacheEventsWorker1} (DBEventsListener.java:507) debezium connector stopped
2022-09-21 06:24:56,573 DEBUG {SpringApplicationShutdownHook} (EmbeddedEngine.java:1060) Interrupting the embedded engine's thread Thread[modelserviceDBCacheEventsWorker1,5,main] (already interrupted: false)
2022-09-21 06:24:56,573 DEBUG {modelserviceDBCacheEventsWorker1} (DBEventsListener.java:161) debezium engine terminated
Any ideas what the problem is??
thx!
Related
When I tried to connect to Nifi UI using http://localhost:8080/nifi, i am getting below error
org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
java.net.BindException: Address already in use: bind
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:331)
at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:299)
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:235)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.server.Server.doStart(Server.java:398)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:935)
at org.apache.nifi.NiFi.<init>(NiFi.java:158)
at org.apache.nifi.NiFi.<init>(NiFi.java:72)
at org.apache.nifi.NiFi.main(NiFi.java:297)
2020-02-27 11:51:11,834 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
2020-02-27 11:51:11,836 INFO [Thread-1] o.eclipse.jetty.server.AbstractConnector Stopped ServerConnector#355ee205{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2020-02-27 11:51:11,837 INFO [Thread-1] org.eclipse.jetty.server.session node 0 Stopped scavenging
Can anyone suggest what is the cause of this issue?
Nifi version- 1.9.2,installed on windows machine
Here is the nifi status logs,
12:33:16.886 [main] DEBUG org.apache.nifi.bootstrap.NotificationServiceManager - Found 0 service elements
12:33:16.896 [main] INFO org.apache.nifi.bootstrap.NotificationServiceManager - Successfully loaded the following 0 services: []
12:33:16.897 [main] INFO org.apache.nifi.bootstrap.RunNiFi - Registered no Notification Services for Notification Type NIFI_STARTED
12:33:16.897 [main] INFO org.apache.nifi.bootstrap.RunNiFi - Registered no Notification Services for Notification Type NIFI_STOPPED
12:33:16.898 [main] INFO org.apache.nifi.bootstrap.RunNiFi - Registered no Notification Services for Notification Type NIFI_DIED
12:33:16.899 [main] DEBUG org.apache.nifi.bootstrap.Command - Status File:
12:33:16.900 [main] DEBUG org.apache.nifi.bootstrap.Command - Properties: {pid=9724}
Failed to determine if Process 9724 is running; assuming that it is not
12:33:16.902 [main] INFO org.apache.nifi.bootstrap.Command - Apache NiFi is not running
The port use by nifi is already used by another process.
you can change web server port in conf/nifi.properties
I have a Spring boot application in version 2.1.2.RELEASE. Once a month or few weeks application crashes without any specific information in logs.
Server where app is deploy is CentOS ver 7, app is launched by a systemd service.
Tried analysing heap dump, no luck, checked logs of postgres, tomcat etc. nothing there.
Stacktrace:
INFO ;read-5;523575402;2019-08-07 17:27:24,714;o.s.i.e.EventDrivenConsumer; Removing {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
INFO ;read-5;523575418;2019-08-07 17:27:24,730;o.s.i.c.PublishSubscribeChannel; Channel 'application.errorChannel' has 0 subscriber(s).
INFO ;read-5;523575418;2019-08-07 17:27:24,730;o.s.i.e.EventDrivenConsumer; stopped _org.springframework.integration.errorLogger
INFO ;read-5;523575502;2019-08-07 17:27:24,814;o.s.s.c.ThreadPoolTaskScheduler; Shutting down ExecutorService 'taskScheduler'
INFO ;read-5;523575541;2019-08-07 17:27:24,853;o.s.s.c.ThreadPoolTaskExecutor; Shutting down ExecutorService 'applicationTaskExecutor'
INFO ;read-5;523575572;2019-08-07 17:27:24,884;o.s.o.j.LocalContainerEntityManagerFactoryBean; Closing JPA EntityManagerFactory for persistence unit 'default'
INFO ;read-5;523575596;2019-08-07 17:27:24,908;c.z.h.HikariDataSource; HikariPool-1 - Shutdown initiated...
INFO ;read-5;523575631;2019-08-07 17:27:24,943;c.z.h.HikariDataSource; HikariPool-1 - Shutdown completed.
WARN ;exec-8;523575668;2019-08-07 17:27:24,980;c.z.h.p.ProxyConnection; HikariPool-1 - Connection org.postgresql.jdbc.PgConnection#6ff70ea8 marked as broken because of SQLSTATE(08006), ErrorCode(0)
org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:335)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:143)
at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:106)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:60)
at org.hibernate.loader.Loader.getResultSet(Loader.java:2173)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1936)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1898)
WARN ;exec-6;523575677;2019-08-07 17:27:24,989;c.z.h.p.ProxyConnection; HikariPool-1 - Connection org.postgresql.jdbc.PgConnection#71c6647 marked as broken because of SQLSTATE(08006), ErrorCode(0)
org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:335)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:143)
at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:106)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
WARN ;exec-8;523575721;2019-08-07 17:27:25,033;o.h.e.j.s.SqlExceptionHelper; SQL Error: 0, SQLState: 08006
WARN ;exec-6;523575721;2019-08-07 17:27:25,033;o.h.e.j.s.SqlExceptionHelper; SQL Error: 0, SQLState: 08006
ERROR;exec-8;523575721;2019-08-07 17:27:25,033;o.h.e.j.s.SqlExceptionHelper; An I/O error occurred while sending to the backend.
ERROR;exec-6;523575721;2019-08-07 17:27:25,033;o.h.e.j.s.SqlExceptionHelper; An I/O error occurred while sending to the backend.
INFO ;exec-6;523575885;2019-08-07 17:27:25,197;o.h.e.i.DefaultLoadEventListener; HHH000327: Error performing load command : org.hibernate.exception.JDBCConnectionException: could not extract ResultSet
org.hibernate.exception.JDBCConnectionException: could not extract ResultSet
at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:115)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
Has anyone got idea how can I what might be going on? Last crash was 3, 4 weeks before that.
That type of error happens when the database connection has been lost, which in turn can happen for a whole number of reasons, the most simple one being that it's being restarted after system updates or as part of a scheduled operation.
HikariCP should be able to recover according to the following link (provided that your HikariCP version is recent enough):
https://github.com/brettwooldridge/HikariCP/wiki/Rapid-Recovery
You may need to configure some properties in your pool. The full list of properties is available here:
https://github.com/brettwooldridge/HikariCP
I recommend you configure the following in HikariCP:
connectionTestQuery: will test the connection periodically, dropping dead connections. Use it in conjunction with socketTimeout in your PostgreSQL connection string
leakDetectionThreshold: will help with applications having connection leaks
I have created a spring boot application to publish the message to the Kafka queue. For that, I am using spring cloud stream and Kafka binder as dependencies. Problem is my application is continuously trying to connect to Kafka broker if it is down for 2 minutes because of the default configuration.
I have reduced that time using the below property and set it to 1000 ms and getting the timeout exception
spring.kafka.properties.request.timeout.ms:1000.
But still, my spring application is running after the exception. I want it to fail if Kafka broker is not available to connect to. I have tried one more property for that spring.kafka.admin.fail-fast=true but still, the application is running.
I have also tried to search for some properties of spring cloud stream and Kafka binder that I can set to fail my application if Kafka broker is not available but couldn't find anything related to that.
Please, help me with this.
Please see below for the log of exception.
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:104)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:274)
at org.springframework.cloud.stream.binder.kafka.provisioning.KafkaTopicProvisioner.createTopicAndPartitions(KafkaTopicProvisioner.java:351)
at org.springframework.cloud.stream.binder.kafka.provisioning.KafkaTopicProvisioner.createTopicIfNecessary(KafkaTopicProvisioner.java:325)
at org.springframework.cloud.stream.binder.kafka.provisioning.KafkaTopicProvisioner.createTopic(KafkaTopicProvisioner.java:302)
... 33 common frames omitted
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
2019-05-22 06:06:25 [main] DEBUG o.s.c.s.DefaultLifecycleProcessor - Successfully started bean 'outputBindingLifecycle'
2019-05-22 06:06:25 [main] DEBUG o.s.c.s.DefaultLifecycleProcessor - Starting beans in phase 2147482647
2019-05-22 06:06:25 [main] DEBUG o.s.c.s.binding.BindableProxyFactory - Binding inputs for :interface kafka.stream.RXXXStreams
2019-05-22 06:06:25 [main] DEBUG o.s.c.s.DefaultLifecycleProcessor - Successfully started bean 'inputBindingLifecycle'
2019-05-22 06:06:25 [main] DEBUG o.s.c.s.DefaultLifecycleProcessor - Starting beans in phase 2147483547
2019-05-22 06:06:25 [main] DEBUG o.s.c.s.DefaultLifecycleProcessor - Successfully started bean 'org.springframework.kafka.config.internalKafkaListenerEndpointRegistry'
2019-05-22 06:06:25 [main] DEBUG o.s.b.a.l.ConditionEvaluationReportLoggingListener -
Do you have spring-boot-web libraries as dependency? If that's the case, your application will not exit. A full log will be also very helpful.
When I tried to upgrade sonarqube 6.4 to 6.6 the server failed to start with following error in the web.log file.
2017.12.21 18:12:29 INFO web[][o.s.s.n.NotificationDaemon] Notification service started (delay 60 sec.)
2017.12.21 18:12:29 INFO web[][o.s.s.e.IndexerStartupTask] Indexing of type [rules/activeRule] ...
2017.12.21 18:12:31 INFO web[][o.s.s.e.IndexerStartupTask] Indexing of type [rules/activeRule] done | time=1480ms
2017.12.21 18:12:31 INFO web[][o.s.s.e.IndexerStartupTask] Indexing of types [rules/ruleExtension],[rules/rule] ...
2017.12.21 18:12:33 ERROR web[][o.s.s.p.Platform] Background initialization failed. Stopping SonarQube
java.lang.IllegalStateException: Unrecoverable indexation failures
at org.sonar.server.es.IndexingListener$1.onFinish(IndexingListener.java:39)
at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:117)
at org.sonar.server.rule.index.RuleIndexer.indexOnStartup(RuleIndexer.java:90)
at org.sonar.server.es.IndexerStartupTask.indexUninitializedTypes(IndexerStartupTask.java:68)
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
at org.sonar.server.es.IndexerStartupTask.execute(IndexerStartupTask.java:55)
at java.util.Optional.ifPresent(Optional.java:159)
at org.sonar.server.platform.platformlevel.PlatformLevelStartup$1.doPrivileged(PlatformLevelStartup.java:91)
at org.sonar.server.user.DoPrivileged.execute(DoPrivileged.java:45)
at org.sonar.server.platform.platformlevel.PlatformLevelStartup.start(PlatformLevelStartup.java:87)
at org.sonar.server.platform.Platform.executeStartupTasks(Platform.java:196)
at org.sonar.server.platform.Platform.access$400(Platform.java:46)
at org.sonar.server.platform.Platform$1.lambda$doRun$1(Platform.java:121)
at org.sonar.server.platform.Platform$AutoStarterRunnable.runIfNotAborted(Platform.java:371)
at org.sonar.server.platform.Platform$1.doRun(Platform.java:121)
at org.sonar.server.platform.Platform$AutoStarterRunnable.run(Platform.java:355)
at java.lang.Thread.run(Thread.java:748)
2017.12.21 18:12:33 INFO web[][o.s.p.StopWatcher] Stopping process
2017.12.21 18:12:33 INFO web[][o.s.s.n.NotificationDaemon] Notification service stopped
2017.12.21 18:12:33 INFO web[][o.s.s.n.NotificationDaemon] Notification service stopped
I was using postgreSQL for DB. The error happened after upgrade of sonarqube from the setup URL.
I am trying to connect to Phoenix via Squirrel client. I am receiving the following logs in the Squirrel logs. The logs suggests that the ClientConnection to zooperkeeper is established however it fails when a SQLClient Connection is being established with a Timeout exception.
I have copied the phoenix client jar into the lib directory of Squirrel and the driver is registered succesfully. Also when I run the SQLLine.py utility in the localhost it loads the SQL commandline to Phoenix succesfully and I can run the commands. Added the phoenix core jars to the $HBASE_HOME/lib folders as well.
2015-06-15 12:48:53,766 [pool-7-thread-1] INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Process identifier=hconnection-0x776a1002 connecting to ZooKeeper ensemble=10.58.126.245:2181
2015-06-15 12:48:53,766 [pool-7-thread-1] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=10.58.126.245:2181 sessionTimeout=90000 watcher=hconnection-0x776a10020x0, quorum=10.58.126.245:2181, baseZNode=/hbase
2015-06-15 12:48:58,287 [pool-7-thread-1-SendThread(10.58.126.245:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.58.126.245/10.58.126.245:2181. Will not attempt to authenticate using SASL (unknown error)
2015-06-15 12:48:58,301 [pool-7-thread-1-SendThread(10.58.126.245:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to 10.58.126.245/10.58.126.245:2181, initiating session
2015-06-15 12:48:58,314 [pool-7-thread-1-SendThread(10.58.126.245:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server 10.58.126.245/10.58.126.245:2181, sessionid = 0x14df5b87b120040, negotiated timeout = 90000
2015-06-15 12:49:58,100 [pool-7-thread-1] INFO org.apache.hadoop.hbase.client.RpcRetryingCaller - Call exception, tries=10, retries=35, started=59774 ms ago, cancelled=false, msg=
2015-06-15 12:50:20,456 [pool-7-thread-1] INFO org.apache.hadoop.hbase.client.RpcRetryingCaller - Call exception, tries=11, retries=35, started=82130 ms ago, cancelled=false, msg=
2015-06-15 12:50:36,114 [AWT-EventQueue-1] ERROR net.sourceforge.squirrel_sql.client.gui.db.ConnectToAliasCallBack - Unexpected Error occurred attempting to open an SQL connection.
java.util.concurrent.TimeoutException
at java.util.concurrent.FutureTask.get(FutureTask.java:201)
at net.sourceforge.squirrel_sql.client.mainframe.action.OpenConnectionCommand.awaitConnection(OpenConnectionCommand.java:132)
at net.sourceforge.squirrel_sql.client.mainframe.action.OpenConnectionCommand.access$100(OpenConnectionCommand.java:45)
at net.sourceforge.squirrel_sql.client.mainframe.action.OpenConnectionCommand$2.run(OpenConnectionCommand.java:115)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
I have the same problem, don't find the solution yet, but I managed to use the "thin" client instead.
Start queryserver https://phoenix.apache.org/server.html should listen on port 8765
Copy JAR phoenix-4.6.0-HBase-1.1-thin-client to Squirel lib folder
Create new driver, the class name is "org.apache.phoenix.queryserver.client.Driver"
Connect with this driver (my URI: jdbc:phoenix:thin:url=http://docker:8765)