I am using impala2.12.0-cdh5.16.1 and connecting to impala with impala_jdbc_2.6.4.1005. Normally it runs very well, but when I run distcp (which cost the Cluster Network IO and HDFS IO), the java program may throw errors.
2019/02/28 12:54:26 531873 ERROR run.QihooStatusTask(run:88) - [Cloudera][ImpalaJDBCDriver](700100) Connection timeout expired. Details: java.net.ConnectException: Connection timed out.
java.sql.SQLException: [Cloudera][ImpalaJDBCDriver](700100) Connection timeout expired. Details: java.net.ConnectException: Connection timed out.
at com.cloudera.impala.hivecommon.core.HiveJDBCCommonConnection.handleException(Unknown Source)
at com.cloudera.impala.jdbc.core.LoginTimeoutConnection.connect(Unknown Source)
at com.cloudera.impala.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
at com.cloudera.impala.jdbc.common.AbstractDriver.connect(Unknown Source)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
The full error Message is in the picture:
I tried to add DriverManager.setLoginTimeout(120) to the program,but the error still exists.
I think it may be happen because the Cluster Network IO is very high and there may be a parameter which can add the value of timeout so the error will disappear.
So, any suggestion about this? Thx.
Related
Almost all the client connections are directed to one server and the application fails to connect after some time with IOException. Seems load balancing is not working properly. We are forced to restart the listener at this point.
jdbc URL : jdbc:oracle:thin:#(DESCRIPTION=(FAILOVER=ON)(LOAD_BALANCE=yes)(ADDRESS=(PROTOCOL=TCP)(HOST=scan-ip)(PORT=1538))(CONNECT_DATA=(service_name=production)(SERVER=DEDICATED)))
Exception:
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Io exception: Connection reset)
at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549)
at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
... 60 more
Caused by: java.sql.SQLException: Io exception: Connection reset
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:387)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:414)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)
at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
at org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556)
at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545)
... 64 more
Version:
11.2.0.4.0 - 64bit
Just to make sure - when you say "all connection are directed to one node" - do you mean that the session are created on one node or that the scan listener taking care of the connection is only being used on one node?
Check and see if you SCAN address contain 3 IP addresses (nslookup scan-ip), make sure that your SCAN listeners are being distributed across nodes (srvctl status scan and srvctl status scan_listener). You can also check to see that the database service you're trying to use (production) is being registered on all appropriate listeners (check database parameter "remote_listener" to make sure where the service is being registered to).
I am trying to establish a jdbc-oci connection to an oracle database on a coldfusion v 10 std server.
The reason for employing jdbc oci is the necessity to encrypt data flowing between the app server and the database (by using common settings in sqlnet.ora). However, no matter what syntax we try, neither a service
based nor a tnsnames based jdbc url will work. Immediately after saving (and automatically testing) the connection, coldfusion errors out with "java.sql.SQLException: Timed out trying to establish connection".
It looks weird that a timeout may have happened since the error will always be thrown immediately. Therefor I suppose there is another root cause for failing to establish the connection.
Any ideas out there?
Please note that we simulated connection establishment with a console application aside coldfusion but using the same jvm and jdbc driver successfully. it does therefore not seem to be a jvm or driver or oci.dll
related issue.
data source settings :
Driver Class : oracle.jdbc.driver.OracleDriver
JDBC URL (using tnsnames.ora) : jdbc:oracle:oci:#<theName>
JDBC URL (using ip, port, service) : jdbc:oracle:oci:#//<IP>:1521/<SID>
error message :
Connection verification failed for data source: theName
java.sql.SQLException: Timed out trying to establish connection
The root cause was that: java.sql.SQLException: Timed out trying to establish connection
environment :
ojdbc6 driver lives in a directory known to coldfusion
oci.dll lives in a directory known to the system path variable, verified by sysinternals processexplorer on the coldfusion service
for the JDBC URL (using tnsnames.ora), the directory containing tnsnames.ora is known as a system variable TNS_ADMIN, verified by sysinternals processexplorer on the coldfusion service
along with tnsnames.ora there is a sqlnet.ora file comprising settings to switch on data encryption
Stacktrace:
A non-SQL error occurred while requesting a connection from "datasource-name here".
Timed out trying to establish connection
Exception in thread "Thread-2120" java.lang.UnsatisfiedLinkError: no ocijdbc11 in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860)
at java.lang.Runtime.loadLibrary0(Runtime.java:845)
at java.lang.System.loadLibrary(System.java:1084)
at oracle.jdbc.driver.T2CConnection$1.run(T2CConnection.java:3560)
at java.security.AccessController.doPrivileged(Native Method)
at oracle.jdbc.driver.T2CConnection.loadNativeLibrary(T2CConnection.java:3556)
at oracle.jdbc.driver.T2CConnection.logon(T2CConnection.java:269)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:553)
at oracle.jdbc.driver.T2CConnection.<init>(T2CConnection.java:165)
at oracle.jdbc.driver.T2CDriverExtension.getConnection(T2CDriverExtension.java:53)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:528)
at coldfusion.server.j2ee.sql.pool.JDBCPool.createPhysicalConnection(JDBCPool.java:614)
at coldfusion.server.j2ee.sql.pool.ConnectionRunner$RunnableConnection.run(ConnectionRunner.java:67)
at java.lang.Thread.run(Thread.java:722)
A non-SQL error occurred while requesting a connection from "datasource-name here".
Timed out trying to establish connection
[Fatal Error] :2662:4: The element type "view" must be terminated by the matching end-tag "</view>".
Adding of the Path to the oracle instant client at the configuration file cf-directory\cfusion\bin\jvm.config has solved the problem.
I deployed a long running Storm topology. After several hours running, the whole topology went down. I checked worker logs, and found these logs . As it says, zookeeper client session timed out and it caused reconnection. I suspect it was relate to my broken topology. Now I try to find out what can cause clients timeout.
2016-02-29T10:34:12.386+0800 o.a.s.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 23789ms for sessionid 0x252f862028c0083, closing socket connection and attempting reconnect
2016-02-29T10:34:12.986+0800 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: SUSPENDED
2016-02-29T10:34:13.059+0800 b.s.cluster [WARN] Received event :disconnected::none: with disconnected Zookeeper.
2016-02-29T10:34:13.197+0800 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server zk-3.cloud.mos/172.16.13.147:2181. Will not attempt to authenticate using SASL (unknown error)
2016-02-29T10:34:13.241+0800 o.a.s.z.ClientCnxn [WARN] Session 0x252f862028c0083 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_31]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) ~[na:1.8.0_31]
at org.apache.storm.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) ~[storm-core-0.9.6.jar:0.9.6]
at org.apache.storm.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) ~[storm-core-0.9.6.jar:0.9.6]
Your client can no longer talk to the ZooKeeper server. The first thing that happened was there was no answer to the heartbeats within the negotiated session timeout:
2016-02-29T10:34:12.386+0800 o.a.s.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 23789ms for sessionid 0x252f862028c0083, closing socket connection and attempting reconnect
Then when it tried to reconnect, it got a connection refused:
2016-02-29T10:34:13.241+0800 o.a.s.z.ClientCnxn [WARN] Session 0x252f862028c0083 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
This means either your ZooKeeper server:
Is not reachable (network connection down)
Is dead (so nothing is listening on the socket)
Is GCing itself to death and cannot communicate (although that might have issued a connection timeout error, I'm not sure)
To tell more you will need to check the ZooKeeper server logs on your (Hadoop?) cluster.
Its worked for me by increasing the connection timeout in server.properties:
zookeeper.connection.timeout.ms=60000
One way that this can happen is if you start zookeeper, then break in the terminal, then try to start kafka.
In order to use kafka, you really should use 3 terminal windows (or 3 PuTTY sessions if you are SSHing into your instance from Windows)
First Session for Zookeeper server.
Second Session for Kafka server.
Third Session for running Kafka commands to do things like create topics.
I have started Kafka in cluster mode with 3 zookeeper server and 3 Kafka server. All zookeeper server started successfully but while starting Kafka server its get disconnected stating "fatal error during Kafka server startup. prepare to shutdown (kafka.server.kafkaserver)". while investigation, I found that Kafka server get disconnected every time after 18 seconds[which is zookeeper.connection.timeout.ms = 18000 default value] so I updated the same and issue get resolved.
always use 2181 as port number for zookeeper connection until you haven't configured your zookeeper !!!
This production issue is now taking over my head. I have configured to use Bitronix as transaction manager with the jBPM5 tool. It works fine from a long time but sudden started breaking with below exception.
Exception stacktrace
Caused by: java.sql.SQLException: unable to get a connection from pool of a PoolingDataSource containing an XAPool of resource jdbc/jbpm-ds with 1 connection(s) (1 still available)
at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:262)
at org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection(InjectedDataSourceConnectionProvider.java:71)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
... 78 more
Caused by: bitronix.tm.internal.BitronixRuntimeException: cannot get valid connection from an XAPool of resource jdbc/jbpm-ds with 1 connection(s) (1 still available) after trying for 30s
at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:160)
at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:91)
at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:258)
... 80 more
Caused by: java.sql.SQLRecoverableException: IO Error: Connection timed out
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:899)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1175)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1296)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3613)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3657)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1495)
at bitronix.tm.resource.jdbc.JdbcPooledConnection.testConnection(JdbcPooledConnection.java:215)
at bitronix.tm.resource.jdbc.JdbcPooledConnection.getConnectionHandle(JdbcPooledConnection.java:299)
at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:130)
... 82 more
Caused by: java.net.SocketException: Connection timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at oracle.net.ns.Packet.receive(Packet.java:308)
at oracle.net.ns.DataPacket.receive(DataPacket.java:106)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:324)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:268)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:190)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:107)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:350)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:227)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:208)
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:886)
... 90 more
Strange thing is that application has sufficient number of XA connections are available in XAPool. It takes the connection, tried to connect, unable to connect and just discard the connection and take new connection to process.
SEVERE: unable to get a connection from pool of a PoolingDataSource containing an XAPool of resource jdbc/jbpm-ds with 9 connection(s) (9 still available)
SEVERE: unable to get a connection from pool of a PoolingDataSource containing an XAPool of resource jdbc/jbpm-ds with 8 connection(s) (8 still available)
SEVERE: unable to get a connection from pool of a PoolingDataSource containing an XAPool of resource jdbc/jbpm-ds with 7 connection(s) (7 still available)
Could anybody please shed some light to recover from this issue?
Update1:
After some try I am able to reproduce the same issue in local. Once database connection is resetted or network connectivity lost between my application & database, pool connections there in PoolingDataSource (provided by Bitronix) pool becomes invalid. My query is Why it is unable auto-recover? Is there any why through which we can auto-recover or establish the new connections?
I am using btm version 2.1.4.
Thanks,
Mayank
I was having the same issue but it was due to user permissions. My user does not have enough permissions so I was getting this issue. After that, I ran these 3 below-mentioned commands to create a new user and it works for me. I hope, it helps someone.
CREATE USER 'newuser'#'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON * . * TO 'newuser'#'localhost';
FLUSH PRIVILEGES;
I am facing a very strange problem.
I am using my application with RMI for 8 month in production but since a new (big) update I dont know what is wrong.
It is a server where 3 clients connect.
And when I am using only client with a certain method I can not find anything wrong especially because all other clients work perfectly.
Is there maybe with RMI any local "cache" which should be deleted otherwise I can't tell.
Here is the output of the client's console:
C:\Users\Claudia\Desktop>java -jar CLIENT_erp_auer_client_v0_8ASASADSD.jar
java.rmi.MarshalException: error marshalling arguments; nested exception is:
java.net.SocketException: Software caused connection abort: socket write error
at sun.rmi.server.UnicastRef.invoke(Unknown Source)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(Unknown Sou
rce)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(Unknown Source)
at $Proxy7.update(Unknown Source)
at client.entity_types.LieferscheinpositionService.aendereLieferscheinposition(
LieferscheinpositionService.java:60)
at client.lieferscheinverwaltung.TreeTableNEULieferschein.addLieferscheinpositi
onen(TreeTableNEULieferschein.java:912)
at client.lieferscheinverwaltung.TreeTableNEULieferschein.<init>(TreeTableNEULi
eferschein.java:204)
at client.lieferscheinverwaltung.LieferscheinErstellung.getTreeTableNeuLiefersc
hein(LieferscheinErstellung.java:916)
at client.lieferscheinverwaltung.LieferscheinErstellung.<init>(LieferscheinErst
ellung.java:773)
at client.lieferscheinverwaltung.LieferscheinManagement.openLieferschein(Liefer
scheinManagement.java:1055)
at client.lieferscheinverwaltung.LieferscheinManagement$9.mouseClicked(Liefersc
Thank you.