Why Impala not working on hbase table? - hadoop

I create an external table B of hbase table A using hive. I can successfully access the data of B.Then I followed the official guide to type in Imapla Shell:
invalidate metadata B;
And then I query this external table B in Impala Shell:
select * from B limit 4;
but it outputs:
ERROR: RuntimeException: couldn't retrieve HBase table (mv_p2pusers) info:
Enable/Disable failed
Here are some logs related:
11:13:58.937 AM INFO jni-util.cc:177
java.lang.RuntimeException: couldn't retrieve HBase table (mv_p2pusers) info:
Enable/Disable failed
at com.cloudera.impala.planner.HBaseScanNode.computeScanRangeLocations(HBaseScanNode.java:300)
at com.cloudera.impala.planner.HBaseScanNode.init(HBaseScanNode.java:125)
at com.cloudera.impala.planner.SingleNodePlanner.createScanNode(SingleNodePlanner.java:891)
at com.cloudera.impala.planner.SingleNodePlanner.createTableRefNode(SingleNodePlanner.java:1082)
at com.cloudera.impala.planner.SingleNodePlanner.createSelectPlan(SingleNodePlanner.java:526)
at com.cloudera.impala.planner.SingleNodePlanner.createQueryPlan(SingleNodePlanner.java:151)
at com.cloudera.impala.planner.SingleNodePlanner.createSingleNodePlan(SingleNodePlanner.java:117)
at com.cloudera.impala.planner.Planner.createPlan(Planner.java:47)
at com.cloudera.impala.service.Frontend.createExecRequest(Frontend.java:842)
at com.cloudera.impala.service.JniFrontend.createExecRequest(JniFrontend.java:146)
11:13:58.939 AM INFO status.cc:114
RuntimeException: couldn't retrieve HBase table (mv_p2pusers) info:
Enable/Disable failed
# 0x78b793 (unknown)
# 0xa68275 (unknown)
# 0x9802c6 (unknown)
# 0x99db78 (unknown)
# 0x99e6e4 (unknown)
# 0x9d50cb (unknown)
# 0xb33687 (unknown)
# 0xb29054 (unknown)
# 0x9ac52b (unknown)
# 0x1571c39 (unknown)
# 0x155d9cf (unknown)
# 0x155f914 (unknown)
# 0x92d363 (unknown)
# 0x92daca (unknown)
# 0xaa4faa (unknown)
# 0xaa7130 (unknown)
# 0xca79b3 (unknown)
# 0x386be079d1 (unknown)
# 0x386bae8b6d (unknown)
11:13:58.940 AM INFO impala-server.cc:824
UnregisterQuery(): query_id=d4269ff898eb4e7:1866144af0d14a7
11:13:58.940 AM INFO impala-server.cc:893
Cancel(): query_id=d4269ff898eb4e7:1866144af0d14a7
11:13:59.935 AM INFO ClientCnxn.java:975
Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
11:13:59.935 AM WARN ClientCnxn.java:1102
Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
Java exception follows:
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
11:14:01.036 AM INFO ClientCnxn.java:975
Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
11:14:01.037 AM WARN ClientCnxn.java:1102
Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
Java exception follows:
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
11:14:02.138 AM INFO ClientCnxn.java:975
Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
11:14:02.138 AM WARN ClientCnxn.java:1102
Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
Java exception follows:
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
11:14:02.199 AM INFO impala-hs2-server.cc:795
GetSchemas(): request=TGetSchemasReq {
01: sessionHandle (struct) = TSessionHandle {
01: sessionId (struct) = THandleIdentifier {
01: guid (string) = "\xf8\xb9n\xe4\xb4\xf6N\xef\xad)9W.\x92#Y",
02: secret (string) = "\xc0?\xc7\xd9\x930C\x9b\xb5\xf6K\x8em\xcb\xf8\xe4",
},
},
}
11:14:02.203 AM INFO MetadataOp.java:414
Returning 19 schemas
It seems the hbase table B is either enabled nor disabled,very strange.I googled around,Is this related to the hbase security issues or the impala version problem?
Did anybody encountered the same problems?How to solve this?Thanks in advance.

Enable HBase service from Impala Configuration.
You can do it from Cloudera manager, Imapala->configuration
search from "Hbase" and enable the service.
PFA.

Related

Cant connect to MQ Spring boot SSL

I am trying to establish a connection to the MQ service
For SSL connection I use the commands
-Djavax.net.ssl.trustStore=/opt/app/key.jks"
-Djavax.net.ssl.trustStorePassword=111111
Appliction.properties config
ibm.mq.connName=10.20.31.25(1414)
ibm.mq.channel=OIV.CHAN
ibm.mq.queueManager=OIV
ibm.mq.user=TEST
ibm.mq.password=passw0rd
ibm.mq.ssl-cipher-spec=TLS_RSA_WITH_AES_256_CBC_SHA
When the application starts, everything is ok
INFO IbmJmsConfiguration - Initializing SSL context:
protocol=TLSv1.2, keyStore=null, trustStore=/opt/app/key.jks
INFO IbmJmsConfiguration - SSL context initialized:
keyManagers item(s) = 0, trustManagers item(s) = 1
But when making a request to MQ, I get an error
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2393;AMQ9771: SSL handshake failed. [1=java.lang.IllegalArgumentException[Unsupported ciphersuite SSL_RSA_WITH_AES_256_CBC_SHA256]
text error
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2393;AMQ9771: SSL handshake
failed. [1=java.lang.IllegalArgumentException[Unsupported ciphersuite
SSL_RSA_WITH_AES_256_CBC_SHA],3=10.90.51.15/10.90.50.15:1414
(10.96.51.15),4=SSLSocket.createSocket,5=default]
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.makeSocketSecure(RemoteTCPConnection.java:2360)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.bindAndConnectSocket(RemoteTCPConnection.java:816)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.protocolConnect(RemoteTCPConnection.java:1381)
at com.ibm.mq.jmqi.remote.impl.RemoteConnection.connect(RemoteConnection.java:976)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getNewConnection(RemoteConnectionSpecification.java:553)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSessionFromNewConnection(RemoteConnectionSpecification.java:233)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSession(RemoteConnectionSpecification.java:141)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionPool.getSession(RemoteConnectionPool.java:127)
at com.ibm.mq.jmqi.remote.api.RemoteFAP$Connector.jmqiConnect(RemoteFAP.java:13302)
... 74 common frames omitted
Caused by: java.lang.IllegalArgumentException: Unsupported ciphersuite SSL_RSA_WITH_AES_256_CBC_SHA
at sun.security.ssl.CipherSuite.valueOf(CipherSuite.java:228)
at sun.security.ssl.CipherSuiteList.<init>(CipherSuiteList.java:79)
at sun.security.ssl.SSLSocketImpl.setEnabledCipherSuites(SSLSocketImpl.java:2491)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.makeSocketSecure(RemoteTCPConnection.java:2351)
... 82 common frames omitted

elasticsearch nest SniffingConnectionPool not working

I'm using Nest.ElasticClient to connect to Elasticsearch cluster. The cluster is located in Azure VM with just one node.
the cluster is accessible outside the vm by url : http://xxxx.cloudapp.net:9200 and also accessible by ElasticClient if not using SniffingConnectionPool. But not accessible by ElasticClient if using SniffingConnectionPool.
Here is the network config
network.host: [_local_, _site_]
Below is the source code I'm using to get client and check index exists.
var pool = new SniffingConnectionPool(urls.Select(url => new Uri(url)));
ConnectionSettings config = new ConnectionSettings(pool) ;
client = new Nest.ElasticClient(config);
IExistsResponse indexExistsResponse = client.IndexExists(indexName);
The debug info message when I try to use the client to check whether a Index exists, the hostnanme and ip address is modified:
Invalid NEST response built from a unsuccessful low level call on HEAD: /globalleads
# Audit trail of this API call:
- SniffOnStartup: Took: 00:00:00.9846171
- SniffSuccess: Node: http://xxxx.cloudapp.net:9200/ Took: 00:00:00.9595496
- PingFailure: Node: http://10.85.xxx.xx:9200/ Exception: PipelineException Took: 00:00:21.4154660
- SniffOnFail: Took: 00:00:21.1967809
- SniffFailure: Node: http://10.85.xxx.xx:9200/ Exception: PipelineException Took: 00:00:21.1787333
# OriginalException: Elasticsearch.Net.ElasticsearchClientException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> Elasticsearch.Net.PipelineException: Failed sniffing cluster state. ---> System.AggregateException: One or more errors occurred. ---> Elasticsearch.Net.PipelineException: An error occurred trying to establish a connection with the specified node.
at Elasticsearch.Net.RequestPipeline.Sniff() in D:\dev\git\elasticsearch-net-2.x\src\Elasticsearch.Net\Transport\Pipeline\RequestPipeline.cs:line 326
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at Elasticsearch.Net.RequestPipeline.Sniff() in D:\dev\git\elasticsearch-net-2.x\src\Elasticsearch.Net\Transport\Pipeline\RequestPipeline.cs:line 341
at Elasticsearch.Net.RequestPipeline.SniffOnConnectionFailure() in D:\dev\git\elasticsearch-net-2.x\src\Elasticsearch.Net\Transport\Pipeline\RequestPipeline.cs:line 301
at Elasticsearch.Net.Transport`1.Ping(IRequestPipeline pipeline, Node node) in D:\dev\git\elasticsearch-net-2.x\src\Elasticsearch.Net\Transport\Transport.cs:line 179
at Elasticsearch.Net.Transport`1.Request[TReturn](HttpMethod method, String path, PostData`1 data, IRequestParameters requestParameters) in D:\dev\git\elasticsearch-net-2.x\src\Elasticsearch.Net\Transport\Transport.cs:line 68
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
# Audit exception in step 2 PingFailure:
Elasticsearch.Net.PipelineException: An error occurred trying to establish a connection with the specified node.
at Elasticsearch.Net.RequestPipeline.Ping(Node node) in D:\dev\git\elasticsearch-net-2.x\src\Elasticsearch.Net\Transport\Pipeline\RequestPipeline.cs:line 248
# Audit exception in step 4 SniffFailure:
Elasticsearch.Net.PipelineException: An error occurred trying to establish a connection with the specified node.
at Elasticsearch.Net.RequestPipeline.Sniff() in D:\dev\git\elasticsearch-net-2.x\src\Elasticsearch.Net\Transport\Pipeline\RequestPipeline.cs:line 326
# Request:
<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>
# Response:
<Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>

SQL error 17002 : Physical DB connection is invalid

I am trying to connect to oracle db(installed in remote machine) from my java application. but, i am getting sql error code 17022. pls let me know how to solve this issue.
Full stack trace
MSG: Connecting to jdbc:oracle:thin:#(description=(address_list=(address=(protocol=tcp)(host=dlocdb04)(port=50000))(address=(protocol=tcp)(host=dlocdb04)(port=50000)))(connect_data=(service_name=os02rtdb01svc.world)(server=dedicated))) with Oracle JDBC driver version: 9.2.0.1.0 [cumulative retries=0]
USER_ID: UNKNOWN
level: WARN
date: 2015-09-21 10:51:45,563
category: com.retek.merch.utils.ConnectionPool
MSG: [SQL Error code: 17002, State: null] Physical DB connection is invalid. Restarting pool due to the following error.
USER_ID: UNKNOWN
java.sql.SQLException: Io exception: Connection refused(DESCRIPTION=(ERR=1153)(VSNNUM=0)(ERROR_STACK=(ERROR=(CODE=1153)(EMFI=4)(ARGS='(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dlocdb0401)(PORT=27320))(CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))null))'))(ERROR=(CODE=303)(EMFI=1))))
at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:134)
at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:179)
at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:333)
at oracle.jdbc.driver.OracleConnection.<init>(OracleConnection.java:404)
at oracle.jdbc.driver.OracleDriver.getConnectionInstance(OracleDriver.java:468)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:314)
at java.sql.DriverManager.getConnection(DriverManager.java:512)
at java.sql.DriverManager.getConnection(DriverManager.java:140)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:169)
at oracle.jdbc.pool.OracleConnectionPoolDataSource.getPhysicalConnection(OracleConnectionPoolDataSource.java:149)
at oracle.jdbc.pool.OracleConnectionPoolDataSource.getPooledConnection(OracleConnectionPoolDataSource.java:95)
at oracle.jdbc.pool.OracleConnectionPoolDataSource.getPooledConnection(OracleConnectionPoolDataSource.java:63)
at oracle.jdbc.pool.OracleConnectionCacheImpl.getNewPoolOrXAConnection(OracleConnectionCacheImpl.java:547)
at oracle.jdbc.pool.OracleConnectionCacheImpl.getPooledConnection(OracleConnectionCacheImpl.java:404)
at oracle.jdbc.pool.OracleConnectionCacheImpl.getConnection(OracleConnectionCacheImpl.java:298)
at oracle.jdbc.pool.OracleConnectionCacheImpl.getConnection(OracleConnectionCacheImpl.java:268)
at com.retek.merch.utils.ConnectionPool.get(ConnectionPool.java:346)
at com.retek.merch.utils.TransactionManager.start(TransactionManager.java:59)
at com.retek.reim.merch.utils.ReIMTransactionManager.start(ReIMTransactionManager.java:49)
at com.retek.reim.manager.LoginManager.login(LoginManager.java:72)
at com.retek.reim.ui.login.LoginAction.perform(LoginAction.java:47)
at org.apache.struts.action.ActionServlet.processActionPerform(ActionServlet.java:1786)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1585)
at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:509)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:638)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:720)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:199)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:145)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:596)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:433)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:955)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:139)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:596)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:433)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:955)
at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2460)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:133)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:596)
at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:119)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:594)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:116)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:594)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:433)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:955)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:127)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:596)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:433)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:955)
at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:157)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:873)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:534)

java.io.IOException: Job status not available about hive

When I use hive with select * from table_name;, it works.
When I use select t.a from table_name t OR select * from table_name where ..., the following error happens :
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1414949555870_118360, Tracking URL = N/A
Kill Command = /usr/local/hadoop-2.5.1/bin/hadoop job -kill job_1414949555870_118360
java.io.IOException: Job status not available
at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:322)
at org.apache.hadoop.mapreduce.Job.getJobState(Job.java:347)
at org.apache.hadoop.mapred.JobClient$NetworkedJob.getJobState(JobClient.java:295)
at org.apache.hadoop.hive.shims.HadoopShimsSecure.isJobPreparing(HadoopShimsSecure.java:104)
at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:242)
at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:541)
at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:431)
at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:136)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1485)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1263)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1091)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:921)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:422)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:790)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:623)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Ended Job = job_1414949555870_118360 with exception java.io.IOException(Job status not available )
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask`
So, what's wrong about hive?
DEBUG info as follow, maybe useful!! Please help me!!!
15/04/14 16:53:48 DEBUG ipc.Client: getting client out of cache: org.apache.hadoop.ipc.Client#60cb201b
15/04/14 16:53:48 DEBUG mapred.ClientServiceDelegate: Failed to contact AM/History for job job_1414949555870_118441 retrying..
java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: "slave109":43759; java.net.UnknownHostException; For more details see: http://wiki.apache.org/hadoop/UnknownHost

Spring-XD Curator Connection Timeout

In Spring-XD the Curator Connection times out:
WARN ConnectionStateManager-0 curator.ConnectionState - Connection
attempt unsuccessful after 63021 (greater than max timeout of 60000).
Resetting connection and trying again with a new connection.
Curator tries to re-establish the connection, but fails. Please check the logs below. Has anyone faced similar issue? Please let me know if you know of any ways to resolve the issue or if you know of any workarounds.
Also the default Curator connection time out is 60000. Is there a way to increase it? Does spring-xd expose a property which can be set?
2014-12-10 01:24:41,003 WARN ConnectionStateManager-0
server.ContainerRegistrar - >>> disconnected container:
1c8a234d-4b8d-4d65-b374-xxxxe8619 2014-12-10 01:24:41,004 INFO
DeploymentsPathChildrenCache-0 server.ContainerRegistrar - Path cache
event: null, type: CONNECTION_SUSPENDED 2014-12-10 01:24:41,005 INFO
ConnectionStateManager-0 server.ContainerRegistrar - Undeploying
module [ModuleDescriptor#350920b1 moduleName = 'rabbit', moduleLabel =
'rabbit', group = 'xxx-ingestion-2', sourceChannelName = [null],
sinkChannelName = [null], sinkChannelName = [null], index = 0, type =
source, parameters = map['vhost' -> 'xxx_virtual_host', 'requeue' ->
'false', 'outputType' -> 'text/plain', 'queues' -> 'xx.xxx.queue',
'addresses' -> 'xxxmq.xx.xxxx.com'], children = list[[empty]]]
2014-12-10 01:24:46,022 ERROR pool-22-thread-1
connection.CachingConnectionFactory - Channel shutdown: clean
connection shutdown; protocol method:
method<connection.close>(reply-code=200, reply-text=OK, class-id=0, method-id=0)
2014-12-10 01:24:56,007 **ERROR CuratorFramework-0
curator.ConnectionState - Connection timed out for connection string
(514.xx.93.xxx:2181,504.58.xxx.xx:2181) and timeout (15000) / elapsed**
(15004) org.apache.curator.CuratorConnectionLossException:
KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:198)
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88)
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:793)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:779)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$400(CuratorFrameworkImpl.java:58)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:265)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744) 2014-12-10 01:24:56,161
ERROR main-EventThread curator.ConnectionState - Connection timed out
for connection string (514.xx.93.xxx:2181,504.58.xxx.xx:2181) and
timeout (15000) / elapsed (15159)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode =
ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:198)
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88)
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:474)
at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:302)
at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:291)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:287)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
at org.springframework.xd.dirt.server.ContainerRegistrar$StreamModuleWatcher.process(ContainerRegistrar.java:744)
at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
2014-12-10 01:25:03,014 ERROR CuratorFramework-0 imps.CuratorFrameworkImpl - **Background retry gave up**
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode =
ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:198)
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88)
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:793)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:779)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$400(CuratorFrameworkImpl.java:58)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:265)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Is this reproducible? Are you running in clustered or single node mode?
The Curator connection timeout (in milliseconds) can be set via system property curator-default-connection-timeout.

Resources