Cassandra 2.06, Oracle Java version "1.7.0_51", Linux Mint 16
I have a cassandra keyspace with about 12 tables that are all the same.
If I load 100,000 rows or so into a couple of those tables in Cassandra, it works fine.
If I load a larger dataset, after a while one of the tables won't do lookups any more (not always the same one).
SELECT recv_time,symbol from table6 where mid='S-AUR01-20140324A-1221';
results in "Request did not complete within rpc_timeout."
where "mid" is the primary key (varchar). If I look at the logs, it has an EOFException ... presumably it's running out of some resource (it's definitely not out of disk space)
ERROR [ReadStage:110] 2014-04-03 12:39:47,018 CassandraDaemon.java (line 196) Exception in thread Thread[ReadStage:110,5,main]
java.io.IOError: java.io.EOFException
at org.apache.cassandra.db.Column$1.computeNext(Column.java:79)
at org.apache.cassandra.db.Column$1.computeNext(Column.java:64)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:88)
at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:37)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82)
at org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:157)
at org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:140)
at org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:200)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:185)
at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1551)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1380)
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:327)
at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1341)
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readFully(Unknown Source)
at java.io.RandomAccessFile.readFully(Unknown Source)
at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:348)
at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392)
at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355)
at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:110)
at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:85)
at org.apache.cassandra.db.Column$1.computeNext(Column.java:75)
... 28 more
It may not be a perfect solution, but may make sense to try
disk_access_mode: standard
This issue seems to be the same thing
https://issues.apache.org/jira/browse/CASSANDRA-6525
(I was not sure where to set disk_access_mode, but this post says you can add it to cassandra.yaml http://mail-archives.apache.org/mod_mbox/cassandra-user/201301.mbox/%3CCA+VSrLouEKurUZ84eRseDNGjuyhC56r6Uk5moHsRD_2ScBs+6g#mail.gmail.com%3E )
Related
In a psuedo distribute hadoop installation ( hbase, yarn, hdfs ), at one point all HBase's tables disappeared, how can i fix them?
In hbase shell: "list" returns 0 row(s)
Creating a table among those dissapeared named transactions:
Api Error: org.apache.hadoop.hbase.TableExistsException: transactions
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.convertResult(HBaseAdmin.java:4352)
at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4310)
at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4244)
at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:649)
at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:579)
at org.apache.hadoop.hbase.thrift.ThriftServerRunner$HBaseHandler.createTable(ThriftServerRunner.java:1192)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hbase.thrift.HbaseHandlerMetricsProxy.invoke(HbaseHandlerMetricsProxy.java:67)
at com.sun.proxy.$Proxy10.createTable(Unknown Source)
at org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$createTable.getResult(Hbase.java:4018)
at org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$createTable.getResult(Hbase.java:4002)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hadoop.hbase.thrift.TBoundedThreadPoolServer$ClientConnnection.run(TBoundedThreadPoolServer.java:289)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.TableExistsException): transactions
at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.prepareCreate(CreateTableProcedure.java:300)
at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:107)
at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:58)
at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:107)
at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:427)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:999)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:803)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:756)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$200(ProcedureExecutor.java:75)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$1.run(ProcedureExecutor.java:441)
look like problem with the zookeeper.
table is not present in the hbase but actually zookeeper keep all the table information i.e. metadata about table.
So you have run zookeeper cli and remove the /hbase directory from the zookeeper by using this command
rmr /hbase
and restart the hbase again it will resolve the problem.
I am trying to use C3P0 connection pools for IBM DB2 Version 10.5.5 Fix Pack 5 with Java 8 .
Initially, I was getting this error so I changed jar to db2jcc4.jar but I still get below error in console,
1350 [C3P0PooledConnectionPoolManager[identityToken->2x0fal9iui9mpmdj6im3|c2e1f26]-HelperThread-#1] INFO com.mchange.v2.c3p0.SQLWarnings – Origination unknown: [10228][11541][4.11.77] Security exceptions occurred while loading driver. ERRORCODE=4223, SQLSTATE=null
com.ibm.db2.jcc.am.SqlWarning: Origination unknown: [10228][11541][4.11.77] Security exceptions occurred while loading driver. ERRORCODE=4223, SQLSTATE=null
at com.ibm.db2.jcc.am.gd.b(gd.java:207)
at com.ibm.db2.jcc.am.gd.b(gd.java:258)
at com.ibm.db2.jcc.am.jb.a(jb.java:887)
at com.ibm.db2.jcc.am.lb.a(lb.java:574)
at com.ibm.db2.jcc.am.lb.<init>(lb.java:555)
at com.ibm.db2.jcc.t4.b.<init>(b.java:305)
at com.ibm.db2.jcc.DB2SimpleDataSource.getConnection(DB2SimpleDataSource.java:214)
at com.ibm.db2.jcc.DB2Driver.connect(DB2Driver.java:460)
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
Caused by: java.security.PrivilegedActionException: java.lang.ClassNotFoundException: sun.io.ByteToCharConverter
at java.security.AccessController.doPrivileged(Native Method)
at com.ibm.db2.jcc.am.jb.r(jb.java:873)
at com.ibm.db2.jcc.am.jb.<clinit>(jb.java:484)
at com.ibm.db2.jcc.DB2Driver.connect(DB2Driver.java:117)
... 9 more
Caused by: java.lang.ClassNotFoundException: sun.io.ByteToCharConverter
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Unknown Source)
at com.ibm.db2.jcc.am.jg.run(jg.java:19)
... 13 more
1350 [C3P0PooledConnectionPoolManager[identityToken->2x0fal9iui9mpmdj6im3|c2e1f26]-HelperThread-#0] INFO com.mchange.v2.c3p0.SQLWarnings – Origination unknown: [10228][11541][4.11.77] Security exceptions occurred while loading driver. ERRORCODE=4223, SQLSTATE=null
com.ibm.db2.jcc.am.SqlWarning: Origination unknown: [10228][11541][4.11.77] Security exceptions occurred while loading driver. ERRORCODE=4223, SQLSTATE=null
at com.ibm.db2.jcc.am.gd.b(gd.java:207)
at com.ibm.db2.jcc.am.gd.b(gd.java:258)
at com.ibm.db2.jcc.am.jb.a(jb.java:887)
at com.ibm.db2.jcc.am.lb.a(lb.java:574)
at com.ibm.db2.jcc.am.lb.<init>(lb.java:555)
at com.ibm.db2.jcc.t4.b.<init>(b.java:305)
at com.ibm.db2.jcc.DB2SimpleDataSource.getConnection(DB2SimpleDataSource.java:214)
at com.ibm.db2.jcc.DB2Driver.connect(DB2Driver.java:460)
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
Caused by: java.security.PrivilegedActionException: java.lang.ClassNotFoundException: sun.io.ByteToCharConverter
at java.security.AccessController.doPrivileged(Native Method)
at com.ibm.db2.jcc.am.jb.r(jb.java:873)
at com.ibm.db2.jcc.am.jb.<clinit>(jb.java:484)
at com.ibm.db2.jcc.DB2Driver.connect(DB2Driver.java:117)
... 9 more
Caused by: java.lang.ClassNotFoundException: sun.io.ByteToCharConverter
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Unknown Source)
at com.ibm.db2.jcc.am.jg.run(jg.java:19)
... 13 more
Even if this error is present in logs, my stand alone Java app is able to make connections to database and gets results i.e. SELECT query is executed successfully.
How to fix this confusion?
The stack traces you are seeing come from SqlWarning objects, which probably means they are not fatal or serious problems. The apparent problem java.lang.ClassNotFoundException: sun.io.ByteToCharConverter seems pretty strange. But since it is coming through as a warning rather than thrown as an Exception, the driver has probably recovered from the problem, which is why your application seems to work.
JDBC Connections are permitted to store a chain of SQLWarnings, which might arise from anything, but which most applications simply ignore. (Most JDBC programmers are unaware they even exist.)
c3p0 is extremely neurotic about periodically checking for Connection warnings, logging them, and then clearing them, so that users of Connections don't see warnings derived from prior clients' uses. The logging of warnings can be annoying, though, so if you want, it is easy to suppress it (or redirect it to a separate file, or whatever). c3p0 logs warnings to a special logger named com.mchange.v2.c3p0.SQLWarnings. In the configuration file for whatever logging library you use, you can suppress this output by setting that logger's level to WARNING or above.
(A bit confusingly SQLWarnings are logged at INFO not WARNING. Since they usually represent information about conditions already recovered from, which driver authors understand that most developers never see, they don't substantively rise to the level of WARNING. So filtering out messages from com.mchange.v2.c3p0.SQLWarnings at INFO and below is sufficient to quiet the warnings. If you'd like to see the responsible code, it is here.)
I'm trying to use Hadoop/Hbase sampler to connect and scan the tables created in Hbase, however my test is failing with the following messages.
Checked the configuration of Hadoop/Hbase all the service are running and listening to the appropriate port. (zookeeper is running and listening to 2181).
2016/04/02 15:44:04 ERROR - jmeter.threads.JMeterThread: Test failed! java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:37)
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:34)
at org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:217)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:185)
at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:237)
at org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:79)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:210)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:185)
at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:237)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:482)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:468)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
at org.apache.hadoop.hbase.security.User.call(User.java:590)
at org.apache.hadoop.hbase.security.User.callStatic(User.java:580)
at org.apache.hadoop.hbase.security.User.access$400(User.java:51)
at org.apache.hadoop.hbase.security.User$SecureHadoopUser.<init>(User.java:397)
at org.apache.hadoop.hbase.security.User$SecureHadoopUser.<init>(User.java:392)
at org.apache.hadoop.hbase.security.User.getCurrent(User.java:140)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionKey.<init>(HConnectionManager.java:435)
at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:180)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:155)
at org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:36)
at org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:265)
at org.apache.hadoop.hbase.client.HTablePool.findOrCreateTable(HTablePool.java:195)
at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:174)
at com.atlantbh.jmeter.plugins.hbasecomponents.config.HBaseConnectionVariable.getTable(HBaseConnectionVariable.java:43)
at com.atlantbh.jmeter.plugins.hbasecomponents.samplers.HBaseScanSampler.sample(HBaseScanSampler.java:94)
at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:434)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:261)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.Configuration
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 35 more**
The problem is on your JMeter side. As the error message states:
Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.Configuration
which means that HBase Scan Sampler requires Apache Commons Configuration library, but looks like Hadoop plug-in does not provide it in its zip package. Try to ask on their forum if they are willing to fix it.
For now as a workaround you can:
Download commons-configuration2-2.0-bin.zip from the site
Unzip it
Copy commons-configuration2-2.0.jar into .../apache-jmeter/lib
Restart jmeter
I got phoenix 4.0 running on hbase 0.98/hadoop 2.3.0 and was impressed by the command line tools.
In the second step I followed the description on the webpage to connect to phoenix using its bundled JDBC driver.
When I try to connect I get the Exception message (on Squirrel side)
java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.sql.SQLException: ERROR 103 (08004): Unable to establish connection.
at java.util.concurrent.FutureTask.report(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at net.sourceforge.squirrel_sql.client.mainframe.action.OpenConnectionCommand.awaitConnection(OpenConnectionCommand.java:132)
at net.sourceforge.squirrel_sql.client.mainframe.action.OpenConnectionCommand.access$100(OpenConnectionCommand.java:45)
at net.sourceforge.squirrel_sql.client.mainframe.action.OpenConnectionCommand$2.run(OpenConnectionCommand.java:115)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.RuntimeException: java.sql.SQLException: ERROR 103 (08004): Unable to establish connection.
at net.sourceforge.squirrel_sql.client.mainframe.action.OpenConnectionCommand.executeConnect(OpenConnectionCommand.java:171)
at net.sourceforge.squirrel_sql.client.mainframe.action.OpenConnectionCommand.access$000(OpenConnectionCommand.java:45)
at net.sourceforge.squirrel_sql.client.mainframe.action.OpenConnectionCommand$1.run(OpenConnectionCommand.java:104)
... 5 more
Caused by: java.sql.SQLException: ERROR 103 (08004): Unable to establish connection.
at org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:309)
at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:133)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnection(ConnectionQueryServicesImpl.java:254)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:1446)
at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:131)
at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.connect(PhoenixEmbeddedDriver.java:112)
at net.sourceforge.squirrel_sql.fw.sql.SQLDriverManager.getConnection(SQLDriverManager.java:133)
at net.sourceforge.squirrel_sql.client.mainframe.action.OpenConnectionCommand.executeConnect(OpenConnectionCommand.java:167)
... 7 more
Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:416)
at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:309)
at org.apache.phoenix.query.HConnectionFactory$HConnectionFactoryImpl.createConnection(HConnectionFactory.java:47)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnection(ConnectionQueryServicesImpl.java:252)
... 12 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:414)
... 15 more
Caused by: java.lang.RuntimeException: Socket Factory class not found: java.lang.ClassNotFoundException: Class org.apache.hadoop.net.StandardSocketFactory not found
at org.apache.hadoop.net.NetUtils.getSocketFactoryFromProperty(NetUtils.java:142)
at org.apache.hadoop.net.NetUtils.getDefaultSocketFactory(NetUtils.java:122)
at org.apache.hadoop.hbase.ipc.RpcClient.<init>(RpcClient.java:1293)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:664)
... 20 more
I double checked the jar files with classfinder to be sure that the class org.apache.hadoop.net.StandardSocketFactory IS in the classpath.
What can I do to get Squirrel connected with Phoenix?
Update:
I saw in the zookeeper log on the server side that the network communication started:
2014-05-28 06:24:29,411 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /192.168.1.106:58172
2014-05-28 06:24:29,412 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /192.168.1.106:58172
2014-05-28 06:24:29,518 INFO [SyncThread:0] server.ZooKeeperServer: Established session 0x146413f6c3a000c with negotiated timeout 90000 for client /192.168.1.106:58172
I solved the problem replacing the downloaded binary version 4.0 of phoenix with the snapshot version 4.1 which I built by myown from the source version cloned via git from
http://git.apache.org/incubator-phoenix.git/
After the successful build I extracted the tarball from the assembly subdirectory and copied the following jars to hbase 0.98's lib dir
phoenix-core-4.1.0-incubating-SNAPSHOT.jar
phoenix-flume-4.1.0-incubating-SNAPSHOT.jar
phoenix-pig-4.1.0-incubating-SNAPSHOT.jar
In Squirrel I used just phoenix-4.1.0-incubating-SNAPSHOT-client.jar as a extra path to get the driver running.
There is a transactional=true service in Grails 2.2 where I do just the following:
...
client.save(flush: true) // it hangs here
...
If I remove flush:true, the execution passes further although it hangs calling a withTransaction later.
I have never met such a problem with my current large project before.
I have an idea to throw out some flush'es and withTransaction's but that's kinda unwanted because there are a bunch of them in the code currently. So it would be a painful change even if it will fix the problem.
Any idea how to cure / investigate that?
Grails 2.2 / Java 1.6 / Windows 7 x64 / Oracle XE
Hanged thread's stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:150)
java.net.SocketInputStream.read(SocketInputStream.java:121)
oracle.net.ns.Packet.receive(Unknown Source)
oracle.net.ns.DataPacket.receive(Unknown Source)
oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
oracle.net.ns.NetInputStream.read(Unknown Source)
oracle.net.ns.NetInputStream.read(Unknown Source)
oracle.net.ns.NetInputStream.read(Unknown Source)
oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:971)
oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:941)
oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:432)
oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:181)
oracle.jdbc.driver.T4CPreparedStatement.execute_for_rows(T4CPreparedStatement.java:543)
oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:8674)
- locked oracle.jdbc.driver.T4CPreparedStatement#36df36e5
- locked oracle.jdbc.driver.T4CConnection#788d1087
org.apache.commons.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297)
org.apache.commons.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297)
org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:70)
org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:268)
org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:114)
org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:109)
org.hibernate.jdbc.AbstractBatcher.prepareBatchStatement(AbstractBatcher.java:244)
org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2412)
org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2875)
org.hibernate.action.EntityInsertAction.execute(EntityInsertAction.java:79)
org.hibernate.engine.ActionQueue.execute(ActionQueue.java:273)
org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:265)
org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:184)
org.codehaus.groovy.grails.orm.hibernate.events.PatchedDefaultFlushEventListener.performExecutions(PatchedDefaultFlushEventListener.java:46)
org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:51)
org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1216)
org.codehaus.groovy.grails.orm.hibernate.metaclass.SavePersistentMethod.flushSession(SavePersistentMethod.java:87)
org.codehaus.groovy.grails.orm.hibernate.metaclass.SavePersistentMethod$1.doInHibernate(SavePersistentMethod.java:60)
org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:406)
org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:339)
org.codehaus.groovy.grails.orm.hibernate.metaclass.SavePersistentMethod.performSave(SavePersistentMethod.java:56)
org.codehaus.groovy.grails.orm.hibernate.metaclass.AbstractSavePersistentMethod.doInvokeInternal(AbstractSavePersistentMethod.java:215)
org.codehaus.groovy.grails.orm.hibernate.metaclass.AbstractDynamicPersistentMethod.invoke(AbstractDynamicPersistentMethod.java:63)
sun.reflect.GeneratedMethodAccessor662.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:601)
org.springsource.loaded.ri.ReflectiveInterceptor.jlrMethodInvoke(ReflectiveInterceptor.java:1243)
org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSite.invoke(PojoMetaMethodSite.java:189)
org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:55)
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)
org.codehaus.groovy.grails.orm.hibernate.HibernateGormInstanceApi.save(HibernateGormEnhancer.groovy:911)
xxx.domain.Consumer.save(Consumer.groovy)
xxx.domain.Consumer$save.call(Unknown Source)
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
xxx.UserService.createUser(UserService.groovy:350)
............
Your stacktrace points at the likely problem:
oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:8674)
- locked oracle.jdbc.driver.T4CPreparedStatement#36df36e5
- locked oracle.jdbc.driver.T4CConnection#788d1087
You have a stuck thread (or threads) on your database and the app is hanging waiting for the threads. You need to check your database status to see if you can determine which queries are hanging. It's all possible updating to the latest JDBC driver will help.
The reason this is happening when you use flush: true is because it's forcing your Hibernate session to flush to the database, where it encounters the stuck thread.