I'm using Hadoop2.4.0 / Hbase 0.98.0 / Hive 0.14.0
Hadoop and HBase were running fine until I restarted my HMaster. The following error appears in hbase-hduser-master-master.log file :
2015-02-17 05:46:15,157 INFO [master:master:60000] master.TableNamespaceManager: Namespace table not found. Creating...
2015-02-17 05:46:15,193 DEBUG [master:master:60000] lock.ZKInterProcessLockBase: Acquired a lock for /hbase/table-lock/hbase:namespace/write-master:600000000000004
2015-02-17 05:46:15,212 DEBUG [master:master:60000] lock.ZKInterProcessLockBase: Released /hbase/table-lock/hbase:namespace/write-master:600000000000004
2015-02-17 05:46:15,212 FATAL [master:master:60000] master.HMaster: Master server abort: loaded coprocessors are: []
2015-02-17 05:46:15,213 FATAL [master:master:60000] master.HMaster: Unhandled exception. Starting shutdown.
org.apache.hadoop.hbase.TableExistsException: hbase:namespace
at org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:120)
at org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:232)
at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86)
at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1049)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:913)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:606)
at java.lang.Thread.run(Unknown Source)
2015-02-17 05:46:15,214 INFO [master:master:60000] master.HMaster: Aborting
2015-02-17 05:46:15,214 INFO [master,60000,1424180766819-BalancerChore] balancer.BalancerChore: master,60000,1424180766819-BalancerChore exiting
2015-02-17 05:46:15,215 INFO [master,60000,1424180766819-ClusterStatusChore] balancer.ClusterStatusChore: master,60000,1424180766819-ClusterStatusChore exiting
2015-02-17 05:46:15,215 INFO [CatalogJanitor-master:60000] master.CatalogJanitor: CatalogJanitor-master:60000 exiting
2015-02-17 05:46:15,216 DEBUG [master:master:60000] master.HMaster: Stopping service threads
2015-02-17 05:46:15,216 INFO [master:master:60000] ipc.RpcServer: Stopping server on 60000
2015-02-17 05:46:15,216 INFO [RpcServer.listener,port=60000] ipc.RpcServer: RpcServer.listener,port=60000: stopping
2015-02-17 05:46:15,218 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2015-02-17 05:46:15,218 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
2015-02-17 05:46:15,218 INFO [master:master:60000.oldLogCleaner] cleaner.LogCleaner: master:master:60000.oldLogCleaner exiting
2015-02-17 05:46:15,218 INFO [master:master:60000.oldLogCleaner] master.ReplicationLogCleaner: Stopping replicationLogCleaner-0x14b97c83f580008, quorum=slave:2181,master:2181, baseZNode=/hbase
2015-02-17 05:46:15,219 INFO [master:master:60000.archivedHFileCleaner] cleaner.HFileCleaner: master:master:60000.archivedHFileCleaner exiting
2015-02-17 05:46:15,219 INFO [master:master:60000] master.HMaster: Stopping infoServer
2015-02-17 05:46:15,223 INFO [master:master:60000.oldLogCleaner] zookeeper.ZooKeeper: Session: 0x14b97c83f580008 closed
2015-02-17 05:46:15,223 INFO [master:master:60000-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-02-17 05:46:15,229 INFO [master:master:60000] mortbay.log: Stopped SelectChannelConnector#0.0.0.0:60010
2015-02-17 05:46:15,236 DEBUG [master:master:60000] catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker#19f9598
2015-02-17 05:46:15,236 INFO [master:master:60000] client.HConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14b97c83f580007
2015-02-17 05:46:15,237 INFO [master:master:60000-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-02-17 05:46:15,238 INFO [master:master:60000] zookeeper.ZooKeeper: Session: 0x14b97c83f580007 closed
2015-02-17 05:46:15,238 INFO [master,60000,1424180766819.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor: master,60000,1424180766819.splitLogManagerTimeoutMonitor exiting
2015-02-17 05:46:15,243 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-02-17 05:46:15,243 INFO [master:master:60000] zookeeper.ZooKeeper: Session: 0x14b97c83f580006 closed
2015-02-17 05:46:15,243 INFO [master:master:60000] master.HMaster: HMaster main thread exiting
2015-02-17 05:46:15,243 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: HMaster Aborted
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:192)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:134)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2785)
what's wrong here and what HMaster Aborted means ?
for more information this is what my hbase-site.xml looks like:
<property>
<name>hbase.rootdir</name>
<value>hdfs://master:54310/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>master,slave</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/usr/local/hbase/zookeeper</value>
</property>
I ran into this problem today! My solution is as follows:
Step 1:stop Hbase.
Step 2:run the follow command
hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair
This command is used to repair MetaData of Hbase
Step 3:delete the data in zookeeper (WARNING It will make you lost you old data)
./opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/zookeeper/bin/zkCli.sh
you can use ls / to scan the data in zookeeper
use rmr /hbase to delete the hbase's data in zookeeper
Step 4：Start hbase
This is based on the other answer but to clarify for upgrade cloudera 5.4
Step 1:
service hbase-regionserver stop
service hbase-master stop
Step 2:
hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair
Step 3: Delete the data in zookeeper (WARNING It will make you lost your old data)
cd /usr/lib/zookeeper/bin/
./zkCli.sh
It opens up the zookeeper shell.
Then run:
ls /
rmr /hbase
Step 4：Start hbase
service hbase-master restart
service hbase-regionserver restart
Related
I configured HBase today and I configured it correctly at first. However, when I ran HBase use the code 'start-all.sh' again, I could not see 'Hmaster' anywhere. It just shows like:
[root#master bin]# jps
25164 QuorumPeerMain
83447 HRegionServer
44542 NameNode
44789 DataNode
45098 SecondaryNameNode
45378 ResourceManager
45536 NodeManager
56678 <no information available>
56949 Jps
when I 'jps' again, '':
enter image description here
and the log shows:
[root#master bin]# cd /home/hadoop/hbase-2.2.3/logs
[root#master logs]# ls
hbase-root-master-master.log hbase-root-regionserver-master.out.1
hbase-root-master-master.out hbase-root-regionserver-master.out.2
hbase-root-master-master.out.1 hbase-root-regionserver-master.out.3
hbase-root-regionserver-master.log hbase-root-regionserver-master.out.4
hbase-root-regionserver-master.out SecurityAuth.audit
[root#master logs]# tail hbase-root-master-master.log
2022-04-28 17:29:56,674 INFO [master/master:16000] zookeeper.ZooKeeper: Session: 0x100000e4a0d0020 closed
2022-04-28 17:29:56,674 INFO [master/master:16000] regionserver.HRegionServer: Exiting; stopping=master,16000,1651138191876; zookeeper connection closed.
2022-04-28 17:29:56,674 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x100000e4a0d0020
2022-04-28 17:29:56,674 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: HMaster Aborted
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:244)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2940)
[root#master logs]#
I solve the problem by adding the following configuration to the configuration file "hbase-site.xml":
<property>
<name>hbase.unsafe.stream.capability.enforce</name>
<value>false</value>
</property>
I do not why, but it works.
I was running some tests with table mappers and reducers on large scale problems. After a certain point my reducers started failing when the job was 80% done. From what I can tell when looking at the syslogs the problem is that one of my zookeepers is attempting to connect to the localhost as opposed to the other zookeepers in the quorum
Oddly it seems to do just fine connecting to the other nodes when mapping is going on, its reducing that it has a problem with. Here are selected portions of the syslog which might be relevant to figuring out whats going on
2014-06-27 09:44:01,599 INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hdev02:5181,hdev01:5181,hdev03:5181 sessionTimeout=10000 watcher=hconnection-0x4aee260b, quorum=hdev02:5181,hdev01:5181,hdev03:5181, baseZNode=/hbase
2014-06-27 09:44:01,612 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x4aee260b connecting to ZooKeeper ensemble=hdev02:5181,hdev01:5181,hdev03:5181
2014-06-27 09:44:01,614 INFO [main-SendThread(hdev02:5181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server hdev02/172.17.43.36:5181. Will not attempt to authenticate using SASL (Unable to locate a login configuration)
2014-06-27 09:44:01,615 INFO [main-SendThread(hdev02:5181)] org.apache.zookeeper.ClientCnxn: Socket connection established to hdev02/172.17.43.36:5181, initiating session
2014-06-27 09:44:01,617 INFO [main-SendThread(hdev02:5181)] org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2014-06-27 09:44:01,723 WARN [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=hdev02:5181,hdev01:5181,hdev03:5181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
2014-06-27 09:44:01,723 INFO [main] org.apache.hadoop.hbase.util.RetryCounter: Sleeping
***
org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 1 on-disk map-outputs
2014-06-27 09:55:12,012 INFO [main] org.apache.hadoop.mapred.Merger: Merging 1 sorted segments
2014-06-27 09:55:12,013 INFO [main] org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 33206049 bytes
2014-06-27 09:55:12,208 INFO [main] org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merged 1 segments, 33206079 bytes to disk to satisfy reduce memory limit
2014-06-27 09:55:12,209 INFO [main] org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 2 files, 265119413 bytes from disk
2014-06-27 09:55:12,209 INFO [main] org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
2014-06-27 09:55:12,210 INFO [main] org.apache.hadoop.mapred.Merger: Merging 2 sorted segments
2014-06-27 09:55:12,212 INFO [main] org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 265119345 bytes
2014-06-27 09:55:12,279 INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x65afdbbb, quorum=localhost:2181, baseZNode=/hbase
2014-06-27 09:55:12,281 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x65afdbbb connecting to ZooKeeper ensemble=localhost:2181
2014-06-27 09:55:12,282 INFO [main-SendThread(localhost.localdomain:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost.localdomain/127.0.0.1:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration)
2014-06-27 09:55:12,283 WARN [main-SendThread(localhost.localdomain:2181)] org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
2014-06-27 09:55:12,384 WARN [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=localhost:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
2014-06-27 09:55:12,384 INFO [main] org.apache.hadoop.hbase.util.RetryCounter: Sleeping 1000ms before retry #0...
2014-06-27 09:55:13,385 INFO [main-SendThread(localhost.localdomain:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost.localdomain/127.0.0.1:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration)
2014-06-27 09:55:13,385 WARN [main-SendThread(localhost.localdomain:2181)] org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing
***
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=localhost:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
2014-06-27 09:55:13,486 ERROR [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: ZooKeeper exists failed after 1 attempts
2014-06-27 09:55:13,486 WARN [main] org.apache.hadoop.hbase.zookeeper.ZKUtil: hconnection-0x65afdbbb, quorum=localhost:2181, baseZNode=/hbase Unable to set watcher on znode (/hbase/hbaseid)
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
I'm pretty sure its configured correctly, here is the relevant portion of my hbase-site.xml.
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>5181</value>
<description>Property from ZooKeeper's config zoo.cfg.
The port at which the clients will connect.
</description>
</property>
<property>
<name>zookeeper.session.timeout</name>
<value>10000</value>
<description></description>
</property>
<property>
<name>hbase.client.retries.number</name>
<value>10</value>
<description></description>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>hdev01,hdev02,hdev03</value>
<description></description>
</property>
So far as I can tell hdev03 is the only server that has any problem with this. Netstating all relevant ports doesn't show me anything strange.
I've had same problem when running HBase through Spark on Yarn. Everything was fine until suddenly it started to trying to connect to localhost instead of quorum. Setting port and quorum programmatically before HBase call fixed the issue
conf.set("hbase.zookeeper.quorum","my.server")
conf.set("hbase.zookeeper.property.clientPort","5181")
I'm using MapR, and it has "unusual" (5181) zookeeper port
Hard to say what is happening with the information, given. I have found the Hadoop Stack (HBase especially) to be quite hostile to even the slightest bit of misconfiguration in DNS or the hosts file.
As the quorum in your hbase-site.xml looks good, I'd start checking with network/hostname resolution related configurations:
Has the nodename slipped into the localhost entry in /etc/hosts on hdev03?
Is there an entry for the host itself in hdev03s /etc/hosts (there should)?
Has Reverse DNS been correctly configured in case you are using DNS for name resolution instead of the hosts file?
These are just a few pointers in the direction I'd look with this kind of issue. Hope it helps!
Add '--driver-class-path ~/hbase-1.1.2/conf' into spark-submit command, so that the task can find the configured zookeeper servers instead of 127.0.0.1.
In todays episode of hbase is bringing me to my wits end we have an issue where the hbase master starts and then very quickly dies. My master log is like so:
2014-06-20 12:52:40,469 FATAL [master:hdev01:60000] master.HMaster: Master serve
r abort: loaded coprocessors are: []
2014-06-20 12:52:40,470 FATAL [master:hdev01:60000] master.HMaster: Unhandled ex
ception. Starting shutdown.
org.apache.hadoop.hbase.TableExistsException: hbase:namespace
at org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(Cre
ateTableHandler.java:120)
at org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceT
able(TableNamespaceManager.java:232)
at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNames
paceManager.java:86)
at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:106
2)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.j
ava:926)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:615)
at java.lang.Thread.run(Thread.java:662)
2014-06-20 12:52:40,473 INFO [master:hdev01:60000] master.HMaster: Aborting
2014-06-20 12:52:40,473 DEBUG [master:hdev01:60000] master.HMaster: Stopping ser
vice threads
2014-06-20 12:52:40,473 INFO [master:hdev01:60000] ipc.RpcServer: Stopping serv
er on 60000
2014-06-20 12:52:40,473 INFO [CatalogJanitor-hdev01:60000] master.CatalogJanito
r: CatalogJanitor-hdev01:60000 exiting
2014-06-20 12:52:40,473 INFO [hdev01,60000,1403283149823-BalancerChore] balance
r.BalancerChore: hdev01,60000,1403283149823-BalancerChore exiting
2014-06-20 12:52:40,474 INFO [RpcServer.listener,port=60000] ipc.RpcServer: Rpc
Server.listener,port=60000: stopping
2014-06-20 12:52:40,474 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.res
ponder: stopped
2014-06-20 12:52:40,474 INFO [master:hdev01:60000] master.HMaster: Stopping inf
oServer
2014-06-20 12:52:40,474 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.res
ponder: stopping
2014-06-20 12:52:40,474 INFO [master:hdev01:60000.oldLogCleaner] cleaner.LogCle
aner: master:hdev01:60000.oldLogCleaner exiting
2014-06-20 12:52:40,475 INFO [hdev01,60000,1403283149823-ClusterStatusChore] ba
lancer.ClusterStatusChore: hdev01,60000,1403283149823-ClusterStatusChore exiting
2014-06-20 12:52:40,476 INFO [master:hdev01:60000.oldLogCleaner] master.Replica
tionLogCleaner: Stopping replicationLogCleaner-0x246ba2ab1e4001c, quorum=hdev02:
5181,hdev01:5181,hdev03:5181, baseZNode=/hbase
2014-06-20 12:52:40,479 INFO [master:hdev01:60000] mortbay.log: Stopped SelectC
hannelConnector#0.0.0.0:16010
2014-06-20 12:52:40,478 INFO [master:hdev01:60000.archivedHFileCleaner] cleaner
.HFileCleaner: master:hdev01:60000.archivedHFileCleaner exiting
2014-06-20 12:52:40,483 INFO [master:hdev01:60000.oldLogCleaner] zookeeper.ZooK
eeper: Session: 0x246ba2ab1e4001c closed
2014-06-20 12:52:40,484 INFO [master:hdev01:60000-EventThread] zookeeper.Client
Cnxn: EventThread shut down
2014-06-20 12:52:40,589 DEBUG [master:hdev01:60000] catalog.CatalogTracker: Stop
ping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker#f3f348b
2014-06-20 12:52:40,591 INFO [master:hdev01:60000] client.HConnectionManager$HC
onnectionImplementation: Closing zookeeper sessionid=0x246ba2ab1e4001b
2014-06-20 12:52:40,592 INFO [master:hdev01:60000] zookeeper.ZooKeeper: Session
: 0x246ba2ab1e4001b closed
2014-06-20 12:52:40,592 INFO [master:hdev01:60000-EventThread] zookeeper.Client
Cnxn: EventThread shut down
2014-06-20 12:52:40,695 INFO [hdev01,60000,1403283149823.splitLogManagerTimeout
Monitor] master.SplitLogManager$TimeoutMonitor: hdev01,60000,1403283149823.split
LogManagerTimeoutMonitor exiting
2014-06-20 12:52:40,696 INFO [master:hdev01:60000] zookeeper.ZooKeeper: Session
: 0x246ba2ab1e4001a closed
2014-06-20 12:52:40,696 INFO [main-EventThread] zookeeper.ClientCnxn: EventThre
ad shut down
2014-06-20 12:52:40,696 INFO [master:hdev01:60000] master.HMaster: HMaster main
thread exiting
2014-06-20 12:52:40,697 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: HMaster Aborted
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMaster
CommandLine.java:194)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandL
ine.java:135)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLi
ne.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2803)
I thought this might be some remnant of an old run so I deleted the files in hbases data directory, the zookeepers data directory and my hdfs. I still got the same error. Strangely my HMaster popper back up again temporarily when I ran stop-hbase.sh although there wasn't much I could do with it.
My Hbase version is 98.3 and my hadoop is 2.2.0. My hbase-site.comf is
<configuration>
<property>
<name>hbase.master</name>
<value>hdev01:60000</value>
<description>The host and port that the HBase master runs at.
A value of 'local' runs the master and a regionserver
in a single process.
</description>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://hdev01:9000/hbase</value>
<description>The directory shared by region servers.</description>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
<description>The mode the cluster will be in. Possible values are
false: standalone and pseudo-distributed setups with managed
Zookeeper true: fully-distributed with unmanaged Zookeeper
Quorum (see hbase-env.sh)
</description>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>5181</value>
<description>Property from ZooKeeper's config zoo.cfg.
The port at which the clients will connect.
</description>
</property>
<property>
<name>zookeeper.session.timeout</name>
<value>10000</value>
<description></description>
</property>
<property>
<name>hbase.client.retries.number</name>
<value>10</value>
<description></description>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>hdev01,hdev02,hdev03</value>
<description>Comma separated list of servers in the ZooKeeper Quorum. For example, "host1.mydomain.com,host2.mydomain.com". By default this is set to localhost for local and pseudo-distributed modes of operation. For a fully-distributed setup, this should be set to a full list of ZooKeeper quorum servers. If
HBASE_MANAGES_ZK is set in hbase-env.sh
this is the list of servers which we will start/stop
ZooKeeper on.
</description>
</property>
</configuration>
EDIT
Attempted hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair, my error now is HBase file layout needs to be upgraded. You have version null and I want version 8. Is your hbase.rootdir valid? If so, you may need to run 'hbase hbck -fixVersionFile'
Which is unhelpful since without a master hbck will not actually run.
Edited edit
I nuked and restarted my dfs and then tried repairing and starting things again, i am now back where i started.
hbase namespace is the internal namespace HBAse uses for its own management tables. Try to run the offline repair tool
from the $HBASE_HOME directory:
./bin/hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair
su - hdfs
hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair
(restart the hbase master.if still u are facing issue then do following)
zookeeper-client (enter)
rmr /hbase
quit
Then restart the hbase master service
#shash:
When HBase manages ZooKeeper( i.e. HBASE_manages_ZK=true), the command to access and clean hbase data is :
hbase zkcli. Afterwards you clean hbae using the command rmr /hbase, then you quit.
I am trying to run Hbase 0.96.1.1 for Hadoop 2 on a Mac book air. When I run ./start-hbase.sh,
starting master, logging to.....
but it crashes right after.
I checked the log file and this the error message it spat out:
Fri Mar 28 12:49:20 PDT 2014 Starting master on ms12
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 256
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 709
virtual memory (kbytes, -v) unlimited
2014-03-28 12:49:21,203 INFO [main] util.VersionInfo: HBase 0.96.1.1-hadoop2
2014-03-28 12:49:21,203 INFO [main] util.VersionInfo: Subversion file:///home/jon/proj/hbase-svn/hbase-0.96.1.1 -r Unknown
2014-03-28 12:49:21,204 INFO [main] util.VersionInfo: Compiled by jon on Tue Dec 17 12:22:12 PST 2013
2014-03-28 12:49:21,894 INFO [main] server.ZooKeeperServer: Server environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
2014-03-28 12:49:21,894 INFO [main] server.ZooKeeperServer: Server environment:host.name=guest-wireless-nup-nat-206-117-89-004.usc.edu
2014-03-28 12:49:21,895 INFO [main] server.ZooKeeperServer: Server environment:java.version=1.6.0_65
2014-03-28 12:49:21,895 INFO [main] server.ZooKeeperServer: Server environment:java.vendor=Apple Inc.
2014-03-28 12:49:21,895 INFO [main] server.ZooKeeperServer: Server environment:java.home=/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
2014-03-28 12:49:21,895 INFO [main] server.ZooKeeperServer: Server environment:java.class.path=/Users/hbase/hbase-0.96.1.1-hadoop2/conf:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/tools.jar:/Users/hbase/hbase-0.96.1.1-hadoop2:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/activation-1.1.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/aopalliance-1.0.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/asm-3.1.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/avro-1.7.4.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/commons-beanutils-1.7.0.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/commons-beanutils-core-1.8.0.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/commons-cli-1.2.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/commons-codec-1.7.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/commons-collections-3.2.1.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/commons-compress-1.4.1.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/commons-configuration-1.6.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/commons-daemon-1.0.13.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/commons-digester-1.8.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/commons-el-1.0.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/commons-httpclient-3.1.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/commons-io-2.4.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/commons-lang-2.6.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/commons-logging-1.1.1.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/commons-math-2.2.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/commons-net-3.1.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/core-3.1.1.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/findbugs-annotations-1.3.9-1.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/gmbal-api-only-3.0.0-b023.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/grizzly-framework-2.1.2.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/grizzly-http-2.1.2.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/grizzly-http-server-2.1.2.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/grizzly-http-servlet-2.1.2.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/grizzly-rcm-2.1.2.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/guava-12.0.1.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/guice-3.0.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/guice-servlet-3.0.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hadoop-annotations-2.2.0.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hadoop-auth-2.2.0.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hadoop-client-2.2.0.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hadoop-common-2.2.0.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hadoop-hdfs-2.2.0-tests.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hadoop-hdfs-2.2.0.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hadoop-mapreduce-client-app-2.2.0.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hadoop-mapreduce-client-common-2.2.0.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hadoop-mapreduce-client-core-2.2.0.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hadoop-mapreduce-client-jobclient-2.2.0.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hadoop-mapreduce-client-shuffle-2.2.0.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hadoop-yarn-api-2.2.0.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hadoop-yarn-client-2.2.0.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hadoop-yarn-common-2.2.0.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hadoop-yarn-server-common-2.2.0.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hadoop-yarn-server-nodemanager-2.2.0.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hamcrest-core-1.3.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hbase-client-0.96.1.1-hadoop2.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hbase-common-0.96.1.1-hadoop2-tests.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hbase-common-0.96.1.1-hadoop2.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hbase-examples-0.96.1.1-hadoop2.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hbase-hadoop-compat-0.96.1.1-hadoop2.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hbase-hadoop2-compat-0.96.1.1-hadoop2.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hbase-it-0.96.1.1-hadoop2-tests.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hbase-it-0.96.1.1-hadoop2.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hbase-prefix-tree-0.96.1.1-hadoop2.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hbase-protocol-0.96.1.1-hadoop2.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hbase-server-0.96.1.1-hadoop2-tests.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hbase-server-0.96.1.1-hadoop2.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hbase-shell-0.96.1.1-hadoop2.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hbase-testing-util-0.96.1.1-hadoop2.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/hbase-thrift-0.96.1.1-hadoop2.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/high-scale-lib-1.1.1.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/htrace-core-2.01.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/httpclient-4.1.3.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/httpcore-4.1.3.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jackson-core-asl-1.8.8.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jackson-jaxrs-1.8.8.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jackson-mapper-asl-1.8.8.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jackson-xc-1.8.8.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jamon-runtime-2.3.1.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jasper-compiler-5.5.23.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jasper-runtime-5.5.23.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/javax.inject-1.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/javax.servlet-3.1.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/javax.servlet-api-3.0.1.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jaxb-api-2.2.2.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jaxb-impl-2.2.3-1.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jersey-client-1.9.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jersey-core-1.8.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jersey-grizzly2-1.9.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jersey-guice-1.9.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jersey-json-1.8.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jersey-server-1.8.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jersey-test-framework-core-1.9.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jersey-test-framework-grizzly2-1.9.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jets3t-0.6.1.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jettison-1.3.1.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jetty-6.1.26.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jetty-sslengine-6.1.26.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jetty-util-6.1.26.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jruby-complete-1.6.8.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jsch-0.1.42.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jsp-2.1-6.1.14.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jsp-api-2.1-6.1.14.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jsp-api-2.1.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/jsr305-1.3.9.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/junit-4.11.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/libthrift-0.9.0.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/log4j-1.2.17.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/management-api-3.0.0-b012.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/metrics-core-2.1.2.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/netty-3.6.6.Final.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/paranamer-2.3.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/protobuf-java-2.5.0.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/servlet-api-2.5-6.1.14.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/servlet-api-2.5.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/slf4j-api-1.6.4.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/slf4j-log4j12-1.6.4.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/snappy-java-1.0.4.1.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/stax-api-1.0.1.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/xmlenc-0.52.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/xz-1.0.jar:/Users/hbase/hbase-0.96.1.1-hadoop2/lib/zookeeper-3.4.5.jar:
2014-03-28 12:49:21,897 INFO [main] server.ZooKeeperServer: Server environment:java.library.path=.:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
2014-03-28 12:49:21,898 INFO [main] server.ZooKeeperServer: Server environment:java.io.tmpdir=/var/folders/ww/vvdhqz_d2ggcht76g3fp2zh00000gn/T/
2014-03-28 12:49:21,898 INFO [main] server.ZooKeeperServer: Server environment:java.compiler=<NA>
2014-03-28 12:49:21,898 INFO [main] server.ZooKeeperServer: Server environment:os.name=Mac OS X
2014-03-28 12:49:21,898 INFO [main] server.ZooKeeperServer: Server environment:os.arch=x86_64
2014-03-28 12:49:21,898 INFO [main] server.ZooKeeperServer: Server environment:os.version=10.9.2
2014-03-28 12:49:21,898 INFO [main] server.ZooKeeperServer: Server environment:user.name=ms12
2014-03-28 12:49:21,898 INFO [main] server.ZooKeeperServer: Server environment:user.home=/Users/ms12
2014-03-28 12:49:21,898 INFO [main] server.ZooKeeperServer: Server environment:user.dir=/Users/hbase/hbase-0.96.1.1-hadoop2/bin
2014-03-28 12:49:21,921 INFO [main] server.ZooKeeperServer: Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /Users/hbase/zookeeper-storage-2/zookeeper_0/version-2 snapdir /Users/hbase/zookeeper-storage-2/zookeeper_0/version-2
2014-03-28 12:49:21,962 INFO [main] server.NIOServerCnxnFactory: binding to port 0.0.0.0/0.0.0.0:2181
2014-03-28 12:49:21,972 INFO [main] persistence.FileTxnSnapLog: Snapshotting: 0x0 to /Users/hbase/zookeeper-storage-2/zookeeper_0/version-2/snapshot.0
2014-03-28 12:49:22,269 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /127.0.0.1:53624
2014-03-28 12:49:22,278 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Processing stat command from /127.0.0.1:53624
2014-03-28 12:49:22,283 INFO [Thread-3] server.NIOServerCnxn: Stat command output
2014-03-28 12:49:22,284 INFO [Thread-3] server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:53624 (no session established for client)
2014-03-28 12:49:22,287 INFO [main] zookeeper.MiniZooKeeperCluster: Started MiniZK Cluster and connect 1 ZK server on client port: 2181
2014-03-28 12:49:22,328 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster
at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:140)
at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:200)
at org.apache.hadoop.hbase.LocalHBaseCluster.<init>(LocalHBaseCluster.java:150)
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:177)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:134)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2779)
Caused by: java.net.UnknownHostException: No such interface $iface
at org.apache.hadoop.net.DNS.getIPs(DNS.java:183)
at org.apache.hadoop.net.DNS.getIPs(DNS.java:145)
at org.apache.hadoop.net.DNS.getHosts(DNS.java:237)
at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:344)
at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:362)
at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:341)
at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:414)
at org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.<init>(HMasterCommandLine.java:256)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:137)
... 7 more
It seems that iface is an network interface on Linux system. Does that mean this version can not be run on Mac?
Edited:
I tested hbase version 0.98 also. Same issue. The only version that is working is hbase 0.94 but it is not compatible with hadoop 2.
It sounds like you used the instructions here:
http://opentsdb.net/setup-hbase.html
But did not do them correctly. The string $iface should not actually show up in your hbase-site.xml. It is expanded to the value of your loopback interface device when you write out your config using the exact commands given in those instructions. If you just copy-paste the config from there it won't work. On a mac it should result in lo0 for each of the below properties...
<property>
<name>hbase.zookeeper.dns.interface</name>
<value>lo0</value>
</property>
<property>
<name>hbase.regionserver.dns.interface</name>
<value>lo0</value>
</property>
<property>
<name>hbase.master.dns.interface</name>
<value>lo0</value>
</property>
I had the same issue running HBase 98.6-hadoop2 on Ubuntu 12.04. It seems that something changed in the configuration needed for a standalone run mode. Try this in your hbase-site.xml configuration file
<configuration>
<property>
<name>hbase.rootdir</name>
<value>file:///{your hbase data directory}</value>
</property>
<property>
<name>hbase.zookeper.property.dataDir</name>
<value>file:///{your zookeper stuff directory}</value>
</property>
<property>
<name>hbase.regionserver.dns.interface</name>
<value>default</value>
</property>
<property>
<name>hbase.master.dns.interface</name>
<value>default</value>
</property>
</configuration>
Maybe these links can be of some help
http://hbase.apache.org/book/config.files.html#hbase_default_configurations
http://www.sujee.net/tech/articles/hadoop/hadoop-dns/
While running a topology in storm we are getting error like this,
8983 [Thread-6] INFO com.netflix.curator.framework.imps.CuratorFrameworkImpl -
Starting
9144 [main] INFO **backtype.storm.daemon.nimbus** - Shutting down master
9199 [Thread-6-EventThread] INFO backtype.storm.zookeeper - Zookeeper state upd
ate: :connected:none
9241 [main] INFO backtype.storm.daemon.nimbus - Shut down master
9273 [Thread-6] INFO com.netflix.curator.framework.imps.CuratorFrameworkImpl -
Starting
9306 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN org.apache.zookeeper.serv
er.NIOServerCnxn - EndOfStreamException: Unable to read additional data from cli
ent sessionid 0x143af55728d0003, likely client has closed socket
9354 [main] INFO backtype.storm.daemon.supervisor - Shutting down c094c3b1-a378
-4c4f-af35-9278647c217a:4beddc09-4675-4fb9-8bdc-9cf5013ce9ca
9358 [main] INFO backtype.storm.daemon.supervisor - Shut down c094c3b1-a378-4c4
f-af35-9278647c217a:4beddc09-4675-4fb9-8bdc-9cf5013ce9ca
9361 [main] INFO **backtype.storm.daemon.superviso**r - Shutting down supervisor c0
94c3b1-a378-4c4f-af35-9278647c217a
9364 [Thread-5] INFO **backtype.storm.event** - Event manager interrupted
9369 [Thread-6] INFO backtype.storm.event - Event manager interrupted
9425 [main] INFO **backtype.storm.daemon.supervisor** - Shutting down supervisor 38
6d8d71-c9b5-4b51-bd6e-f9f605034ea0
9428 [Thread-8] INFO backtype.storm.event - Event manager interrupted
9429 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN org.apache.zookeeper.serv
er.NIOServerCnxn - EndOfStreamException: Unable to read additional data from cli
ent sessionid 0x143af55728d0007, likely client has closed socket
9429 [Thread-9] INFO backtype.storm.event - Event manager interrupted
9473 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN org.apache.zookeeper.serv
er.NIOServerCnxn - EndOfStreamException: Unable to read additional data from cli
ent sessionid 0x143af55728d0009, likely client has closed socket
9476 [main] INFO backtype.storm.testing - Shutting down in process zookeeper
9503 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN org.apache.zookeeper.serv
er.NIOServerCnxn - Ignoring exception
**java.nio.channels.ClosedChannelException**: null
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.jav
a:211) ~[na:1.7.0_03]
at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.j
ava:242) ~[zookeeper-3.3.3.jar:3.3.3-1073969]
9510 [main] INFO **backtype.storm.testing** - Done shutting down in process zookeep
er
9513 [main] INFO backtype.storm.testing - Deleting temporary path C:\Users\sowm
iya\AppData\Local\Temp\c9b1bc1a-a950-4098-af77-f81a4d2b112f
9520 [main] INFO backtype.storm.testing - Deleting temporary path C:\Users\sowm
iya\AppData\Local\Temp\7e75c468-18ea-4787-a4ac-496fb108db71
9527 [main] INFO backtype.storm.testing - Unable to delete file: C:\Users\sowmi
ya\AppData\Local\Temp\7e75c468-18ea-4787-a4ac-496fb108db71\version-2\log.1
9529 [main] INFO backtype.storm.testing - Deleting temporary path C:\Users\sowm
iya\AppData\Local\Temp\fa7b3c9b-ac93-4090-b9e2-63f10019e61f
9543 [main] INFO backtype.storm.testing - Deleting temporary path C:\Users\sowm
iya\AppData\Local\Temp\55f1fd11-508e-43bb-b340-0d9b79f3af33
9579 [Thread-6-EventThread] INFO com.netflix.curator.framework.state.Connection
StateManager - State change: SUSPENDED
9580 [ConnectionStateManager-0] WARN com.netflix.curator.framework.state.Connec
tionStateManager - There are no ConnectionStateListeners registered.
9583 [Thread-6-EventThread] WARN backtype.storm.cluster - Received event :disco
nnected::none: with disconnected Zookeeper.
11232 [Thread-6-SendThread(localhost:2000)] WARN org.apache.zookeeper.ClientCnx
n - Session 0x143af55728d000b for server null, unexpected error, closing socket
connection and attempting reconnect
**java.net.ConnectException: Connection refused: no further information**
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_0
3]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701
) ~[na:1.7.0_03]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
~[zookeeper-3.3.3.jar:3.3.3-1073969]
13992 [Thread-6-SendThread(localhost:2000)] WARN org.apache.zookeeper.ClientCnx
n - Session 0x143af55728d000b for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_0
3]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701
) ~[na:1.7.0_03]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
Whwn we are trying to run the topology jar file all the operation like nimbus,zookeeper and supervisor process going to dead.please help us to know why this is happened.
Please help us to rectify this error and help to proceed further.
Thank you,
Sowmiya
Priya
This looks like a zookeeper issue. It looks like your processes are not being able to connect to zookeeper. Can't say more without more information.