HBase's RegionServer crashs - hadoop

Question

I'm trying to create about 589 tables and make random insertions. I start processing table by table: so I create one table then make all of my insertions, then create another one until all of the data get ingested.
The architecture of this solution is :
Python client located in one machine which ingests HBase with data.
Cloudera server hosting HBase configured in stand-alone which is a VM located in the same machine as the client and indentified by its IP address. The caracteristics of this server are as follows: 64GB of storage, 4GB of RAM and 1 CPU.
The client communicates with an HBase Thrift Server.
So the problem here is that when I try to ingest all of that amount of data. The client is only able to create and insert about 300MB before the regionserver shuts down (about 45 tables created and respective rows inserted and then the server crashs at the 46th table's data ingestion). I have tested all of this with different machine caracteristics, the size of the ingested data varies from machine to another (If the machine has more memory, more data will be inserted [Have tested this with different VM hardware caracteristics]). I'm suspecting that it's coming from the management of the Java Heap Memory, so I have tried to make different configurations. But it didn't make it better. Here is my main configuration of HBase :
hbase-site.xml
<property>
<name>hbase.rest.port</name>
<value>8070</value>
<description>The port for the HBase REST server.</description>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://quickstart.cloudera:8020/hbase</value>
</property>
<property>
<name>hbase.regionserver.ipc.address</name>
<value>0.0.0.0</value>
</property>
<property>
<name>hbase.master.ipc.address</name>
<value>0.0.0.0</value>
</property>
<property>
<name>hbase.thrift.info.bindAddress</name>
<value>0.0.0.0</value>
</property>
<property>
<name>hbase.hregion.max.filesize</name>
<value>10737418240</value> <!-- 10 GB -->
</property>
<property>
<name>hbase.hregion.memstore.flush.size</name>
<value>33554432</value> <!-- 32 MB -->
</property>
<property>
<name>hbase.client.write.buffer</name>
<value>8388608</value>
</property>
<property>
<name>hbase.client.scanner.caching</name>
<value>10000</value>
</property>
<property>
<name>hbase.regionserver.handler.count</name>
<value>64</value>
</property>
hbase-env.sh
# The maximum amount of heap to use. Default is left to JVM default.
export HBASE_HEAPSIZE=4G
# Uncomment below if you intend to use off heap cache. For example, to allocate 8G of
# offheap, set the value to "8G".
# export HBASE_OFFHEAPSIZE=1G
# Extra Java runtime options.
# Below are what we set by default. May only work with SUN JVM.
# For more on why as well as other possible settings,
# see http://wiki.apache.org/hadoop/PerformanceTuning
export HBASE_OPTS="-XX:+UseConcMarkSweepGC"
# Configure PermSize. Only needed in JDK7. You can safely remove it for JDK8+
export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m"
export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=4g -XX:MaxPermSize=4g"
Here is the error that I get from the Master Server's log:
util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC):
pause of approximately 1016msNo GCs detected
and nothing appears in the RegionServer's log.
On the other hand, when I try to create only one table and to insert a greater amount of data this works!
Any brilliant idea about how to fix this, please?
Thanks in advance.

Answer 1

Your VM's memory is way too low. Try bumping it up to at least 12GB. You're forgetting that a Java process's heap is only one part of the memory footprint. By setting HBASE_HEAPSIZE=4G you're saying you want HBase to allocate all your VM's memory. The VM also needs to run Linux daemons and your Cloudera services besides HBase.