Memory Requirements

Dear all,

Today I ran HBase in the pseudo-distributed mode. Since I need to save the
meta information about Web pages crawled from more than 10,000 Web sites
simultaneously and periodically. I believe the data size is large. After
running for about 30 minutes, it got the following exceptions.

     [java] ...............2012-3-10 10:09:04
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
processBatchCallback
     [java] Warning: Failed all from
region=PlayTable,,1331343367325.2bf6bf580051da504c347c19eb08f9d5.,
hostname=greatfreeinpain, port=60020
     [java] java.util.concurrent.ExecutionException:
java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
     [java] at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
     [java] at java.util.concurrent.FutureTask.get(FutureTask.java:83)
     [java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1557)
     [java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1409)
     [java] at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:900)
     [java] at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:777)
     [java] at org.apache.hadoop.hbase.client.HTable.put(HTable.java:760)
     [java] at
org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:402)
     [java] at
com.greatfree.hbase.AuthorityPersister.InsertInitialPlayRecord(AuthorityPersister.java:128)
     [java] at
com.greatfree.hbase.RandomPersistDispatcher.run(RandomPersistDispatcher.java:92)
     [java] at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
     [java] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
     [java] at java.lang.Thread.run(Thread.java:662)
     [java] Caused by: java.lang.RuntimeException:
java.lang.OutOfMemoryError: Java heap space
     [java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithoutRetries(HConnectionManager.java:1371)
     [java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1383)
     [java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1381)
     [java] at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
     [java] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
     [java] ... 3 more
     [java] Caused by: java.lang.OutOfMemoryError: Java heap space
     [java] at java.util.Arrays.copyOf(Arrays.java:2786)
     [java] at
java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
     [java] at java.io.DataOutputStream.write(DataOutputStream.java:90)
     [java] at org.apache.hadoop.hbase.client.Put.write(Put.java:422)
     [java] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.writeObject(HbaseObjectWritable.java:464)
     [java] at org.apache.hadoop.hbase.client.Action.write(Action.java:94)
     [java] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.writeObject(HbaseObjectWritable.java:464)
     [java] at
org.apache.hadoop.hbase.client.MultiAction.write(MultiAction.java:102)
     [java] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.writeObject(HbaseObjectWritable.java:464)
     [java] at
org.apache.hadoop.hbase.ipc.Invocation.write(Invocation.java:138)
     [java] at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.java:537)
     [java] at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:879)
     [java] at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
     [java] at $Proxy6.multi(Unknown Source)
     [java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1386)
     [java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1384)
     [java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithoutRetries(HConnectionManager.java:1365)
     [java] ... 7 more

The machine running the HBase has 1G memory only over Ubuntu 10. According
to the book, HBase the Definitive Guide, 64-bit OS and 4GB are required. I
need to upgrade my system?

I change the parameter, HBASE_HEAPSIZE, from 1000 to 2048. The exceptions
still occurred. When the value was changed to 3072, it got the following
errors when I ran start-hbase.sh.

    Could not create the Java virtual machine.
    Could not create the Java virtual machine.
    starting master, logging to
/opt/hbase-0.92.0/bin/../logs/hbase-libing-master-greatfreeinpain.out
    Error occurred during initialization of VM
    Could not reserve enough space for object heap
    Could not create the Java virtual machine.
    localhost: starting regionserver, logging to
/opt/hbase-0.92.0/bin/../logs/hbase-libing-regionserver-greatfreeinpain.out
    localhost: Error occurred during initialization of VM
    localhost: Could not reserve enough space for object heap
    localhost: Could not create the Java virtual machine.

It seems that I need to increase the memory size? Do I need to install a
64-bit Ubuntu Linux?

Thanks so much for your help!

Best regards,
Bing