Issue with bulk loader tool

View: New views
20 Messages — Rating Filter:   Alert me  
< Prev | 1 - 2 | Next >

Issue with bulk loader tool

by Murali Krishna. P :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hi guys,
  I created a table according to hbase-48. A mapreduce job which creates HFiles and then used loadtable.rb script to create the table. Everything worked fine and i was able to scan the table. But when i do a get for a key displayed in the scan output, it is not retrieving the row. shell says 0 row.

 I tried using one reducer to ensure total ordering, but still same issue.

 
My mapper is like:
   context.write(new ImmutableBytesWritable(((Text)key).toString().getBytes()), new KeyValue(((Text)key).toString().getBytes(), "family1".getBytes(),
                    "column1".getBytes(), getValueBytes()));


Please help me investigate this.

Thanks,
Murali Krishna

Re: Issue with bulk loader tool

by stack-3 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Can you send a listing of ".META."?

hbase> scan ".META."

Also, can you bring a region down from hdfs, tar and gzip it, and then put
it someplace I can pull so I can take a look?

Thanks,
St.Ack


On Fri, Oct 30, 2009 at 3:31 AM, Murali Krishna. P
<muralikpbhat@...>wrote:

> Hi guys,
>  I created a table according to hbase-48. A mapreduce job which creates
> HFiles and then used loadtable.rb script to create the table. Everything
> worked fine and i was able to scan the table. But when i do a get for a key
> displayed in the scan output, it is not retrieving the row. shell says 0
> row.
>
>  I tried using one reducer to ensure total ordering, but still same issue.
>
>
> My mapper is like:
>   context.write(new
> ImmutableBytesWritable(((Text)key).toString().getBytes()), new
> KeyValue(((Text)key).toString().getBytes(), "family1".getBytes(),
>                    "column1".getBytes(), getValueBytes()));
>
>
> Please help me investigate this.
>
> Thanks,
> Murali Krishna
>

Re: Issue with bulk loader tool

by Murali Krishna. P :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Some parts of this message have been removed. Learn more about Nabble's security policy.
Attached ".META"

Interesting, I was able to get the row from HTable via java code. But from the shell, still getting following

hbase(main):004:0> get 'TestTable2', 'ffffef95bcbf2638'
0 row(s) in 1.2250 seconds
 
Thanks,
Murali Krishna
 
Thanks,
Murali Krishna



From: stack <stack@...>
To: hbase-user@...
Sent: Fri, 30 October, 2009 8:39:46 PM
Subject: Re: Issue with bulk loader tool

Can you send a listing of ".META."?

hbase> scan ".META."

Also, can you bring a region down from hdfs, tar and gzip it, and then put
it someplace I can pull so I can take a look?

Thanks,
St.Ack


On Fri, Oct 30, 2009 at 3:31 AM, Murali Krishna. P
<muralikpbhat@...>wrote:

> Hi guys,
>  I created a table according to hbase-48. A mapreduce job which creates
> HFiles and then used loadtable.rb script to create the table. Everything
> worked fine and i was able to scan the table. But when i do a get for a key
> displayed in the scan output, it is not retrieving the row. shell says 0
> row.
>
>  I tried using one reducer to ensure total ordering, but still same issue.
>
>
> My mapper is like:
>  context.write(new
> ImmutableBytesWritable(((Text)key).toString().getBytes()), new
> KeyValue(((Text)key).toString().getBytes(), "family1".getBytes(),
>                    "column1".getBytes(), getValueBytes()));
>
>
> Please help me investigate this.
>
> Thanks,
> Murali Krishna
>

hbase(main):002:0* scan ".META."
ROW                          COLUMN+CELL                                                                      
 ImageT,,1256884986587       column=info:regioninfo, timestamp=1256884986985, value=REGION => {NAME => 'ImageT
                             ,,1256884986587', STARTKEY => '', ENDKEY => '03f5d71481d81916', ENCODED => 639463
                             575, TABLE => {{NAME => 'ImageT', FAMILIES => [{NAME => 'image', VERSIONS => '3',
                              COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => '
                             false', BLOCKCACHE => 'true'}]}}                                                
 ImageT,,1256884986587       column=info:server, timestamp=1256884990281, value=76.13.20.50:60020            
 ImageT,,1256884986587       column=info:serverstartcode, timestamp=1256884990281, value=1256882626416        
 ImageT,03f5d71481d81916,125 column=info:regioninfo, timestamp=1256885095013, value=REGION => {NAME => 'ImageT
 6885093957                  ,03f5d71481d81916,1256885093957', STARTKEY => '03f5d71481d81916', ENDKEY => '08c9
                             87e0101ff198', ENCODED => 1448135134, TABLE => {{NAME => 'ImageT', FAMILIES => [{
                             NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLO
                             CKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,03f5d71481d81916,125 column=info:server, timestamp=1256885099301, value=76.13.20.114:60020            
 6885093957                                                                                                  
 ImageT,03f5d71481d81916,125 column=info:serverstartcode, timestamp=1256885099301, value=1256882627359        
 6885093957                                                                                                  
 ImageT,08c987e0101ff198,125 column=info:regioninfo, timestamp=1256885168291, value=REGION => {NAME => 'ImageT
 6885164907                  ,08c987e0101ff198,1256885164907', STARTKEY => '08c987e0101ff198', ENDKEY => '0bff
                             40b2ba034168', ENCODED => 50087697, TABLE => {{NAME => 'ImageT', FAMILIES => [{NA
                             ME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCK
                             SIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                  
 ImageT,08c987e0101ff198,125 column=info:server, timestamp=1256885171799, value=76.13.20.58:60020            
 6885164907                                                                                                  
 ImageT,08c987e0101ff198,125 column=info:serverstartcode, timestamp=1256885171799, value=1256882624416        
 6885164907                                                                                                  
 ImageT,0bff40b2ba034168,125 column=info:regioninfo, timestamp=1256885168295, value=REGION => {NAME => 'ImageT
 6885164907                  ,0bff40b2ba034168,1256885164907', STARTKEY => '0bff40b2ba034168', ENDKEY => '0f83
                             d095e9436ba8', ENCODED => 1894223621, TABLE => {{NAME => 'ImageT', FAMILIES => [{
                             NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLO
                             CKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,0bff40b2ba034168,125 column=info:server, timestamp=1256885171525, value=76.13.20.63:60020            
 6885164907                                                                                                  
 ImageT,0bff40b2ba034168,125 column=info:serverstartcode, timestamp=1256885171525, value=1256882625465        
 6885164907                                                                                                  
 ImageT,0f83d095e9436ba8,125 column=info:regioninfo, timestamp=1256885139707, value=REGION => {NAME => 'ImageT
 6885136168                  ,0f83d095e9436ba8,1256885136168', STARTKEY => '0f83d095e9436ba8', ENDKEY => '1563
                             f3e1be6b0d34', ENCODED => 1707265904, TABLE => {{NAME => 'ImageT', FAMILIES => [{
                             NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLO
                             CKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,0f83d095e9436ba8,125 column=info:server, timestamp=1256885144450, value=76.13.20.114:60020            
 6885136168                                                                                                  
 ImageT,0f83d095e9436ba8,125 column=info:serverstartcode, timestamp=1256885144450, value=1256882627359        
 6885136168                                                                                                  
 ImageT,1563f3e1be6b0d34,125 column=info:regioninfo, timestamp=1256885202224, value=REGION => {NAME => 'ImageT
 6885201491                  ,1563f3e1be6b0d34,1256885201491', STARTKEY => '1563f3e1be6b0d34', ENDKEY => '1cb9
                             681fcdae603c', ENCODED => 137770842, TABLE => {{NAME => 'ImageT', FAMILIES => [{N
                             AME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOC
                             KSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,1563f3e1be6b0d34,125 column=info:server, timestamp=1256885207566, value=76.13.20.112:60020            
 6885201491                                                                                                  
 ImageT,1563f3e1be6b0d34,125 column=info:serverstartcode, timestamp=1256885207566, value=1256882626109        
 6885201491                                                                                                  
 ImageT,1cb9681fcdae603c,125 column=info:regioninfo, timestamp=1256885202227, value=REGION => {NAME => 'ImageT
 6885201491                  ,1cb9681fcdae603c,1256885201491', STARTKEY => '1cb9681fcdae603c', ENDKEY => '2592
                             4867555b537e', ENCODED => 632365571, TABLE => {{NAME => 'ImageT', FAMILIES => [{N
                             AME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOC
                             KSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,1cb9681fcdae603c,125 column=info:server, timestamp=1256885207565, value=76.13.20.112:60020            
 6885201491                                                                                                  
 ImageT,1cb9681fcdae603c,125 column=info:serverstartcode, timestamp=1256885207565, value=1256882626109        
 6885201491                                                                                                  
 ImageT,25924867555b537e,125 column=info:regioninfo, timestamp=1256885182622, value=REGION => {NAME => 'ImageT
 6885180871                  ,25924867555b537e,1256885180871', STARTKEY => '25924867555b537e', ENDKEY => '30a8
                             4e133119d38e', ENCODED => 668319718, TABLE => {{NAME => 'ImageT', FAMILIES => [{N
                             AME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOC
                             KSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,25924867555b537e,125 column=info:server, timestamp=1256885937694, value=76.13.20.62:60020            
 6885180871                                                                                                  
 ImageT,25924867555b537e,125 column=info:serverstartcode, timestamp=1256885937694, value=1256882624666        
 6885180871                                                                                                  
 ImageT,30a84e133119d38e,125 column=info:regioninfo, timestamp=1256885180684, value=REGION => {NAME => 'ImageT
 6885177746                  ,30a84e133119d38e,1256885177746', STARTKEY => '30a84e133119d38e', ENDKEY => '3cb3
                             df2948a7855e', ENCODED => 1441582148, TABLE => {{NAME => 'ImageT', FAMILIES => [{
                             NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLO
                             CKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,30a84e133119d38e,125 column=info:server, timestamp=1256885186602, value=76.13.20.116:60020            
 6885177746                                                                                                  
 ImageT,30a84e133119d38e,125 column=info:serverstartcode, timestamp=1256885186602, value=1256882633981        
 6885177746                                                                                                  
 ImageT,3cb3df2948a7855e,125 column=info:regioninfo, timestamp=1256885216627, value=REGION => {NAME => 'ImageT
 6885214717                  ,3cb3df2948a7855e,1256885214717', STARTKEY => '3cb3df2948a7855e', ENDKEY => '45e4
                             f91b9cc31ea0', ENCODED => 1055393603, TABLE => {{NAME => 'ImageT', FAMILIES => [{
                             NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLO
                             CKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,3cb3df2948a7855e,125 column=info:server, timestamp=1256885219778, value=76.13.20.116:60020            
 6885214717                                                                                                  
 ImageT,3cb3df2948a7855e,125 column=info:serverstartcode, timestamp=1256885219778, value=1256882633981        
 6885214717                                                                                                  
 ImageT,45e4f91b9cc31ea0,125 column=info:regioninfo, timestamp=1256885265634, value=REGION => {NAME => 'ImageT
 6885265515                  ,45e4f91b9cc31ea0,1256885265515', STARTKEY => '45e4f91b9cc31ea0', ENDKEY => '4abb
                             e35563fbd63c', ENCODED => 698170365, TABLE => {{NAME => 'ImageT', FAMILIES => [{N
                             AME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOC
                             KSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,45e4f91b9cc31ea0,125 column=info:server, timestamp=1256885271033, value=76.13.20.58:60020            
 6885265515                                                                                                  
 ImageT,45e4f91b9cc31ea0,125 column=info:serverstartcode, timestamp=1256885271033, value=1256882624416        
 6885265515                                                                                                  
 ImageT,4abbe35563fbd63c,125 column=info:regioninfo, timestamp=1256885265635, value=REGION => {NAME => 'ImageT
 6885265515                  ,4abbe35563fbd63c,1256885265515', STARTKEY => '4abbe35563fbd63c', ENDKEY => '4ecf
                             c14b65c0dc9c', ENCODED => 1536760014, TABLE => {{NAME => 'ImageT', FAMILIES => [{
                             NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLO
                             CKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,4abbe35563fbd63c,125 column=info:server, timestamp=1256906337436, value=76.13.20.50:60020            
 6885265515                                                                                                  
 ImageT,4abbe35563fbd63c,125 column=info:serverstartcode, timestamp=1256906337436, value=1256882626416        
 6885265515                                                                                                  
 ImageT,4ecfc14b65c0dc9c,125 column=info:regioninfo, timestamp=1256885229605, value=REGION => {NAME => 'ImageT
 6885229017                  ,4ecfc14b65c0dc9c,1256885229017', STARTKEY => '4ecfc14b65c0dc9c', ENDKEY => '5226
                             a9ae34e48c14', ENCODED => 1848734906, TABLE => {{NAME => 'ImageT', FAMILIES => [{
                             NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLO
                             CKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,4ecfc14b65c0dc9c,125 column=info:server, timestamp=1256885234701, value=76.13.20.57:60020            
 6885229017                                                                                                  
 ImageT,4ecfc14b65c0dc9c,125 column=info:serverstartcode, timestamp=1256885234701, value=1256882624829        
 6885229017                                                                                                  
 ImageT,5226a9ae34e48c14,125 column=info:regioninfo, timestamp=1256885229606, value=REGION => {NAME => 'ImageT
 6885229017                  ,5226a9ae34e48c14,1256885229017', STARTKEY => '5226a9ae34e48c14', ENDKEY => '5967
                             725249bdf5f8', ENCODED => 1230880207, TABLE => {{NAME => 'ImageT', FAMILIES => [{
                             NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLO
                             CKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,5226a9ae34e48c14,125 column=info:server, timestamp=1256885234702, value=76.13.20.57:60020            
 6885229017                                                                                                  
 ImageT,5226a9ae34e48c14,125 column=info:serverstartcode, timestamp=1256885234702, value=1256882624829        
 6885229017                                                                                                  
 ImageT,5967725249bdf5f8,125 column=info:regioninfo, timestamp=1256885195429, value=REGION => {NAME => 'ImageT
 6885191766                  ,5967725249bdf5f8,1256885191766', STARTKEY => '5967725249bdf5f8', ENDKEY => '63e1
                             4e15f025de3a', ENCODED => 1533462994, TABLE => {{NAME => 'ImageT', FAMILIES => [{
                             NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLO
                             CKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,5967725249bdf5f8,125 column=info:server, timestamp=1256885198611, value=76.13.20.57:60020            
 6885191766                                                                                                  
 ImageT,5967725249bdf5f8,125 column=info:serverstartcode, timestamp=1256885198611, value=1256882624829        
 6885191766                                                                                                  
 ImageT,63e14e15f025de3a,125 column=info:regioninfo, timestamp=1256885225495, value=REGION => {NAME => 'ImageT
 6885225104                  ,63e14e15f025de3a,1256885225104', STARTKEY => '63e14e15f025de3a', ENDKEY => '69d7
                             3d16da0a8bb0', ENCODED => 1196191118, TABLE => {{NAME => 'ImageT', FAMILIES => [{
                             NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLO
                             CKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,63e14e15f025de3a,125 column=info:server, timestamp=1256885228681, value=76.13.20.62:60020            
 6885225104                                                                                                  
 ImageT,63e14e15f025de3a,125 column=info:serverstartcode, timestamp=1256885228681, value=1256882624666        
 6885225104                                                                                                  
 ImageT,69d73d16da0a8bb0,125 column=info:regioninfo, timestamp=1256885225496, value=REGION => {NAME => 'ImageT
 6885225104                  ,69d73d16da0a8bb0,1256885225104', STARTKEY => '69d73d16da0a8bb0', ENDKEY => '6e8e
                             ed317c735f56', ENCODED => 748167878, TABLE => {{NAME => 'ImageT', FAMILIES => [{N
                             AME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOC
                             KSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,69d73d16da0a8bb0,125 column=info:server, timestamp=1256885228683, value=76.13.20.62:60020            
 6885225104                                                                                                  
 ImageT,69d73d16da0a8bb0,125 column=info:serverstartcode, timestamp=1256885228683, value=1256882624666        
 6885225104                                                                                                  
 ImageT,6e8eed317c735f56,125 column=info:regioninfo, timestamp=1256885249207, value=REGION => {NAME => 'ImageT
 6885248639                  ,6e8eed317c735f56,1256885248639', STARTKEY => '6e8eed317c735f56', ENDKEY => '7536
                             775d59b6aac0', ENCODED => 1222575052, TABLE => {{NAME => 'ImageT', FAMILIES => [{
                             NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLO
                             CKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,6e8eed317c735f56,125 column=info:server, timestamp=1256885252751, value=76.13.20.57:60020            
 6885248639                                                                                                  
 ImageT,6e8eed317c735f56,125 column=info:serverstartcode, timestamp=1256885252751, value=1256882624829        
 6885248639                                                                                                  
 ImageT,7536775d59b6aac0,125 column=info:regioninfo, timestamp=1256885249209, value=REGION => {NAME => 'ImageT
 6885248639                  ,7536775d59b6aac0,1256885248639', STARTKEY => '7536775d59b6aac0', ENDKEY => '7904
                             f0f8591ae90a', ENCODED => 614081076, TABLE => {{NAME => 'ImageT', FAMILIES => [{N
                             AME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOC
                             KSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,7536775d59b6aac0,125 column=info:server, timestamp=1256885252752, value=76.13.20.62:60020            
 6885248639                                                                                                  
 ImageT,7536775d59b6aac0,125 column=info:serverstartcode, timestamp=1256885252752, value=1256882624666        
 6885248639                                                                                                  
 ImageT,7904f0f8591ae90a,125 column=info:regioninfo, timestamp=1256885258874, value=REGION => {NAME => 'ImageT
 6885258467                  ,7904f0f8591ae90a,1256885258467', STARTKEY => '7904f0f8591ae90a', ENDKEY => '7d2e
                             f5167e8302de', ENCODED => 1557064883, TABLE => {{NAME => 'ImageT', FAMILIES => [{
                             NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLO
                             CKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,7904f0f8591ae90a,125 column=info:server, timestamp=1256885262010, value=76.13.20.58:60020            
 6885258467                                                                                                  
 ImageT,7904f0f8591ae90a,125 column=info:serverstartcode, timestamp=1256885262010, value=1256882624416        
 6885258467                                                                                                  
 ImageT,7d2ef5167e8302de,125 column=info:regioninfo, timestamp=1256885258874, value=REGION => {NAME => 'ImageT
 6885258467                  ,7d2ef5167e8302de,1256885258467', STARTKEY => '7d2ef5167e8302de', ENDKEY => '841d
                             2cc113068e36', ENCODED => 760090184, TABLE => {{NAME => 'ImageT', FAMILIES => [{N
                             AME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOC
                             KSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,7d2ef5167e8302de,125 column=info:server, timestamp=1256885262009, value=76.13.20.58:60020            
 6885258467                                                                                                  
 ImageT,7d2ef5167e8302de,125 column=info:serverstartcode, timestamp=1256885262009, value=1256882624416        
 6885258467                                                                                                  
 ImageT,841d2cc113068e36,125 column=info:regioninfo, timestamp=1256885201197, value=REGION => {NAME => 'ImageT
 6885195867                  ,841d2cc113068e36,1256885195867', STARTKEY => '841d2cc113068e36', ENDKEY => '8e8d
                             b6081a12c0ea', ENCODED => 633180967, TABLE => {{NAME => 'ImageT', FAMILIES => [{N
                             AME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOC
                             KSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,841d2cc113068e36,125 column=info:server, timestamp=1256885204878, value=76.13.20.58:60020            
 6885195867                                                                                                  
 ImageT,841d2cc113068e36,125 column=info:serverstartcode, timestamp=1256885204878, value=1256882624416        
 6885195867                                                                                                  
 ImageT,8e8db6081a12c0ea,125 column=info:regioninfo, timestamp=1256885253477, value=REGION => {NAME => 'ImageT
 6885253377                  ,8e8db6081a12c0ea,1256885253377', STARTKEY => '8e8db6081a12c0ea', ENDKEY => '9395
                             196b1cf58114', ENCODED => 1747241179, TABLE => {{NAME => 'ImageT', FAMILIES => [{
                             NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLO
                             CKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,8e8db6081a12c0ea,125 column=info:server, timestamp=1256885260835, value=76.13.20.50:60020            
 6885253377                                                                                                  
 ImageT,8e8db6081a12c0ea,125 column=info:serverstartcode, timestamp=1256885260835, value=1256882626416        
 6885253377                                                                                                  
 ImageT,9395196b1cf58114,125 column=info:regioninfo, timestamp=1256885253478, value=REGION => {NAME => 'ImageT
 6885253377                  ,9395196b1cf58114,1256885253377', STARTKEY => '9395196b1cf58114', ENDKEY => '9b4a
                             f5d66e47f826', ENCODED => 687473094, TABLE => {{NAME => 'ImageT', FAMILIES => [{N
                             AME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOC
                             KSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,9395196b1cf58114,125 column=info:server, timestamp=1256885259448, value=76.13.20.56:60020            
 6885253377                                                                                                  
 ImageT,9395196b1cf58114,125 column=info:serverstartcode, timestamp=1256885259448, value=1256882627260        
 6885253377                                                                                                  
 ImageT,9b4af5d66e47f826,125 column=info:regioninfo, timestamp=1256885272641, value=REGION => {NAME => 'ImageT
 6885272540                  ,9b4af5d66e47f826,1256885272540', STARTKEY => '9b4af5d66e47f826', ENDKEY => 'a0dd
                             7fd210889030', ENCODED => 823389088, TABLE => {{NAME => 'ImageT', FAMILIES => [{N
                             AME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOC
                             KSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,9b4af5d66e47f826,125 column=info:server, timestamp=1256906343629, value=76.13.20.58:60020            
 6885272540                                                                                                  
 ImageT,9b4af5d66e47f826,125 column=info:serverstartcode, timestamp=1256906343629, value=1256882624416        
 6885272540                                                                                                  
 ImageT,a0dd7fd210889030,125 column=info:regioninfo, timestamp=1256885272641, value=REGION => {NAME => 'ImageT
 6885272540                  ,a0dd7fd210889030,1256885272540', STARTKEY => 'a0dd7fd210889030', ENDKEY => 'a849
                             ebe8791a992e', ENCODED => 305420769, TABLE => {{NAME => 'ImageT', FAMILIES => [{N
                             AME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOC
                             KSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,a0dd7fd210889030,125 column=info:server, timestamp=1256885277055, value=76.13.20.58:60020            
 6885272540                                                                                                  
 ImageT,a0dd7fd210889030,125 column=info:serverstartcode, timestamp=1256885277055, value=1256882624416        
 6885272540                                                                                                  
 ImageT,a849ebe8791a992e,125 column=info:regioninfo, timestamp=1256885195124, value=REGION => {NAME => 'ImageT
 6885191729                  ,a849ebe8791a992e,1256885191729', STARTKEY => 'a849ebe8791a992e', ENDKEY => 'b014
                             d8b011c1a860', ENCODED => 1377696935, TABLE => {{NAME => 'ImageT', FAMILIES => [{
                             NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLO
                             CKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,a849ebe8791a992e,125 column=info:server, timestamp=1256885199292, value=76.13.20.56:60020            
 6885191729                                                                                                  
 ImageT,a849ebe8791a992e,125 column=info:serverstartcode, timestamp=1256885199292, value=1256882627260        
 6885191729                                                                                                  
 ImageT,b014d8b011c1a860,125 column=info:regioninfo, timestamp=1256885195128, value=REGION => {NAME => 'ImageT
 6885191729                  ,b014d8b011c1a860,1256885191729', STARTKEY => 'b014d8b011c1a860', ENDKEY => 'bda1
                             2e7a08b491b8', ENCODED => 627096885, TABLE => {{NAME => 'ImageT', FAMILIES => [{N
                             AME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOC
                             KSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,b014d8b011c1a860,125 column=info:server, timestamp=1256885199288, value=76.13.20.56:60020            
 6885191729                                                                                                  
 ImageT,b014d8b011c1a860,125 column=info:serverstartcode, timestamp=1256885199288, value=1256882627260        
 6885191729                                                                                                  
 ImageT,bda12e7a08b491b8,125 column=info:regioninfo, timestamp=1256885221234, value=REGION => {NAME => 'ImageT
 6885218911                  ,bda12e7a08b491b8,1256885218911', STARTKEY => 'bda12e7a08b491b8', ENDKEY => 'c81f
                             8a0424c3c6ea', ENCODED => 886819172, TABLE => {{NAME => 'ImageT', FAMILIES => [{N
                             AME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOC
                             KSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,bda12e7a08b491b8,125 column=info:server, timestamp=1256885257815, value=76.13.20.50:60020            
 6885218911                                                                                                  
 ImageT,bda12e7a08b491b8,125 column=info:serverstartcode, timestamp=1256885257815, value=1256882626416        
 6885218911                                                                                                  
 ImageT,c81f8a0424c3c6ea,125 column=info:regioninfo, timestamp=1256885221235, value=REGION => {NAME => 'ImageT
 6885218911                  ,c81f8a0424c3c6ea,1256885218911', STARTKEY => 'c81f8a0424c3c6ea', ENDKEY => 'd298
                             2f8b2b7043e4', ENCODED => 523110207, TABLE => {{NAME => 'ImageT', FAMILIES => [{N
                             AME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOC
                             KSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,c81f8a0424c3c6ea,125 column=info:server, timestamp=1256885226370, value=76.13.20.56:60020            
 6885218911                                                                                                  
 ImageT,c81f8a0424c3c6ea,125 column=info:serverstartcode, timestamp=1256885226370, value=1256882627260        
 6885218911                                                                                                  
 ImageT,d2982f8b2b7043e4,125 column=info:regioninfo, timestamp=1256885244434, value=REGION => {NAME => 'ImageT
 6885242924                  ,d2982f8b2b7043e4,1256885242924', STARTKEY => 'd2982f8b2b7043e4', ENDKEY => 'd84e
                             b4bd63510a86', ENCODED => 1621311569, TABLE => {{NAME => 'ImageT', FAMILIES => [{
                             NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLO
                             CKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,d2982f8b2b7043e4,125 column=info:server, timestamp=1256885249986, value=76.13.20.58:60020            
 6885242924                                                                                                  
 ImageT,d2982f8b2b7043e4,125 column=info:serverstartcode, timestamp=1256885249986, value=1256882624416        
 6885242924                                                                                                  
 ImageT,d84eb4bd63510a86,125 column=info:regioninfo, timestamp=1256885244435, value=REGION => {NAME => 'ImageT
 6885242924                  ,d84eb4bd63510a86,1256885242924', STARTKEY => 'd84eb4bd63510a86', ENDKEY => 'dec5
                             4131f34f07f4', ENCODED => 1019397957, TABLE => {{NAME => 'ImageT', FAMILIES => [{
                             NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLO
                             CKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,d84eb4bd63510a86,125 column=info:server, timestamp=1256885249899, value=76.13.20.116:60020            
 6885242924                                                                                                  
 ImageT,d84eb4bd63510a86,125 column=info:serverstartcode, timestamp=1256885249899, value=1256882633981        
 6885242924                                                                                                  
 ImageT,dec54131f34f07f4,125 column=info:regioninfo, timestamp=1256885261086, value=REGION => {NAME => 'ImageT
 6885260266                  ,dec54131f34f07f4,1256885260266', STARTKEY => 'dec54131f34f07f4', ENDKEY => 'e2e7
                             4c148c200a80', ENCODED => 288282898, TABLE => {{NAME => 'ImageT', FAMILIES => [{N
                             AME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOC
                             KSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,dec54131f34f07f4,125 column=info:server, timestamp=1256885265019, value=76.13.20.58:60020            
 6885260266                                                                                                  
 ImageT,dec54131f34f07f4,125 column=info:serverstartcode, timestamp=1256885265019, value=1256882624416        
 6885260266                                                                                                  
 ImageT,e2e74c148c200a80,125 column=info:regioninfo, timestamp=1256885261086, value=REGION => {NAME => 'ImageT
 6885260266                  ,e2e74c148c200a80,1256885260266', STARTKEY => 'e2e74c148c200a80', ENDKEY => 'ec9d
                             9ec033589756', ENCODED => 1108705048, TABLE => {{NAME => 'ImageT', FAMILIES => [{
                             NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLO
                             CKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,e2e74c148c200a80,125 column=info:server, timestamp=1256885264940, value=76.13.20.116:60020            
 6885260266                                                                                                  
 ImageT,e2e74c148c200a80,125 column=info:serverstartcode, timestamp=1256885264940, value=1256882633981        
 6885260266                                                                                                  
 ImageT,ec9d9ec033589756,125 column=info:regioninfo, timestamp=1256885197340, value=REGION => {NAME => 'ImageT
 6885195678                  ,ec9d9ec033589756,1256885195678', STARTKEY => 'ec9d9ec033589756', ENDKEY => 'f074
                             d6629be5ebea', ENCODED => 92716390, TABLE => {{NAME => 'ImageT', FAMILIES => [{NA
                             ME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCK
                             SIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                  
 ImageT,ec9d9ec033589756,125 column=info:server, timestamp=1256885201640, value=76.13.20.63:60020            
 6885195678                                                                                                  
 ImageT,ec9d9ec033589756,125 column=info:serverstartcode, timestamp=1256885201640, value=1256882625465        
 6885195678                                                                                                  
 ImageT,f074d6629be5ebea,125 column=info:regioninfo, timestamp=1256885206092, value=REGION => {NAME => 'ImageT
 6885205591                  ,f074d6629be5ebea,1256885205591', STARTKEY => 'f074d6629be5ebea', ENDKEY => 'f7c1
                             d53736e1c6e2', ENCODED => 794820350, TABLE => {{NAME => 'ImageT', FAMILIES => [{N
                             AME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOC
                             KSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,f074d6629be5ebea,125 column=info:server, timestamp=1256885210677, value=76.13.20.114:60020            
 6885205591                                                                                                  
 ImageT,f074d6629be5ebea,125 column=info:serverstartcode, timestamp=1256885210677, value=1256882627359        
 6885205591                                                                                                  
 ImageT,f7c1d53736e1c6e2,125 column=info:regioninfo, timestamp=1256885240594, value=REGION => {NAME => 'ImageT
 6885239703                  ,f7c1d53736e1c6e2,1256885239703', STARTKEY => 'f7c1d53736e1c6e2', ENDKEY => 'fc52
                             06fd1a7cd4fe', ENCODED => 1873557108, TABLE => {{NAME => 'ImageT', FAMILIES => [{
                             NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLO
                             CKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 ImageT,f7c1d53736e1c6e2,125 column=info:server, timestamp=1256885243857, value=76.13.20.116:60020            
 6885239703                                                                                                  
 ImageT,f7c1d53736e1c6e2,125 column=info:serverstartcode, timestamp=1256885243857, value=1256882633981        
 6885239703                                                                                                  
 ImageT,fc5206fd1a7cd4fe,125 column=info:regioninfo, timestamp=1256885240594, value=REGION => {NAME => 'ImageT
 6885239703                  ,fc5206fd1a7cd4fe,1256885239703', STARTKEY => 'fc5206fd1a7cd4fe', ENDKEY => '', E
                             NCODED => 609893267, TABLE => {{NAME => 'ImageT', FAMILIES => [{NAME => 'image',
                             VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536'
                             , IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                                
 ImageT,fc5206fd1a7cd4fe,125 column=info:server, timestamp=1256885243775, value=76.13.20.114:60020            
 6885239703                                                                                                  
 ImageT,fc5206fd1a7cd4fe,125 column=info:serverstartcode, timestamp=1256885243775, value=1256882627359        
 6885239703                                                                                                  
 TestTable,,1256882760704    column=info:regioninfo, timestamp=1256882760761, value=REGION => {NAME => 'TestTa
                             ble,,1256882760704', STARTKEY => '', ENDKEY => '2b5c45c26fc74b94', ENCODED => 727
                             538258, TABLE => {{NAME => 'TestTable', FAMILIES => [{NAME => 'image', VERSIONS =
                             > '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMOR
                             Y => 'false', BLOCKCACHE => 'true'}]}}                                          
 TestTable,,1256882760704    column=info:server, timestamp=1256885279702, value=76.13.20.112:60020            
 TestTable,,1256882760704    column=info:serverstartcode, timestamp=1256885279702, value=1256882626109        
 TestTable,2b5c45c26fc74b94, column=info:regioninfo, timestamp=1256882760766, value=REGION => {NAME => 'TestTa
 1256882760723               ble,2b5c45c26fc74b94,1256882760723', STARTKEY => '2b5c45c26fc74b94', ENDKEY => '2
                             b662b322e4943a8', ENCODED => 1842655645, TABLE => {{NAME => 'TestTable', FAMILIES
                              => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647
                             ', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}          
 TestTable,2b5c45c26fc74b94, column=info:server, timestamp=1256882808403, value=76.13.20.56:60020            
 1256882760723                                                                                                
 TestTable,2b5c45c26fc74b94, column=info:serverstartcode, timestamp=1256882808403, value=1256882627260        
 1256882760723                                                                                                
 TestTable,2b662b322e4943a8, column=info:regioninfo, timestamp=1256882760785, value=REGION => {NAME => 'TestTa
 1256882760743               ble,2b662b322e4943a8,1256882760743', STARTKEY => '2b662b322e4943a8', ENDKEY => '2
                             b7b62a95e3699cc', ENCODED => 78224151, TABLE => {{NAME => 'TestTable', FAMILIES =
                             > [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647',
                              BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}            
 TestTable,2b662b322e4943a8, column=info:server, timestamp=1256882808982, value=76.13.20.57:60020            
 1256882760743                                                                                                
 TestTable,2b662b322e4943a8, column=info:serverstartcode, timestamp=1256882808982, value=1256882624829        
 1256882760743                                                                                                
 TestTable,2b7b62a95e3699cc, column=info:regioninfo, timestamp=1256882760788, value=REGION => {NAME => 'TestTa
 1256882760747               ble,2b7b62a95e3699cc,1256882760747', STARTKEY => '2b7b62a95e3699cc', ENDKEY => '2
                             bb81b408603973e', ENCODED => 609616255, TABLE => {{NAME => 'TestTable', FAMILIES
                             => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647'
                             , BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}          
 TestTable,2b7b62a95e3699cc, column=info:server, timestamp=1256884554707, value=76.13.20.50:60020            
 1256882760747                                                                                                
 TestTable,2b7b62a95e3699cc, column=info:serverstartcode, timestamp=1256884554707, value=1256882626416        
 1256882760747                                                                                                
 TestTable,2bb81b408603973e, column=info:regioninfo, timestamp=1256882760792, value=REGION => {NAME => 'TestTa
 1256882760750               ble,2bb81b408603973e,1256882760750', STARTKEY => '2bb81b408603973e', ENDKEY => '5
                             6f33bf5d7d70e4a', ENCODED => 1795819235, TABLE => {{NAME => 'TestTable', FAMILIES
                              => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647
                             ', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}          
 TestTable,2bb81b408603973e, column=info:server, timestamp=1256882807603, value=76.13.20.50:60020            
 1256882760750                                                                                                
 TestTable,2bb81b408603973e, column=info:serverstartcode, timestamp=1256882807603, value=1256882626416        
 1256882760750                                                                                                
 TestTable,56f33bf5d7d70e4a, column=info:regioninfo, timestamp=1256882760798, value=REGION => {NAME => 'TestTa
 1256882760754               ble,56f33bf5d7d70e4a,1256882760754', STARTKEY => '56f33bf5d7d70e4a', ENDKEY => '5
                             6fa2c490e8ba15a', ENCODED => 40140990, TABLE => {{NAME => 'TestTable', FAMILIES =
                             > [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647',
                              BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}            
 TestTable,56f33bf5d7d70e4a, column=info:server, timestamp=1256882859840, value=76.13.20.114:60020            
 1256882760754                                                                                                
 TestTable,56f33bf5d7d70e4a, column=info:serverstartcode, timestamp=1256882859840, value=1256882627359        
 1256882760754                                                                                                
 TestTable,56fa2c490e8ba15a, column=info:regioninfo, timestamp=1256882760802, value=REGION => {NAME => 'TestTa
 1256882760760               ble,56fa2c490e8ba15a,1256882760760', STARTKEY => '56fa2c490e8ba15a', ENDKEY => '5
                             7407d9d445b0ad2', ENCODED => 1880544407, TABLE => {{NAME => 'TestTable', FAMILIES
                              => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647
                             ', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}          
 TestTable,56fa2c490e8ba15a, column=info:server, timestamp=1256882871902, value=76.13.20.62:60020            
 1256882760760                                                                                                
 TestTable,56fa2c490e8ba15a, column=info:serverstartcode, timestamp=1256882871902, value=1256882624666        
 1256882760760                                                                                                
 TestTable,57407d9d445b0ad2, column=info:regioninfo, timestamp=1256882760806, value=REGION => {NAME => 'TestTa
 1256882760764               ble,57407d9d445b0ad2,1256882760764', STARTKEY => '57407d9d445b0ad2', ENDKEY => '5
                             744e0d10646819a', ENCODED => 1100854720, TABLE => {{NAME => 'TestTable', FAMILIES
                              => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647
                             ', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}          
 TestTable,57407d9d445b0ad2, column=info:server, timestamp=1256884564730, value=76.13.20.62:60020            
 1256882760764                                                                                                
 TestTable,57407d9d445b0ad2, column=info:serverstartcode, timestamp=1256884564730, value=1256882624666        
 1256882760764                                                                                                
 TestTable,5744e0d10646819a, column=info:regioninfo, timestamp=1256882760810, value=REGION => {NAME => 'TestTa
 1256882760768               ble,5744e0d10646819a,1256882760768', STARTKEY => '5744e0d10646819a', ENDKEY => '8
                             26b66ae73fe2c7a', ENCODED => 1951484261, TABLE => {{NAME => 'TestTable', FAMILIES
                              => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647
                             ', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}          
 TestTable,5744e0d10646819a, column=info:server, timestamp=1256882816618, value=76.13.20.50:60020            
 1256882760768                                                                                                
 TestTable,5744e0d10646819a, column=info:serverstartcode, timestamp=1256882816618, value=1256882626416        
 1256882760768                                                                                                
 TestTable,826b66ae73fe2c7a, column=info:regioninfo, timestamp=1256882760814, value=REGION => {NAME => 'TestTa
 1256882760772               ble,826b66ae73fe2c7a,1256882760772', STARTKEY => '826b66ae73fe2c7a', ENDKEY => '8
                             285a3fc5e78049c', ENCODED => 246699909, TABLE => {{NAME => 'TestTable', FAMILIES
                             => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647'
                             , BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}          
 TestTable,826b66ae73fe2c7a, column=info:server, timestamp=1256882811822, value=76.13.20.62:60020            
 1256882760772                                                                                                
 TestTable,826b66ae73fe2c7a, column=info:serverstartcode, timestamp=1256882811822, value=1256882624666        
 1256882760772                                                                                                
 TestTable,8285a3fc5e78049c, column=info:regioninfo, timestamp=1256882760819, value=REGION => {NAME => 'TestTa
 1256882760776               ble,8285a3fc5e78049c,1256882760776', STARTKEY => '8285a3fc5e78049c', ENDKEY => '8
                             2d8fb79b938efa6', ENCODED => 378816599, TABLE => {{NAME => 'TestTable', FAMILIES
                             => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647'
                             , BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}          
 TestTable,8285a3fc5e78049c, column=info:server, timestamp=1256882835804, value=76.13.20.114:60020            
 1256882760776                                                                                                
 TestTable,8285a3fc5e78049c, column=info:serverstartcode, timestamp=1256882835804, value=1256882627359        
 1256882760776                                                                                                
 TestTable,82d8fb79b938efa6, column=info:regioninfo, timestamp=1256882760841, value=REGION => {NAME => 'TestTa
 1256882760781               ble,82d8fb79b938efa6,1256882760781', STARTKEY => '82d8fb79b938efa6', ENDKEY => '8
                             2f90ecda47d7da8', ENCODED => 183751274, TABLE => {{NAME => 'TestTable', FAMILIES
                             => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647'
                             , BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}          
 TestTable,82d8fb79b938efa6, column=info:server, timestamp=1256882872057, value=76.13.20.57:60020            
 1256882760781                                                                                                
 TestTable,82d8fb79b938efa6, column=info:serverstartcode, timestamp=1256882872057, value=1256882624829        
 1256882760781                                                                                                
 TestTable,82f90ecda47d7da8, column=info:regioninfo, timestamp=1256882760845, value=REGION => {NAME => 'TestTa
 1256882760803               ble,82f90ecda47d7da8,1256882760803', STARTKEY => '82f90ecda47d7da8', ENDKEY => 'a
                             e3df481272c53da', ENCODED => 650489927, TABLE => {{NAME => 'TestTable', FAMILIES
                             => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647'
                             , BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}          
 TestTable,82f90ecda47d7da8, column=info:server, timestamp=1256882865893, value=76.13.20.62:60020            
 1256882760803                                                                                                
 TestTable,82f90ecda47d7da8, column=info:serverstartcode, timestamp=1256882865893, value=1256882624666        
 1256882760803                                                                                                
 TestTable,ae3df481272c53da, column=info:regioninfo, timestamp=1256882760852, value=REGION => {NAME => 'TestTa
 1256882760807               ble,ae3df481272c53da,1256882760807', STARTKEY => 'ae3df481272c53da', ENDKEY => 'a
                             e4252c4639c74ba', ENCODED => 391625222, TABLE => {{NAME => 'TestTable', FAMILIES
                             => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647'
                             , BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}          
 TestTable,ae3df481272c53da, column=info:server, timestamp=1256884553355, value=76.13.20.56:60020            
 1256882760807                                                                                                
 TestTable,ae3df481272c53da, column=info:serverstartcode, timestamp=1256884553355, value=1256882627260        
 1256882760807                                                                                                
 TestTable,ae4252c4639c74ba, column=info:regioninfo, timestamp=1256882760856, value=REGION => {NAME => 'TestTa
 1256882760814               ble,ae4252c4639c74ba,1256882760814', STARTKEY => 'ae4252c4639c74ba', ENDKEY => 'a
                             e541400d28fa93e', ENCODED => 1786514437, TABLE => {{NAME => 'TestTable', FAMILIES
                              => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647
                             ', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}          
 TestTable,ae4252c4639c74ba, column=info:server, timestamp=1256882809077, value=76.13.20.116:60020            
 1256882760814                                                                                                
 TestTable,ae4252c4639c74ba, column=info:serverstartcode, timestamp=1256882809077, value=1256882633981        
 1256882760814                                                                                                
 TestTable,ae541400d28fa93e, column=info:regioninfo, timestamp=1256882760864, value=REGION => {NAME => 'TestTa
 1256882760819               ble,ae541400d28fa93e,1256882760819', STARTKEY => 'ae541400d28fa93e', ENDKEY => 'a
                             e68889a04ffbc10', ENCODED => 917437746, TABLE => {{NAME => 'TestTable', FAMILIES
                             => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647'
                             , BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}          
 TestTable,ae541400d28fa93e, column=info:server, timestamp=1256882807389, value=76.13.20.112:60020            
 1256882760819                                                                                                
 TestTable,ae541400d28fa93e, column=info:serverstartcode, timestamp=1256882807389, value=1256882626109        
 1256882760819                                                                                                
 TestTable,ae68889a04ffbc10, column=info:regioninfo, timestamp=1256882760869, value=REGION => {NAME => 'TestTa
 1256882760826               ble,ae68889a04ffbc10,1256882760826', STARTKEY => 'ae68889a04ffbc10', ENDKEY => 'd
                             9e649f8f718a852', ENCODED => 1352908714, TABLE => {{NAME => 'TestTable', FAMILIES
                              => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647
                             ', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}          
 TestTable,ae68889a04ffbc10, column=info:server, timestamp=1256884553355, value=76.13.20.56:60020            
 1256882760826                                                                                                
 TestTable,ae68889a04ffbc10, column=info:serverstartcode, timestamp=1256884553355, value=1256882627260        
 1256882760826                                                                                                
 TestTable,d9e649f8f718a852, column=info:regioninfo, timestamp=1256882760873, value=REGION => {NAME => 'TestTa
 1256882760831               ble,d9e649f8f718a852,1256882760831', STARTKEY => 'd9e649f8f718a852', ENDKEY => 'd
                             a0ebadcfc066410', ENCODED => 301922464, TABLE => {{NAME => 'TestTable', FAMILIES
                             => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647'
                             , BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}          
 TestTable,d9e649f8f718a852, column=info:server, timestamp=1256882809078, value=76.13.20.116:60020            
 1256882760831                                                                                                
 TestTable,d9e649f8f718a852, column=info:serverstartcode, timestamp=1256882809078, value=1256882633981        
 1256882760831                                                                                                
 TestTable,da0ebadcfc066410, column=info:regioninfo, timestamp=1256882760876, value=REGION => {NAME => 'TestTa
 1256882760835               ble,da0ebadcfc066410,1256882760835', STARTKEY => 'da0ebadcfc066410', ENDKEY => 'd
                             a1305c9a54be076', ENCODED => 1412596867, TABLE => {{NAME => 'TestTable', FAMILIES
                              => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647
                             ', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}          
 TestTable,da0ebadcfc066410, column=info:server, timestamp=1256884564704, value=76.13.20.57:60020            
 1256882760835                                                                                                
 TestTable,da0ebadcfc066410, column=info:serverstartcode, timestamp=1256884564704, value=1256882624829        
 1256882760835                                                                                                
 TestTable,da1305c9a54be076, column=info:regioninfo, timestamp=1256882760880, value=REGION => {NAME => 'TestTa
 1256882760838               ble,da1305c9a54be076,1256882760838', STARTKEY => 'da1305c9a54be076', ENDKEY => 'd
                             a28109015f5307a', ENCODED => 183286062, TABLE => {{NAME => 'TestTable', FAMILIES
                             => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647'
                             , BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}          
 TestTable,da1305c9a54be076, column=info:server, timestamp=1256882866049, value=76.13.20.57:60020            
 1256882760838                                                                                                
 TestTable,da1305c9a54be076, column=info:serverstartcode, timestamp=1256882866049, value=1256882624829        
 1256882760838                                                                                                
 TestTable,da28109015f5307a, column=info:regioninfo, timestamp=1256882760883, value=REGION => {NAME => 'TestTa
 1256882760842               ble,da28109015f5307a,1256882760842', STARTKEY => 'da28109015f5307a', ENDKEY => 'f
                             fffac8dec34b3a0', ENCODED => 558976149, TABLE => {{NAME => 'TestTable', FAMILIES
                             => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647'
                             , BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}          
 TestTable,da28109015f5307a, column=info:server, timestamp=1256882808983, value=76.13.20.57:60020            
 1256882760842                                                                                                
 TestTable,da28109015f5307a, column=info:serverstartcode, timestamp=1256882808983, value=1256882624829        
 1256882760842                                                                                                
 TestTable,ffffac8dec34b3a0, column=info:regioninfo, timestamp=1256882760886, value=REGION => {NAME => 'TestTa
 1256882760845               ble,ffffac8dec34b3a0,1256882760845', STARTKEY => 'ffffac8dec34b3a0', ENDKEY => 'f
                             fffc088d034902a', ENCODED => 1134025611, TABLE => {{NAME => 'TestTable', FAMILIES
                              => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647
                             ', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}          
 TestTable,ffffac8dec34b3a0, column=info:server, timestamp=1256884564731, value=76.13.20.62:60020            
 1256882760845                                                                                                
 TestTable,ffffac8dec34b3a0, column=info:serverstartcode, timestamp=1256884564731, value=1256882624666        
 1256882760845                                                                                                
 TestTable,ffffc088d034902a, column=info:regioninfo, timestamp=1256882760890, value=REGION => {NAME => 'TestTa
 1256882760848               ble,ffffc088d034902a,1256882760848', STARTKEY => 'ffffc088d034902a', ENDKEY => 'f
                             fffef95bcbf2638', ENCODED => 1820800522, TABLE => {{NAME => 'TestTable', FAMILIES
                              => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647
                             ', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}          
 TestTable,ffffc088d034902a, column=info:server, timestamp=1256882808823, value=76.13.20.62:60020            
 1256882760848                                                                                                
 TestTable,ffffc088d034902a, column=info:serverstartcode, timestamp=1256882808823, value=1256882624666        
 1256882760848                                                                                                
 TestTable,ffffef95bcbf2638, column=info:regioninfo, timestamp=1256882760893, value=REGION => {NAME => 'TestTa
 1256882760852               ble,ffffef95bcbf2638,1256882760852', STARTKEY => 'ffffef95bcbf2638', ENDKEY => ''
                             , ENCODED => 35149396, TABLE => {{NAME => 'TestTable', FAMILIES => [{NAME => 'ima
                             ge', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '6
                             5536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                            
 TestTable,ffffef95bcbf2638, column=info:server, timestamp=1256882810607, value=76.13.20.50:60020            
 1256882760852                                                                                                
 TestTable,ffffef95bcbf2638, column=info:serverstartcode, timestamp=1256882810607, value=1256882626416        
 1256882760852                                                                                                
 TestTable1,,1256884494497   column=info:regioninfo, timestamp=1256884494588, value=REGION => {NAME => 'TestTa
                             ble1,,1256884494497', STARTKEY => '', ENDKEY => '0ae6b8a96914acc8', ENCODED => 86
                             9917892, TABLE => {{NAME => 'TestTable1', FAMILIES => [{NAME => 'image', VERSIONS
                              => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEM
                             ORY => 'false', BLOCKCACHE => 'true'}]}}                                        
 TestTable1,,1256884494497   column=info:server, timestamp=1256884548697, value=76.13.20.50:60020            
 TestTable1,,1256884494497   column=info:serverstartcode, timestamp=1256884548697, value=1256882626416        
 TestTable1,0ae6b8a96914acc8 column=info:regioninfo, timestamp=1256884494593, value=REGION => {NAME => 'TestTa
 ,1256884494564              ble1,0ae6b8a96914acc8,1256884494564', STARTKEY => '0ae6b8a96914acc8', ENDKEY => '
                             15c8d306751425a0', ENCODED => 387303415, TABLE => {{NAME => 'TestTable1', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable1,0ae6b8a96914acc8 column=info:server, timestamp=1256884564706, value=76.13.20.112:60020            
 ,1256884494564                                                                                              
 TestTable1,0ae6b8a96914acc8 column=info:serverstartcode, timestamp=1256884564706, value=1256882626109        
 ,1256884494564                                                                                              
 TestTable1,15c8d306751425a0 column=info:regioninfo, timestamp=1256884494597, value=REGION => {NAME => 'TestTa
 ,1256884494569              ble1,15c8d306751425a0,1256884494569', STARTKEY => '15c8d306751425a0', ENDKEY => '
                             20a4755f6984c8a4', ENCODED => 1173329143, TABLE => {{NAME => 'TestTable1', FAMILI
                             ES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21474836
                             47', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable1,15c8d306751425a0 column=info:server, timestamp=1256884549669, value=76.13.20.63:60020            
 ,1256884494569                                                                                              
 TestTable1,15c8d306751425a0 column=info:serverstartcode, timestamp=1256884549669, value=1256882625465        
 ,1256884494569                                                                                              
 TestTable1,20a4755f6984c8a4 column=info:regioninfo, timestamp=1256884494601, value=REGION => {NAME => 'TestTa
 ,1256884494572              ble1,20a4755f6984c8a4,1256884494572', STARTKEY => '20a4755f6984c8a4', ENDKEY => '
                             2b7de9fc0c1671fc', ENCODED => 845027916, TABLE => {{NAME => 'TestTable1', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable1,20a4755f6984c8a4 column=info:server, timestamp=1256884558695, value=76.13.20.112:60020            
 ,1256884494572                                                                                              
 TestTable1,20a4755f6984c8a4 column=info:serverstartcode, timestamp=1256884558695, value=1256882626109        
 ,1256884494572                                                                                              
 TestTable1,2b7de9fc0c1671fc column=info:regioninfo, timestamp=1256884494605, value=REGION => {NAME => 'TestTa
 ,1256884494576              ble1,2b7de9fc0c1671fc,1256884494576', STARTKEY => '2b7de9fc0c1671fc', ENDKEY => '
                             3666d964ee23718a', ENCODED => 375539670, TABLE => {{NAME => 'TestTable1', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable1,2b7de9fc0c1671fc column=info:server, timestamp=1256884549685, value=76.13.20.57:60020            
 ,1256884494576                                                                                              
 TestTable1,2b7de9fc0c1671fc column=info:serverstartcode, timestamp=1256884549685, value=1256882624829        
 ,1256884494576                                                                                              
 TestTable1,3666d964ee23718a column=info:regioninfo, timestamp=1256884494616, value=REGION => {NAME => 'TestTa
 ,1256884494581              ble1,3666d964ee23718a,1256884494581', STARTKEY => '3666d964ee23718a', ENDKEY => '
                             41472e380ecd4d22', ENCODED => 946903562, TABLE => {{NAME => 'TestTable1', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable1,3666d964ee23718a column=info:server, timestamp=1256884558696, value=76.13.20.112:60020            
 ,1256884494581                                                                                              
 TestTable1,3666d964ee23718a column=info:serverstartcode, timestamp=1256884558696, value=1256882626109        
 ,1256884494581                                                                                              
 TestTable1,41472e380ecd4d22 column=info:regioninfo, timestamp=1256884494620, value=REGION => {NAME => 'TestTa
 ,1256884494592              ble1,41472e380ecd4d22,1256884494592', STARTKEY => '41472e380ecd4d22', ENDKEY => '
                             4c218f292e94a074', ENCODED => 717276198, TABLE => {{NAME => 'TestTable1', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable1,41472e380ecd4d22 column=info:server, timestamp=1256884549683, value=76.13.20.112:60020            
 ,1256884494592                                                                                              
 TestTable1,41472e380ecd4d22 column=info:serverstartcode, timestamp=1256884549683, value=1256882626109        
 ,1256884494592                                                                                              
 TestTable1,4c218f292e94a074 column=info:regioninfo, timestamp=1256884494624, value=REGION => {NAME => 'TestTa
 ,1256884494595              ble1,4c218f292e94a074,1256884494595', STARTKEY => '4c218f292e94a074', ENDKEY => '
                             571cb9d584b08a1c', ENCODED => 167834146, TABLE => {{NAME => 'TestTable1', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable1,4c218f292e94a074 column=info:server, timestamp=1256884558698, value=76.13.20.112:60020            
 ,1256884494595                                                                                              
 TestTable1,4c218f292e94a074 column=info:serverstartcode, timestamp=1256884558698, value=1256882626109        
 ,1256884494595                                                                                              
 TestTable1,571cb9d584b08a1c column=info:regioninfo, timestamp=1256884494628, value=REGION => {NAME => 'TestTa
 ,1256884494600              ble1,571cb9d584b08a1c,1256884494600', STARTKEY => '571cb9d584b08a1c', ENDKEY => '
                             61f8562a687a4ef0', ENCODED => 1725664881, TABLE => {{NAME => 'TestTable1', FAMILI
                             ES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21474836
                             47', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable1,571cb9d584b08a1c column=info:server, timestamp=1256884547348, value=76.13.20.56:60020            
 ,1256884494600                                                                                              
 TestTable1,571cb9d584b08a1c column=info:serverstartcode, timestamp=1256884547348, value=1256882627260        
 ,1256884494600                                                                                              
 TestTable1,61f8562a687a4ef0 column=info:regioninfo, timestamp=1256884494632, value=REGION => {NAME => 'TestTa
 ,1256884494603              ble1,61f8562a687a4ef0,1256884494603', STARTKEY => '61f8562a687a4ef0', ENDKEY => '
                             6cd6cb3fac9bfe8e', ENCODED => 222355120, TABLE => {{NAME => 'TestTable1', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable1,61f8562a687a4ef0 column=info:server, timestamp=1256885278870, value=76.13.20.50:60020            
 ,1256884494603                                                                                              
 TestTable1,61f8562a687a4ef0 column=info:serverstartcode, timestamp=1256885278870, value=1256882626416        
 ,1256884494603                                                                                              
 TestTable1,6cd6cb3fac9bfe8e column=info:regioninfo, timestamp=1256884494636, value=REGION => {NAME => 'TestTa
 ,1256884494607              ble1,6cd6cb3fac9bfe8e,1256884494607', STARTKEY => '6cd6cb3fac9bfe8e', ENDKEY => '
                             77b0388e552b68dc', ENCODED => 1852820124, TABLE => {{NAME => 'TestTable1', FAMILI
                             ES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21474836
                             47', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable1,6cd6cb3fac9bfe8e column=info:server, timestamp=1256884548696, value=76.13.20.50:60020            
 ,1256884494607                                                                                              
 TestTable1,6cd6cb3fac9bfe8e column=info:serverstartcode, timestamp=1256884548696, value=1256882626416        
 ,1256884494607                                                                                              
 TestTable1,77b0388e552b68dc column=info:regioninfo, timestamp=1256884494639, value=REGION => {NAME => 'TestTa
 ,1256884494611              ble1,77b0388e552b68dc,1256884494611', STARTKEY => '77b0388e552b68dc', ENDKEY => '
                             82aeb97287e15180', ENCODED => 798859786, TABLE => {{NAME => 'TestTable1', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable1,77b0388e552b68dc column=info:server, timestamp=1256884549654, value=76.13.20.116:60020            
 ,1256884494611                                                                                              
 TestTable1,77b0388e552b68dc column=info:serverstartcode, timestamp=1256884549654, value=1256882633981        
 ,1256884494611                                                                                              
 TestTable1,82aeb97287e15180 column=info:regioninfo, timestamp=1256884494645, value=REGION => {NAME => 'TestTa
 ,1256884494617              ble1,82aeb97287e15180,1256884494617', STARTKEY => '82aeb97287e15180', ENDKEY => '
                             8d80de2029040fb2', ENCODED => 898219140, TABLE => {{NAME => 'TestTable1', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable1,82aeb97287e15180 column=info:server, timestamp=1256884549661, value=76.13.20.114:60020            
 ,1256884494617                                                                                              
 TestTable1,82aeb97287e15180 column=info:serverstartcode, timestamp=1256884549661, value=1256882627359        
 ,1256884494617                                                                                              
 TestTable1,8d80de2029040fb2 column=info:regioninfo, timestamp=1256884494651, value=REGION => {NAME => 'TestTa
 ,1256884494621              ble1,8d80de2029040fb2,1256884494621', STARTKEY => '8d80de2029040fb2', ENDKEY => '
                             986db344e405461a', ENCODED => 1694923622, TABLE => {{NAME => 'TestTable1', FAMILI
                             ES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21474836
                             47', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable1,8d80de2029040fb2 column=info:server, timestamp=1256884549663, value=76.13.20.63:60020            
 ,1256884494621                                                                                              
 TestTable1,8d80de2029040fb2 column=info:serverstartcode, timestamp=1256884549663, value=1256882625465        
 ,1256884494621                                                                                              
 TestTable1,986db344e405461a column=info:regioninfo, timestamp=1256884494656, value=REGION => {NAME => 'TestTa
 ,1256884494627              ble1,986db344e405461a,1256884494627', STARTKEY => '986db344e405461a', ENDKEY => '
                             a3650c660e297076', ENCODED => 153022928, TABLE => {{NAME => 'TestTable1', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable1,986db344e405461a column=info:server, timestamp=1256884547347, value=76.13.20.56:60020            
 ,1256884494627                                                                                              
 TestTable1,986db344e405461a column=info:serverstartcode, timestamp=1256884547347, value=1256882627260        
 ,1256884494627                                                                                              
 TestTable1,a3650c660e297076 column=info:regioninfo, timestamp=1256884494663, value=REGION => {NAME => 'TestTa
 ,1256884494633              ble1,a3650c660e297076,1256884494633', STARTKEY => 'a3650c660e297076', ENDKEY => '
                             ae4f6ea0ff9ec432', ENCODED => 445131689, TABLE => {{NAME => 'TestTable1', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable1,a3650c660e297076 column=info:server, timestamp=1256884547194, value=76.13.20.58:60020            
 ,1256884494633                                                                                              
 TestTable1,a3650c660e297076 column=info:serverstartcode, timestamp=1256884547194, value=1256882624416        
 ,1256884494633                                                                                              
 TestTable1,ae4f6ea0ff9ec432 column=info:regioninfo, timestamp=1256884494668, value=REGION => {NAME => 'TestTa
 ,1256884494641              ble1,ae4f6ea0ff9ec432,1256884494641', STARTKEY => 'ae4f6ea0ff9ec432', ENDKEY => '
                             b944f96a7a2def60', ENCODED => 624258913, TABLE => {{NAME => 'TestTable1', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable1,ae4f6ea0ff9ec432 column=info:server, timestamp=1256884549667, value=76.13.20.63:60020            
 ,1256884494641                                                                                              
 TestTable1,ae4f6ea0ff9ec432 column=info:serverstartcode, timestamp=1256884549667, value=1256882625465        
 ,1256884494641                                                                                              
 TestTable1,b944f96a7a2def60 column=info:regioninfo, timestamp=1256884494672, value=REGION => {NAME => 'TestTa
 ,1256884494644              ble1,b944f96a7a2def60,1256884494644', STARTKEY => 'b944f96a7a2def60', ENDKEY => '
                             c43bb76eab2e6d22', ENCODED => 862125229, TABLE => {{NAME => 'TestTable1', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable1,b944f96a7a2def60 column=info:server, timestamp=1256884549656, value=76.13.20.116:60020            
 ,1256884494644                                                                                              
 TestTable1,b944f96a7a2def60 column=info:serverstartcode, timestamp=1256884549656, value=1256882633981        
 ,1256884494644                                                                                              
 TestTable1,c43bb76eab2e6d22 column=info:regioninfo, timestamp=1256884494675, value=REGION => {NAME => 'TestTa
 ,1256884494648              ble1,c43bb76eab2e6d22,1256884494648', STARTKEY => 'c43bb76eab2e6d22', ENDKEY => '
                             cf28bf0fea611b0c', ENCODED => 949626418, TABLE => {{NAME => 'TestTable1', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable1,c43bb76eab2e6d22 column=info:server, timestamp=1256884547193, value=76.13.20.58:60020            
 ,1256884494648                                                                                              
 TestTable1,c43bb76eab2e6d22 column=info:serverstartcode, timestamp=1256884547193, value=1256882624416        
 ,1256884494648                                                                                              
 TestTable1,cf28bf0fea611b0c column=info:regioninfo, timestamp=1256884494683, value=REGION => {NAME => 'TestTa
 ,1256884494653              ble1,cf28bf0fea611b0c,1256884494653', STARTKEY => 'cf28bf0fea611b0c', ENDKEY => '
                             da0cf7e7f55c8e14', ENCODED => 502466148, TABLE => {{NAME => 'TestTable1', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable1,cf28bf0fea611b0c column=info:server, timestamp=1256906339182, value=76.13.20.114:60020            
 ,1256884494653                                                                                              
 TestTable1,cf28bf0fea611b0c column=info:serverstartcode, timestamp=1256906339182, value=1256882627359        
 ,1256884494653                                                                                              
 TestTable1,da0cf7e7f55c8e14 column=info:regioninfo, timestamp=1256884494686, value=REGION => {NAME => 'TestTa
 ,1256884494659              ble1,da0cf7e7f55c8e14,1256884494659', STARTKEY => 'da0cf7e7f55c8e14', ENDKEY => '
                             e4e9fa98855ae512', ENCODED => 742354393, TABLE => {{NAME => 'TestTable1', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable1,da0cf7e7f55c8e14 column=info:server, timestamp=1256884549668, value=76.13.20.63:60020            
 ,1256884494659                                                                                              
 TestTable1,da0cf7e7f55c8e14 column=info:serverstartcode, timestamp=1256884549668, value=1256882625465        
 ,1256884494659                                                                                              
 TestTable1,e4e9fa98855ae512 column=info:regioninfo, timestamp=1256884494692, value=REGION => {NAME => 'TestTa
 ,1256884494663              ble1,e4e9fa98855ae512,1256884494663', STARTKEY => 'e4e9fa98855ae512', ENDKEY => '
                             efcae9e5d4a2bcdc', ENCODED => 126718000, TABLE => {{NAME => 'TestTable1', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable1,e4e9fa98855ae512 column=info:server, timestamp=1256885162504, value=76.13.20.114:60020            
 ,1256884494663                                                                                              
 TestTable1,e4e9fa98855ae512 column=info:serverstartcode, timestamp=1256885162504, value=1256882627359        
 ,1256884494663                                                                                              
 TestTable1,efcae9e5d4a2bcdc column=info:regioninfo, timestamp=1256884494695, value=REGION => {NAME => 'TestTa
 ,1256884494668              ble1,efcae9e5d4a2bcdc,1256884494668', STARTKEY => 'efcae9e5d4a2bcdc', ENDKEY => '
                             fab70d25ef5044cc', ENCODED => 1385744012, TABLE => {{NAME => 'TestTable1', FAMILI
                             ES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21474836
                             47', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable1,efcae9e5d4a2bcdc column=info:server, timestamp=1256885937653, value=76.13.20.114:60020            
 ,1256884494668                                                                                              
 TestTable1,efcae9e5d4a2bcdc column=info:serverstartcode, timestamp=1256885937653, value=1256882627359        
 ,1256884494668                                                                                              
 TestTable1,fab70d25ef5044cc column=info:regioninfo, timestamp=1256884494702, value=REGION => {NAME => 'TestTa
 ,1256884494674              ble1,fab70d25ef5044cc,1256884494674', STARTKEY => 'fab70d25ef5044cc', ENDKEY => '
                             ', ENCODED => 1182602639, TABLE => {{NAME => 'TestTable1', FAMILIES => [{NAME =>
                             'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE =
                             > '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                        
 TestTable1,fab70d25ef5044cc column=info:server, timestamp=1256885135410, value=76.13.20.112:60020            
 ,1256884494674                                                                                              
 TestTable1,fab70d25ef5044cc column=info:serverstartcode, timestamp=1256885135410, value=1256882626109        
 ,1256884494674                                                                                              
 TestTable2,,1256885885986   column=info:regioninfo, timestamp=1256885886061, value=REGION => {NAME => 'TestTa
                             ble2,,1256885885986', STARTKEY => '', ENDKEY => '0ae6b8a96914acc8', ENCODED => 12
                             00249546, TABLE => {{NAME => 'TestTable2', FAMILIES => [{NAME => 'image', VERSION
                             S => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_ME
                             MORY => 'false', BLOCKCACHE => 'true'}]}}                                        
 TestTable2,,1256885885986   column=info:server, timestamp=1256885928628, value=76.13.20.63:60020            
 TestTable2,,1256885885986   column=info:serverstartcode, timestamp=1256885928628, value=1256882625465        
 TestTable2,0ae6b8a96914acc8 column=info:regioninfo, timestamp=1256885886066, value=REGION => {NAME => 'TestTa
 ,1256885886054              ble2,0ae6b8a96914acc8,1256885886054', STARTKEY => '0ae6b8a96914acc8', ENDKEY => '
                             15c8d306751425a0', ENCODED => 1216902726, TABLE => {{NAME => 'TestTable2', FAMILI
                             ES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21474836
                             47', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable2,0ae6b8a96914acc8 column=info:server, timestamp=1256885931655, value=76.13.20.57:60020            
 ,1256885886054                                                                                              
 TestTable2,0ae6b8a96914acc8 column=info:serverstartcode, timestamp=1256885931655, value=1256882624829        
 ,1256885886054                                                                                              
 TestTable2,15c8d306751425a0 column=info:regioninfo, timestamp=1256885886070, value=REGION => {NAME => 'TestTa
 ,1256885886057              ble2,15c8d306751425a0,1256885886057', STARTKEY => '15c8d306751425a0', ENDKEY => '
                             20a4755f6984c8a4', ENCODED => 1661986340, TABLE => {{NAME => 'TestTable2', FAMILI
                             ES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21474836
                             47', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable2,15c8d306751425a0 column=info:server, timestamp=1256885928750, value=76.13.20.58:60020            
 ,1256885886057                                                                                              
 TestTable2,15c8d306751425a0 column=info:serverstartcode, timestamp=1256885928750, value=1256882624416        
 ,1256885886057                                                                                              
 TestTable2,20a4755f6984c8a4 column=info:regioninfo, timestamp=1256885886074, value=REGION => {NAME => 'TestTa
 ,1256885886061              ble2,20a4755f6984c8a4,1256885886061', STARTKEY => '20a4755f6984c8a4', ENDKEY => '
                             2b7de9fc0c1671fc', ENCODED => 1166498848, TABLE => {{NAME => 'TestTable2', FAMILI
                             ES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21474836
                             47', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable2,20a4755f6984c8a4 column=info:server, timestamp=1256885937641, value=76.13.20.63:60020            
 ,1256885886061                                                                                              
 TestTable2,20a4755f6984c8a4 column=info:serverstartcode, timestamp=1256885937641, value=1256882625465        
 ,1256885886061                                                                                              
 TestTable2,2b7de9fc0c1671fc column=info:regioninfo, timestamp=1256885886077, value=REGION => {NAME => 'TestTa
 ,1256885886065              ble2,2b7de9fc0c1671fc,1256885886065', STARTKEY => '2b7de9fc0c1671fc', ENDKEY => '
                             3666d964ee23718a', ENCODED => 1024476365, TABLE => {{NAME => 'TestTable2', FAMILI
                             ES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21474836
                             47', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable2,2b7de9fc0c1671fc column=info:server, timestamp=1256885928677, value=76.13.20.62:60020            
 ,1256885886065                                                                                              
 TestTable2,2b7de9fc0c1671fc column=info:serverstartcode, timestamp=1256885928677, value=1256882624666        
 ,1256885886065                                                                                              
 TestTable2,3666d964ee23718a column=info:regioninfo, timestamp=1256885886082, value=REGION => {NAME => 'TestTa
 ,1256885886068              ble2,3666d964ee23718a,1256885886068', STARTKEY => '3666d964ee23718a', ENDKEY => '
                             41472e380ecd4d22', ENCODED => 158620377, TABLE => {{NAME => 'TestTable2', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable2,3666d964ee23718a column=info:server, timestamp=1256885937662, value=76.13.20.57:60020            
 ,1256885886068                                                                                              
 TestTable2,3666d964ee23718a column=info:serverstartcode, timestamp=1256885937662, value=1256882624829        
 ,1256885886068                                                                                              
 TestTable2,41472e380ecd4d22 column=info:regioninfo, timestamp=1256885886086, value=REGION => {NAME => 'TestTa
 ,1256885886073              ble2,41472e380ecd4d22,1256885886073', STARTKEY => '41472e380ecd4d22', ENDKEY => '
                             4c218f292e94a074', ENCODED => 1806152406, TABLE => {{NAME => 'TestTable2', FAMILI
                             ES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21474836
                             47', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable2,41472e380ecd4d22 column=info:server, timestamp=1256885928636, value=76.13.20.114:60020            
 ,1256885886073                                                                                              
 TestTable2,41472e380ecd4d22 column=info:serverstartcode, timestamp=1256885928636, value=1256882627359        
 ,1256885886073                                                                                              
 TestTable2,4c218f292e94a074 column=info:regioninfo, timestamp=1256885886089, value=REGION => {NAME => 'TestTa
 ,1256885886077              ble2,4c218f292e94a074,1256885886077', STARTKEY => '4c218f292e94a074', ENDKEY => '
                             571cb9d584b08a1c', ENCODED => 440169477, TABLE => {{NAME => 'TestTable2', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable2,4c218f292e94a074 column=info:server, timestamp=1256885927759, value=76.13.20.50:60020            
 ,1256885886077                                                                                              
 TestTable2,4c218f292e94a074 column=info:serverstartcode, timestamp=1256885927759, value=1256882626416        
 ,1256885886077                                                                                              
 TestTable2,571cb9d584b08a1c column=info:regioninfo, timestamp=1256885886092, value=REGION => {NAME => 'TestTa
 ,1256885886080              ble2,571cb9d584b08a1c,1256885886080', STARTKEY => '571cb9d584b08a1c', ENDKEY => '
                             61f8562a687a4ef0', ENCODED => 526233708, TABLE => {{NAME => 'TestTable2', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable2,571cb9d584b08a1c column=info:server, timestamp=1256885929337, value=76.13.20.56:60020            
 ,1256885886080                                                                                              
 TestTable2,571cb9d584b08a1c column=info:serverstartcode, timestamp=1256885929337, value=1256882627260        
 ,1256885886080                                                                                              
 TestTable2,61f8562a687a4ef0 column=info:regioninfo, timestamp=1256885886096, value=REGION => {NAME => 'TestTa
 ,1256885886083              ble2,61f8562a687a4ef0,1256885886083', STARTKEY => '61f8562a687a4ef0', ENDKEY => '
                             6cd6cb3fac9bfe8e', ENCODED => 1164359658, TABLE => {{NAME => 'TestTable2', FAMILI
                             ES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21474836
                             47', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable2,61f8562a687a4ef0 column=info:server, timestamp=1256885928834, value=76.13.20.116:60020            
 ,1256885886083                                                                                              
 TestTable2,61f8562a687a4ef0 column=info:serverstartcode, timestamp=1256885928834, value=1256882633981        
 ,1256885886083                                                                                              
 TestTable2,6cd6cb3fac9bfe8e column=info:regioninfo, timestamp=1256885886099, value=REGION => {NAME => 'TestTa
 ,1256885886087              ble2,6cd6cb3fac9bfe8e,1256885886087', STARTKEY => '6cd6cb3fac9bfe8e', ENDKEY => '
                             77b0388e552b68dc', ENCODED => 1532403259, TABLE => {{NAME => 'TestTable2', FAMILI
                             ES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21474836
                             47', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable2,6cd6cb3fac9bfe8e column=info:server, timestamp=1256885931641, value=76.13.20.114:60020            
 ,1256885886087                                                                                              
 TestTable2,6cd6cb3fac9bfe8e column=info:serverstartcode, timestamp=1256885931641, value=1256882627359        
 ,1256885886087                                                                                              
 TestTable2,77b0388e552b68dc column=info:regioninfo, timestamp=1256885886102, value=REGION => {NAME => 'TestTa
 ,1256885886090              ble2,77b0388e552b68dc,1256885886090', STARTKEY => '77b0388e552b68dc', ENDKEY => '
                             82aeb97287e15180', ENCODED => 792062037, TABLE => {{NAME => 'TestTable2', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable2,77b0388e552b68dc column=info:server, timestamp=1256885928626, value=76.13.20.63:60020            
 ,1256885886090                                                                                              
 TestTable2,77b0388e552b68dc column=info:serverstartcode, timestamp=1256885928626, value=1256882625465        
 ,1256885886090                                                                                              
 TestTable2,82aeb97287e15180 column=info:regioninfo, timestamp=1256885886106, value=REGION => {NAME => 'TestTa
 ,1256885886093              ble2,82aeb97287e15180,1256885886093', STARTKEY => '82aeb97287e15180', ENDKEY => '
                             8d80de2029040fb2', ENCODED => 1786139740, TABLE => {{NAME => 'TestTable2', FAMILI
                             ES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21474836
                             47', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable2,82aeb97287e15180 column=info:server, timestamp=1256885928626, value=76.13.20.63:60020            
 ,1256885886093                                                                                              
 TestTable2,82aeb97287e15180 column=info:serverstartcode, timestamp=1256885928626, value=1256882625465        
 ,1256885886093                                                                                              
 TestTable2,8d80de2029040fb2 column=info:regioninfo, timestamp=1256885886109, value=REGION => {NAME => 'TestTa
 ,1256885886097              ble2,8d80de2029040fb2,1256885886097', STARTKEY => '8d80de2029040fb2', ENDKEY => '
                             986db344e405461a', ENCODED => 1513432549, TABLE => {{NAME => 'TestTable2', FAMILI
                             ES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21474836
                             47', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable2,8d80de2029040fb2 column=info:server, timestamp=1256885928629, value=76.13.20.63:60020            
 ,1256885886097                                                                                              
 TestTable2,8d80de2029040fb2 column=info:serverstartcode, timestamp=1256885928629, value=1256882625465        
 ,1256885886097                                                                                              
 TestTable2,986db344e405461a column=info:regioninfo, timestamp=1256885886113, value=REGION => {NAME => 'TestTa
 ,1256885886100              ble2,986db344e405461a,1256885886100', STARTKEY => '986db344e405461a', ENDKEY => '
                             a3650c660e297076', ENCODED => 1389182535, TABLE => {{NAME => 'TestTable2', FAMILI
                             ES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21474836
                             47', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable2,986db344e405461a column=info:server, timestamp=1256885928628, value=76.13.20.63:60020            
 ,1256885886100                                                                                              
 TestTable2,986db344e405461a column=info:serverstartcode, timestamp=1256885928628, value=1256882625465        
 ,1256885886100                                                                                              
 TestTable2,a3650c660e297076 column=info:regioninfo, timestamp=1256885886137, value=REGION => {NAME => 'TestTa
 ,1256885886105              ble2,a3650c660e297076,1256885886105', STARTKEY => 'a3650c660e297076', ENDKEY => '
                             ae4f6ea0ff9ec432', ENCODED => 564184164, TABLE => {{NAME => 'TestTable2', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable2,a3650c660e297076 column=info:server, timestamp=1256885931681, value=76.13.20.62:60020            
 ,1256885886105                                                                                              
 TestTable2,a3650c660e297076 column=info:serverstartcode, timestamp=1256885931681, value=1256882624666        
 ,1256885886105                                                                                              
 TestTable2,ae4f6ea0ff9ec432 column=info:regioninfo, timestamp=1256885886140, value=REGION => {NAME => 'TestTa
 ,1256885886128              ble2,ae4f6ea0ff9ec432,1256885886128', STARTKEY => 'ae4f6ea0ff9ec432', ENDKEY => '
                             b944f96a7a2def60', ENCODED => 631244644, TABLE => {{NAME => 'TestTable2', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable2,ae4f6ea0ff9ec432 column=info:server, timestamp=1256885928623, value=76.13.20.63:60020            
 ,1256885886128                                                                                              
 TestTable2,ae4f6ea0ff9ec432 column=info:serverstartcode, timestamp=1256885928623, value=1256882625465        
 ,1256885886128                                                                                              
 TestTable2,b944f96a7a2def60 column=info:regioninfo, timestamp=1256885886143, value=REGION => {NAME => 'TestTa
 ,1256885886131              ble2,b944f96a7a2def60,1256885886131', STARTKEY => 'b944f96a7a2def60', ENDKEY => '
                             c43bb76eab2e6d22', ENCODED => 1332082347, TABLE => {{NAME => 'TestTable2', FAMILI
                             ES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21474836
                             47', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable2,b944f96a7a2def60 column=info:server, timestamp=1256885928655, value=76.13.20.57:60020            
 ,1256885886131                                                                                              
 TestTable2,b944f96a7a2def60 column=info:serverstartcode, timestamp=1256885928655, value=1256882624829        
 ,1256885886131                                                                                              
 TestTable2,c43bb76eab2e6d22 column=info:regioninfo, timestamp=1256885886146, value=REGION => {NAME => 'TestTa
 ,1256885886134              ble2,c43bb76eab2e6d22,1256885886134', STARTKEY => 'c43bb76eab2e6d22', ENDKEY => '
                             cf28bf0fea611b0c', ENCODED => 614982466, TABLE => {{NAME => 'TestTable2', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable2,c43bb76eab2e6d22 column=info:server, timestamp=1256885928635, value=76.13.20.114:60020            
 ,1256885886134                                                                                              
 TestTable2,c43bb76eab2e6d22 column=info:serverstartcode, timestamp=1256885928635, value=1256882627359        
 ,1256885886134                                                                                              
 TestTable2,cf28bf0fea611b0c column=info:regioninfo, timestamp=1256885886151, value=REGION => {NAME => 'TestTa
 ,1256885886137              ble2,cf28bf0fea611b0c,1256885886137', STARTKEY => 'cf28bf0fea611b0c', ENDKEY => '
                             da0cf7e7f55c8e14', ENCODED => 333301889, TABLE => {{NAME => 'TestTable2', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable2,cf28bf0fea611b0c column=info:server, timestamp=1256885930756, value=76.13.20.50:60020            
 ,1256885886137                                                                                              
 TestTable2,cf28bf0fea611b0c column=info:serverstartcode, timestamp=1256885930756, value=1256882626416        
 ,1256885886137                                                                                              
 TestTable2,da0cf7e7f55c8e14 column=info:regioninfo, timestamp=1256885886155, value=REGION => {NAME => 'TestTa
 ,1256885886142              ble2,da0cf7e7f55c8e14,1256885886142', STARTKEY => 'da0cf7e7f55c8e14', ENDKEY => '
                             e4e9fa98855ae512', ENCODED => 1956344953, TABLE => {{NAME => 'TestTable2', FAMILI
                             ES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21474836
                             47', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable2,da0cf7e7f55c8e14 column=info:server, timestamp=1256885931588, value=76.13.20.112:60020            
 ,1256885886142                                                                                              
 TestTable2,da0cf7e7f55c8e14 column=info:serverstartcode, timestamp=1256885931588, value=1256882626109        
 ,1256885886142                                                                                              
 TestTable2,e4e9fa98855ae512 column=info:regioninfo, timestamp=1256885886158, value=REGION => {NAME => 'TestTa
 ,1256885886146              ble2,e4e9fa98855ae512,1256885886146', STARTKEY => 'e4e9fa98855ae512', ENDKEY => '
                             efcae9e5d4a2bcdc', ENCODED => 868690103, TABLE => {{NAME => 'TestTable2', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable2,e4e9fa98855ae512 column=info:server, timestamp=1256885928627, value=76.13.20.63:60020            
 ,1256885886146                                                                                              
 TestTable2,e4e9fa98855ae512 column=info:serverstartcode, timestamp=1256885928627, value=1256882625465        
 ,1256885886146                                                                                              
 TestTable2,efcae9e5d4a2bcdc column=info:regioninfo, timestamp=1256885886161, value=REGION => {NAME => 'TestTa
 ,1256885886149              ble2,efcae9e5d4a2bcdc,1256885886149', STARTKEY => 'efcae9e5d4a2bcdc', ENDKEY => '
                             fab70d25ef5044cc', ENCODED => 367125625, TABLE => {{NAME => 'TestTable2', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable2,efcae9e5d4a2bcdc column=info:server, timestamp=1256885928583, value=76.13.20.112:60020            
 ,1256885886149                                                                                              
 TestTable2,efcae9e5d4a2bcdc column=info:serverstartcode, timestamp=1256885928583, value=1256882626109        
 ,1256885886149                                                                                              
 TestTable2,fab70d25ef5044cc column=info:regioninfo, timestamp=1256885886164, value=REGION => {NAME => 'TestTa
 ,1256885886152              ble2,fab70d25ef5044cc,1256885886152', STARTKEY => 'fab70d25ef5044cc', ENDKEY => '
                             ', ENCODED => 1504762318, TABLE => {{NAME => 'TestTable2', FAMILIES => [{NAME =>
                             'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE =
                             > '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                        
 TestTable2,fab70d25ef5044cc column=info:server, timestamp=1256885937694, value=76.13.20.62:60020            
 ,1256885886152                                                                                              
 TestTable2,fab70d25ef5044cc column=info:serverstartcode, timestamp=1256885937694, value=1256882624666        
 ,1256885886152                                                                                              
 TestTable3,,1256906288855   column=info:regioninfo, timestamp=1256906288881, value=REGION => {NAME => 'TestTa
                             ble3,,1256906288855', STARTKEY => '', ENDKEY => '15c59281ac450dc6', ENCODED => 15
                             85342138, TABLE => {{NAME => 'TestTable3', FAMILIES => [{NAME => 'image', VERSION
                             S => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_ME
                             MORY => 'false', BLOCKCACHE => 'true'}]}}                                        
 TestTable3,,1256906288855   column=info:server, timestamp=1256906328324, value=76.13.20.56:60020            
 TestTable3,,1256906288855   column=info:serverstartcode, timestamp=1256906328324, value=1256882627260        
 TestTable3,15c59281ac450dc6 column=info:regioninfo, timestamp=1256906288896, value=REGION => {NAME => 'TestTa
 ,1256906288863              ble3,15c59281ac450dc6,1256906288863', STARTKEY => '15c59281ac450dc6', ENDKEY => '
                             15ccfdb905503a28', ENCODED => 875510056, TABLE => {{NAME => 'TestTable3', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable3,15c59281ac450dc6 column=info:server, timestamp=1256906339181, value=76.13.20.114:60020            
 ,1256906288863                                                                                              
 TestTable3,15c59281ac450dc6 column=info:serverstartcode, timestamp=1256906339181, value=1256882627359        
 ,1256906288863                                                                                              
 TestTable3,15ccfdb905503a28 column=info:regioninfo, timestamp=1256906288901, value=REGION => {NAME => 'TestTa
 ,1256906288878              ble3,15ccfdb905503a28,1256906288878', STARTKEY => '15ccfdb905503a28', ENDKEY => '
                             2b617bb97f70d25e', ENCODED => 1198918454, TABLE => {{NAME => 'TestTable3', FAMILI
                             ES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21474836
                             47', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable3,15ccfdb905503a28 column=info:server, timestamp=1256906329929, value=76.13.20.116:60020            
 ,1256906288878                                                                                              
 TestTable3,15ccfdb905503a28 column=info:serverstartcode, timestamp=1256906329929, value=1256882633981        
 ,1256906288878                                                                                              
 TestTable3,2b617bb97f70d25e column=info:regioninfo, timestamp=1256906288906, value=REGION => {NAME => 'TestTa
 ,1256906288882              ble3,2b617bb97f70d25e,1256906288882', STARTKEY => '2b617bb97f70d25e', ENDKEY => '
                             2b98bd771ca127d2', ENCODED => 444643706, TABLE => {{NAME => 'TestTable3', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable3,2b617bb97f70d25e column=info:server, timestamp=1256906329927, value=76.13.20.116:60020            
 ,1256906288882                                                                                              
 TestTable3,2b617bb97f70d25e column=info:serverstartcode, timestamp=1256906329927, value=1256882633981        
 ,1256906288882                                                                                              
 TestTable3,2b98bd771ca127d2 column=info:regioninfo, timestamp=1256906288910, value=REGION => {NAME => 'TestTa
 ,1256906288888              ble3,2b98bd771ca127d2,1256906288888', STARTKEY => '2b98bd771ca127d2', ENDKEY => '
                             4139ed31840dfcae', ENCODED => 450743192, TABLE => {{NAME => 'TestTable3', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable3,2b98bd771ca127d2 column=info:server, timestamp=1256906328612, value=76.13.20.58:60020            
 ,1256906288888                                                                                              
 TestTable3,2b98bd771ca127d2 column=info:serverstartcode, timestamp=1256906328612, value=1256882624416        
 ,1256906288888                                                                                              
 TestTable3,4139ed31840dfcae column=info:regioninfo, timestamp=1256906288916, value=REGION => {NAME => 'TestTa
 ,1256906288893              ble3,4139ed31840dfcae,1256906288893', STARTKEY => '4139ed31840dfcae', ENDKEY => '
                             415357bce4dca692', ENCODED => 2080427073, TABLE => {{NAME => 'TestTable3', FAMILI
                             ES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21474836
                             47', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable3,4139ed31840dfcae column=info:server, timestamp=1256906349768, value=76.13.20.112:60020            
 ,1256906288893                                                                                              
 TestTable3,4139ed31840dfcae column=info:serverstartcode, timestamp=1256906349768, value=1256882626109        
 ,1256906288893                                                                                              
 TestTable3,415357bce4dca692 column=info:regioninfo, timestamp=1256906288920, value=REGION => {NAME => 'TestTa
 ,1256906288898              ble3,415357bce4dca692,1256906288898', STARTKEY => '415357bce4dca692', ENDKEY => '
                             571837b11549fc5a', ENCODED => 737329246, TABLE => {{NAME => 'TestTable3', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable3,415357bce4dca692 column=info:server, timestamp=1256906328326, value=76.13.20.56:60020            
 ,1256906288898                                                                                              
 TestTable3,415357bce4dca692 column=info:serverstartcode, timestamp=1256906328326, value=1256882627260        
 ,1256906288898                                                                                              
 TestTable3,571837b11549fc5a column=info:regioninfo, timestamp=1256906288923, value=REGION => {NAME => 'TestTa
 ,1256906288901              ble3,571837b11549fc5a,1256906288901', STARTKEY => '571837b11549fc5a', ENDKEY => '
                             5721e1d5ef9fb522', ENCODED => 389627358, TABLE => {{NAME => 'TestTable3', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable3,571837b11549fc5a column=info:server, timestamp=1256906329932, value=76.13.20.116:60020            
 ,1256906288901                                                                                              
 TestTable3,571837b11549fc5a column=info:serverstartcode, timestamp=1256906329932, value=1256882633981        
 ,1256906288901                                                                                              
 TestTable3,5721e1d5ef9fb522 column=info:regioninfo, timestamp=1256906288928, value=REGION => {NAME => 'TestTa
 ,1256906288905              ble3,5721e1d5ef9fb522,1256906288905', STARTKEY => '5721e1d5ef9fb522', ENDKEY => '
                             6cc5b5ac0bb8a3b2', ENCODED => 1209365811, TABLE => {{NAME => 'TestTable3', FAMILI
                             ES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21474836
                             47', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable3,5721e1d5ef9fb522 column=info:server, timestamp=1256906329123, value=76.13.20.63:60020            
 ,1256906288905                                                                                              
 TestTable3,5721e1d5ef9fb522 column=info:serverstartcode, timestamp=1256906329123, value=1256882625465        
 ,1256906288905                                                                                              
 TestTable3,6cc5b5ac0bb8a3b2 column=info:regioninfo, timestamp=1256906288931, value=REGION => {NAME => 'TestTa
 ,1256906288909              ble3,6cc5b5ac0bb8a3b2,1256906288909', STARTKEY => '6cc5b5ac0bb8a3b2', ENDKEY => '
                             6ce5c17bc042016a', ENCODED => 1915440069, TABLE => {{NAME => 'TestTable3', FAMILI
                             ES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21474836
                             47', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable3,6cc5b5ac0bb8a3b2 column=info:server, timestamp=1256906337434, value=76.13.20.50:60020            
 ,1256906288909                                                                                              
 TestTable3,6cc5b5ac0bb8a3b2 column=info:serverstartcode, timestamp=1256906337434, value=1256882626416        
 ,1256906288909                                                                                              
 TestTable3,6ce5c17bc042016a column=info:regioninfo, timestamp=1256906288935, value=REGION => {NAME => 'TestTa
 ,1256906288913              ble3,6ce5c17bc042016a,1256906288913', STARTKEY => '6ce5c17bc042016a', ENDKEY => '
                             82a0229a5527f7a8', ENCODED => 187449751, TABLE => {{NAME => 'TestTable3', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable3,6ce5c17bc042016a column=info:server, timestamp=1256906349767, value=76.13.20.112:60020            
 ,1256906288913                                                                                              
 TestTable3,6ce5c17bc042016a column=info:serverstartcode, timestamp=1256906349767, value=1256882626109        
 ,1256906288913                                                                                              
 TestTable3,82a0229a5527f7a8 column=info:regioninfo, timestamp=1256906288939, value=REGION => {NAME => 'TestTa
 ,1256906288917              ble3,82a0229a5527f7a8,1256906288917', STARTKEY => '82a0229a5527f7a8', ENDKEY => '
                             82bd52706fcdc5ea', ENCODED => 2143948812, TABLE => {{NAME => 'TestTable3', FAMILI
                             ES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21474836
                             47', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable3,82a0229a5527f7a8 column=info:server, timestamp=1256906338945, value=76.13.20.116:60020            
 ,1256906288917                                                                                              
 TestTable3,82a0229a5527f7a8 column=info:serverstartcode, timestamp=1256906338945, value=1256882633981        
 ,1256906288917                                                                                              
 TestTable3,82bd52706fcdc5ea column=info:regioninfo, timestamp=1256906288943, value=REGION => {NAME => 'TestTa
 ,1256906288921              ble3,82bd52706fcdc5ea,1256906288921', STARTKEY => '82bd52706fcdc5ea', ENDKEY => '
                             985f09bb4ca22a3e', ENCODED => 1108646739, TABLE => {{NAME => 'TestTable3', FAMILI
                             ES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21474836
                             47', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable3,82bd52706fcdc5ea column=info:server, timestamp=1256906329928, value=76.13.20.116:60020            
 ,1256906288921                                                                                              
 TestTable3,82bd52706fcdc5ea column=info:serverstartcode, timestamp=1256906329928, value=1256882633981        
 ,1256906288921                                                                                              
 TestTable3,985f09bb4ca22a3e column=info:regioninfo, timestamp=1256906288947, value=REGION => {NAME => 'TestTa
 ,1256906288925              ble3,985f09bb4ca22a3e,1256906288925', STARTKEY => '985f09bb4ca22a3e', ENDKEY => '
                             9879b388cd1fef26', ENCODED => 226668565, TABLE => {{NAME => 'TestTable3', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable3,985f09bb4ca22a3e column=info:server, timestamp=1256906349459, value=76.13.20.50:60020            
 ,1256906288925                                                                                              
 TestTable3,985f09bb4ca22a3e column=info:serverstartcode, timestamp=1256906349459, value=1256882626416        
 ,1256906288925                                                                                              
 TestTable3,9879b388cd1fef26 column=info:regioninfo, timestamp=1256906288951, value=REGION => {NAME => 'TestTa
 ,1256906288928              ble3,9879b388cd1fef26,1256906288928', STARTKEY => '9879b388cd1fef26', ENDKEY => '
                             ae401b9496741f3e', ENCODED => 1083791189, TABLE => {{NAME => 'TestTable3', FAMILI
                             ES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21474836
                             47', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable3,9879b388cd1fef26 column=info:server, timestamp=1256906328323, value=76.13.20.56:60020            
 ,1256906288928                                                                                              
 TestTable3,9879b388cd1fef26 column=info:serverstartcode, timestamp=1256906328323, value=1256882627260        
 ,1256906288928                                                                                              
 TestTable3,ae401b9496741f3e column=info:regioninfo, timestamp=1256906288984, value=REGION => {NAME => 'TestTa
 ,1256906288934              ble3,ae401b9496741f3e,1256906288934', STARTKEY => 'ae401b9496741f3e', ENDKEY => '
                             ae5f34669cb32560', ENCODED => 361518302, TABLE => {{NAME => 'TestTable3', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable3,ae401b9496741f3e column=info:server, timestamp=1256906339181, value=76.13.20.114:60020            
 ,1256906288934                                                                                              
 TestTable3,ae401b9496741f3e column=info:serverstartcode, timestamp=1256906339181, value=1256882627359        
 ,1256906288934                                                                                              
 TestTable3,ae5f34669cb32560 column=info:regioninfo, timestamp=1256906288987, value=REGION => {NAME => 'TestTa
 ,1256906288965              ble3,ae5f34669cb32560,1256906288965', STARTKEY => 'ae5f34669cb32560', ENDKEY => '
                             c41cd52e478c6976', ENCODED => 878953630, TABLE => {{NAME => 'TestTable3', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable3,ae5f34669cb32560 column=info:server, timestamp=1256906349639, value=76.13.20.58:60020            
 ,1256906288965                                                                                              
 TestTable3,ae5f34669cb32560 column=info:serverstartcode, timestamp=1256906349639, value=1256882624416        
 ,1256906288965                                                                                              
 TestTable3,c41cd52e478c6976 column=info:regioninfo, timestamp=1256906288991, value=REGION => {NAME => 'TestTa
 ,1256906288969              ble3,c41cd52e478c6976,1256906288969', STARTKEY => 'c41cd52e478c6976', ENDKEY => '
                             c45a3ecddae903ee', ENCODED => 1527290548, TABLE => {{NAME => 'TestTable3', FAMILI
                             ES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21474836
                             47', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable3,c41cd52e478c6976 column=info:server, timestamp=1256906343444, value=76.13.20.50:60020            
 ,1256906288969                                                                                              
 TestTable3,c41cd52e478c6976 column=info:serverstartcode, timestamp=1256906343444, value=1256882626416        
 ,1256906288969                                                                                              
 TestTable3,c45a3ecddae903ee column=info:regioninfo, timestamp=1256906288994, value=REGION => {NAME => 'TestTa
 ,1256906288972              ble3,c45a3ecddae903ee,1256906288972', STARTKEY => 'c45a3ecddae903ee', ENDKEY => '
                             da08c465fd2996ce', ENCODED => 1343866493, TABLE => {{NAME => 'TestTable3', FAMILI
                             ES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21474836
                             47', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable3,c45a3ecddae903ee column=info:server, timestamp=1256906328323, value=76.13.20.56:60020            
 ,1256906288972                                                                                              
 TestTable3,c45a3ecddae903ee column=info:serverstartcode, timestamp=1256906328323, value=1256882627260        
 ,1256906288972                                                                                              
 TestTable3,da08c465fd2996ce column=info:regioninfo, timestamp=1256906288999, value=REGION => {NAME => 'TestTa
 ,1256906288976              ble3,da08c465fd2996ce,1256906288976', STARTKEY => 'da08c465fd2996ce', ENDKEY => '
                             da11211958b11eb2', ENCODED => 720566141, TABLE => {{NAME => 'TestTable3', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable3,da08c465fd2996ce column=info:server, timestamp=1256906337341, value=76.13.20.56:60020            
 ,1256906288976                                                                                              
 TestTable3,da08c465fd2996ce column=info:serverstartcode, timestamp=1256906337341, value=1256882627260        
 ,1256906288976                                                                                              
 TestTable3,da11211958b11eb2 column=info:regioninfo, timestamp=1256906289002, value=REGION => {NAME => 'TestTa
 ,1256906288980              ble3,da11211958b11eb2,1256906288980', STARTKEY => 'da11211958b11eb2', ENDKEY => '
                             efbd728372bbac94', ENCODED => 171871942, TABLE => {{NAME => 'TestTable3', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable3,da11211958b11eb2 column=info:server, timestamp=1256906327954, value=76.13.20.57:60020            
 ,1256906288980                                                                                              
 TestTable3,da11211958b11eb2 column=info:serverstartcode, timestamp=1256906327954, value=1256882624829        
 ,1256906288980                                                                                              
 TestTable3,efbd728372bbac94 column=info:regioninfo, timestamp=1256906289006, value=REGION => {NAME => 'TestTa
 ,1256906288984              ble3,efbd728372bbac94,1256906288984', STARTKEY => 'efbd728372bbac94', ENDKEY => '
                             efd8908b81fe45e6', ENCODED => 31225578, TABLE => {{NAME => 'TestTable3', FAMILIES
                              => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647
                             ', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}          
 TestTable3,efbd728372bbac94 column=info:server, timestamp=1256906329928, value=76.13.20.116:60020            
 ,1256906288984                                                                                              
 TestTable3,efbd728372bbac94 column=info:serverstartcode, timestamp=1256906329928, value=1256882633981        
 ,1256906288984                                                                                              
 TestTable3,efd8908b81fe45e6 column=info:regioninfo, timestamp=1256906289010, value=REGION => {NAME => 'TestTa
 ,1256906288988              ble3,efd8908b81fe45e6,1256906288988', STARTKEY => 'efd8908b81fe45e6', ENDKEY => '
                             ffffef95bcbf2638', ENCODED => 128185424, TABLE => {{NAME => 'TestTable3', FAMILIE
                             S => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '214748364
                             7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}        
 TestTable3,efd8908b81fe45e6 column=info:server, timestamp=1256906328737, value=76.13.20.112:60020            
 ,1256906288988                                                                                              
 TestTable3,efd8908b81fe45e6 column=info:serverstartcode, timestamp=1256906328737, value=1256882626109        
 ,1256906288988                                                                                              
 TestTable3,ffffef95bcbf2638 column=info:regioninfo, timestamp=1256906289015, value=REGION => {NAME => 'TestTa
 ,1256906288991              ble3,ffffef95bcbf2638,1256906288991', STARTKEY => 'ffffef95bcbf2638', ENDKEY => '
                             ', ENCODED => 1603180960, TABLE => {{NAME => 'TestTable3', FAMILIES => [{NAME =>
                             'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE =
                             > '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                        
 TestTable3,ffffef95bcbf2638 column=info:server, timestamp=1256906329929, value=76.13.20.116:60020            
 ,1256906288991                                                                                              
 TestTable3,ffffef95bcbf2638 column=info:serverstartcode, timestamp=1256906329929, value=1256882633981        
 ,1256906288991                                                                                              
134 row(s) in 1.4630 seconds

Re: Issue with bulk loader tool

by stack-3 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

That is interesting.  It'd almost point to a shell issue.  Enable DEBUG so
client can see it.  Then rerun shell.  Is it at least loading the right
region?  (The regions start and end keys span the asked for key?).  I took a
look at your attached .META. scan.  All looks good there.  The region
specifications look right.  If you want to bundle up the region that is
failing -- the one that the failing key comes out of, I can take a look
here.  You could also try playing with the HFile tool: ./bin/hbase
org.apache.hadoop.hbase.io.hfile.HFile.  Run the former and it'll output
usage.  You should be able to get it to dump content of the region (You need
to supply flags like -v to see actual keys to the HFile tool else it just
runs its check silently).    Check for your key.  Check things like
timestamp on it.  Maybe its 100 years in advance of now or something?

Yours,
St.Ack

On Fri, Oct 30, 2009 at 9:01 AM, Murali Krishna. P
<muralikpbhat@...>wrote:

> Attached ".META"
>
> Interesting, I was able to get the row from HTable via java code. But from
> the shell, still getting following
>
> hbase(main):004:0> get 'TestTable2', 'ffffef95bcbf2638'
> 0 row(s) in 1.2250 seconds
>
> Thanks,
> Murali Krishna
>
> Thanks,
> Murali Krishna
>
>
> ------------------------------
> *From:* stack <stack@...>
> *To:* hbase-user@...
> *Sent:* Fri, 30 October, 2009 8:39:46 PM
> *Subject:* Re: Issue with bulk loader tool
>
> Can you send a listing of ".META."?
>
> hbase> scan ".META."
>
> Also, can you bring a region down from hdfs, tar and gzip it, and then put
> it someplace I can pull so I can take a look?
>
> Thanks,
> St.Ack
>
>
> On Fri, Oct 30, 2009 at 3:31 AM, Murali Krishna. P
> <muralikpbhat@...>wrote:
>
> > Hi guys,
> >  I created a table according to hbase-48. A mapreduce job which creates
> > HFiles and then used loadtable.rb script to create the table. Everything
> > worked fine and i was able to scan the table. But when i do a get for a
> key
> > displayed in the scan output, it is not retrieving the row. shell says 0
> > row.
> >
> >  I tried using one reducer to ensure total ordering, but still same
> issue.
> >
> >
> > My mapper is like:
> >  context.write(new
> > ImmutableBytesWritable(((Text)key).toString().getBytes()), new
> > KeyValue(((Text)key).toString().getBytes(), "family1".getBytes(),
> >                    "column1".getBytes(), getValueBytes()));
> >
> >
> > Please help me investigate this.
> >
> > Thanks,
> > Murali Krishna
> >
>

Re: Issue with bulk loader tool

by stack-3 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Murali:

Any developments worth mentioning?

St.Ack


On Fri, Oct 30, 2009 at 10:14 AM, stack <stack@...> wrote:

> That is interesting.  It'd almost point to a shell issue.  Enable DEBUG so
> client can see it.  Then rerun shell.  Is it at least loading the right
> region?  (The regions start and end keys span the asked for key?).  I took a
> look at your attached .META. scan.  All looks good there.  The region
> specifications look right.  If you want to bundle up the region that is
> failing -- the one that the failing key comes out of, I can take a look
> here.  You could also try playing with the HFile tool: ./bin/hbase
> org.apache.hadoop.hbase.io.hfile.HFile.  Run the former and it'll output
> usage.  You should be able to get it to dump content of the region (You need
> to supply flags like -v to see actual keys to the HFile tool else it just
> runs its check silently).    Check for your key.  Check things like
> timestamp on it.  Maybe its 100 years in advance of now or something?
>
> Yours,
> St.Ack
>
>
> On Fri, Oct 30, 2009 at 9:01 AM, Murali Krishna. P <muralikpbhat@...
> > wrote:
>
>> Attached ".META"
>>
>> Interesting, I was able to get the row from HTable via java code. But from
>> the shell, still getting following
>>
>> hbase(main):004:0> get 'TestTable2', 'ffffef95bcbf2638'
>> 0 row(s) in 1.2250 seconds
>>
>> Thanks,
>> Murali Krishna
>>
>> Thanks,
>> Murali Krishna
>>
>>
>> ------------------------------
>> *From:* stack <stack@...>
>> *To:* hbase-user@...
>> *Sent:* Fri, 30 October, 2009 8:39:46 PM
>> *Subject:* Re: Issue with bulk loader tool
>>
>> Can you send a listing of ".META."?
>>
>> hbase> scan ".META."
>>
>> Also, can you bring a region down from hdfs, tar and gzip it, and then put
>> it someplace I can pull so I can take a look?
>>
>> Thanks,
>> St.Ack
>>
>>
>> On Fri, Oct 30, 2009 at 3:31 AM, Murali Krishna. P
>> <muralikpbhat@...>wrote:
>>
>> > Hi guys,
>> >  I created a table according to hbase-48. A mapreduce job which creates
>> > HFiles and then used loadtable.rb script to create the table. Everything
>> > worked fine and i was able to scan the table. But when i do a get for a
>> key
>> > displayed in the scan output, it is not retrieving the row. shell says 0
>> > row.
>> >
>> >  I tried using one reducer to ensure total ordering, but still same
>> issue.
>> >
>> >
>> > My mapper is like:
>> >  context.write(new
>> > ImmutableBytesWritable(((Text)key).toString().getBytes()), new
>> > KeyValue(((Text)key).toString().getBytes(), "family1".getBytes(),
>> >                    "column1".getBytes(), getValueBytes()));
>> >
>> >
>> > Please help me investigate this.
>> >
>> > Thanks,
>> > Murali Krishna
>> >
>>
>
>

Re: Issue with bulk loader tool

by Murali Krishna. P :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hi Stack,
Sorry, could not look into this last week...

I got problem with the Htable interface as well. Some records i am not retrieve from Htable as well.
I lost the old table, but reproduced the problem with a different table.

I cannot send the region since it is very huge. will try to give as much info as possible here :)

There are total 5 regions as below in that table:
Name

Encoded Name
Start Key
End Key
test1,,1257414794600
 106817540
 fffe9c7f87c8332a
test1,fffe9c7f87c8332a,1257414794616
 1346846599 fffe9c7f87c8332a fffebe279c0ac4d2
test1,fffebe279c0ac4d2,1257414794628
 1835851728 fffebe279c0ac4d2 fffec418284d6fbc
test1,fffec418284d6fbc,1257414794637
 1078205908 fffec418284d6fbc fffef7a12ea22498
test1,fffef7a12ea22498,1257414794647
 1515378663 fffef7a12ea22498
 
I am looking for a key, say 000011d1bc8cd6fe . This should be in the first region ?

using hfile tool,
org.apache.hadoop.hbase.io.hfile.HFile -k -f /hbase/test1/106817540/image/3828859735461759684 -v -m -p |  grep 000011d1bc8cd6fe
The first region doesn't have it. Not sure what happened to that record.

For a working key, it gives the record properly as below
K: \x00\x100003bdd08ca88ee2\x05imagevalue\x7F\xFF\xFF\xFF\xFF\xFF\xFF\xFF\x04 V: \xFF...

Please let me know if you need more information

 Thanks,
Murali Krishna




________________________________
From: stack <stack@...>
To: hbase-user@...
Sent: Mon, 2 November, 2009 11:05:43 PM
Subject: Re: Issue with bulk loader tool

Murali:

Any developments worth mentioning?

St.Ack


On Fri, Oct 30, 2009 at 10:14 AM, stack <stack@...> wrote:

> That is interesting.  It'd almost point to a shell issue.  Enable DEBUG so
> client can see it.  Then rerun shell.  Is it at least loading the right
> region?  (The regions start and end keys span the asked for key?).  I took a
> look at your attached .META. scan.  All looks good there.  The region
> specifications look right.  If you want to bundle up the region that is
> failing -- the one that the failing key comes out of, I can take a look
> here.  You could also try playing with the HFile tool: ./bin/hbase
> org.apache.hadoop.hbase.io.hfile.HFile.  Run the former and it'll output
> usage.  You should be able to get it to dump content of the region (You need
> to supply flags like -v to see actual keys to the HFile tool else it just
> runs its check silently).    Check for your key.  Check things like
> timestamp on it.  Maybe its 100 years in advance of now or something?
>
> Yours,
> St.Ack
>
>
> On Fri, Oct 30, 2009 at 9:01 AM, Murali Krishna. P <muralikpbhat@...
> > wrote:
>
>> Attached ".META"
>>
>> Interesting, I was able to get the row from HTable via java code. But from
>> the shell, still getting following
>>
>> hbase(main):004:0> get 'TestTable2', 'ffffef95bcbf2638'
>> 0 row(s) in 1.2250 seconds
>>
>> Thanks,
>> Murali Krishna
>>
>> Thanks,
>> Murali Krishna
>>
>>
>> ------------------------------
>> *From:* stack <stack@...>
>> *To:* hbase-user@...
>> *Sent:* Fri, 30 October, 2009 8:39:46 PM
>> *Subject:* Re: Issue with bulk loader tool
>>
>> Can you send a listing of ".META."?
>>
>> hbase> scan ".META."
>>
>> Also, can you bring a region down from hdfs, tar and gzip it, and then put
>> it someplace I can pull so I can take a look?
>>
>> Thanks,
>> St.Ack
>>
>>
>> On Fri, Oct 30, 2009 at 3:31 AM, Murali Krishna. P
>> <muralikpbhat@...>wrote:
>>
>> > Hi guys,
>> >  I created a table according to hbase-48. A mapreduce job which creates
>> > HFiles and then used loadtable.rb script to create the table. Everything
>> > worked fine and i was able to scan the table. But when i do a get for a
>> key
>> > displayed in the scan output, it is not retrieving the row. shell says 0
>> > row.
>> >
>> >  I tried using one reducer to ensure total ordering, but still same
>> issue.
>> >
>> >
>> > My mapper is like:
>> >  context.write(new
>> > ImmutableBytesWritable(((Text)key).toString().getBytes()), new
>> > KeyValue(((Text)key).toString().getBytes(), "family1".getBytes(),
>> >                    "column1".getBytes(), getValueBytes()));
>> >
>> >
>> > Please help me investigate this.
>> >
>> > Thanks,
>> > Murali Krishna
>> >
>>
>
>

Re: Issue with bulk loader tool

by Murali Krishna. P :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hi,
 If I increase hbase.hregion.max.filesize so that all the records holds in one region (and one reducer ), all the records as retrievable. If one reducer creates multiple hfile or multiple reducer creates one hfile each, the problem occurs.

Does that give any clue?

Thanks,
Murali Krishna




________________________________
From: Murali Krishna. P <muralikpbhat@...>
To: hbase-user@...
Sent: Thu, 5 November, 2009 6:34:20 PM
Subject: Re: Issue with bulk loader tool

Hi Stack,
Sorry, could not look into this last week...

I got problem with the Htable interface as well. Some records i am not retrieve from Htable as well.
I lost the old table, but reproduced the problem with a different table.

I cannot send the region since it is very huge. will try to give as much info as possible here :)

There are total 5 regions as below in that table:
Name

Encoded Name
Start Key
End Key
test1,,1257414794600
106817540
fffe9c7f87c8332a
test1,fffe9c7f87c8332a,1257414794616
1346846599 fffe9c7f87c8332a fffebe279c0ac4d2
test1,fffebe279c0ac4d2,1257414794628
1835851728 fffebe279c0ac4d2 fffec418284d6fbc
test1,fffec418284d6fbc,1257414794637
1078205908 fffec418284d6fbc fffef7a12ea22498
test1,fffef7a12ea22498,1257414794647
1515378663 fffef7a12ea22498

I am looking for a key, say 000011d1bc8cd6fe . This should be in the first region ?

using hfile tool,
org.apache.hadoop.hbase.io.hfile.HFile -k -f /hbase/test1/106817540/image/3828859735461759684 -v -m -p |  grep 000011d1bc8cd6fe
The first region doesn't have it. Not sure what happened to that record.

For a working key, it gives the record properly as below
K: \x00\x100003bdd08ca88ee2\x05imagevalue\x7F\xFF\xFF\xFF\xFF\xFF\xFF\xFF\x04 V: \xFF...

Please let me know if you need more information

Thanks,
Murali Krishna




________________________________
From: stack <stack@...>
To: hbase-user@...
Sent: Mon, 2 November, 2009 11:05:43 PM
Subject: Re: Issue with bulk loader tool

Murali:

Any developments worth mentioning?

St.Ack


On Fri, Oct 30, 2009 at 10:14 AM, stack <stack@...> wrote:

> That is interesting.  It'd almost point to a shell issue.  Enable DEBUG so
> client can see it.  Then rerun shell.  Is it at least loading the right
> region?  (The regions start and end keys span the asked for key?).  I took a
> look at your attached .META. scan.  All looks good there.  The region
> specifications look right.  If you want to bundle up the region that is
> failing -- the one that the failing key comes out of, I can take a look
> here.  You could also try playing with the HFile tool: ./bin/hbase
> org.apache.hadoop.hbase.io.hfile.HFile.  Run the former and it'll output
> usage.  You should be able to get it to dump content of the region (You need
> to supply flags like -v to see actual keys to the HFile tool else it just
> runs its check silently).    Check for your key.  Check things like
> timestamp on it.  Maybe its 100 years in advance of now or something?
>
> Yours,
> St.Ack
>
>
> On Fri, Oct 30, 2009 at 9:01 AM, Murali Krishna. P <muralikpbhat@...
> > wrote:
>
>> Attached ".META"
>>
>> Interesting, I was able to get the row from HTable via java code. But from
>> the shell, still getting following
>>
>> hbase(main):004:0> get 'TestTable2', 'ffffef95bcbf2638'
>> 0 row(s) in 1.2250 seconds
>>
>> Thanks,
>> Murali Krishna
>>
>> Thanks,
>> Murali Krishna
>>
>>
>> ------------------------------
>> *From:* stack <stack@...>
>> *To:* hbase-user@...
>> *Sent:* Fri, 30 October, 2009 8:39:46 PM
>> *Subject:* Re: Issue with bulk loader tool
>>
>> Can you send a listing of ".META."?
>>
>> hbase> scan ".META."
>>
>> Also, can you bring a region down from hdfs, tar and gzip it, and then put
>> it someplace I can pull so I can take a look?
>>
>> Thanks,
>> St.Ack
>>
>>
>> On Fri, Oct 30, 2009 at 3:31 AM, Murali Krishna. P
>> <muralikpbhat@...>wrote:
>>
>> > Hi guys,
>> >  I created a table according to hbase-48. A mapreduce job which creates
>> > HFiles and then used loadtable.rb script to create the table. Everything
>> > worked fine and i was able to scan the table. But when i do a get for a
>> key
>> > displayed in the scan output, it is not retrieving the row. shell says 0
>> > row.
>> >
>> >  I tried using one reducer to ensure total ordering, but still same
>> issue.
>> >
>> >
>> > My mapper is like:
>> >  context.write(new
>> > ImmutableBytesWritable(((Text)key).toString().getBytes()), new
>> > KeyValue(((Text)key).toString().getBytes(), "family1".getBytes(),
>> >                    "column1".getBytes(), getValueBytes()));
>> >
>> >
>> > Please help me investigate this.
>> >
>> > Thanks,
>> > Murali Krishna
>> >
>>
>
>

Re: Issue with bulk loader tool

by stack-3 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Thu, Nov 5, 2009 at 5:04 AM, Murali Krishna. P <muralikpbhat@...>wrote:

> There are total 5 regions as below in that table:
> Name
>
> Encoded Name
> Start Key
> End Key
> test1,,1257414794600
>  106817540
>  fffe9c7f87c8332a
> test1,fffe9c7f87c8332a,1257414794616
>  1346846599 fffe9c7f87c8332a fffebe279c0ac4d2
> test1,fffebe279c0ac4d2,1257414794628
>  1835851728 fffebe279c0ac4d2 fffec418284d6fbc
> test1,fffec418284d6fbc,1257414794637
>  1078205908 fffec418284d6fbc fffef7a12ea22498
> test1,fffef7a12ea22498,1257414794647
>  1515378663 fffef7a12ea22498
>
> I am looking for a key, say 000011d1bc8cd6fe . This should be in the first
> region ?
>
>
yes.



> using hfile tool,
> org.apache.hadoop.hbase.io.hfile.HFile -k -f
> /hbase/test1/106817540/image/3828859735461759684 -v -m -p |  grep
> 000011d1bc8cd6fe
> The first region doesn't have it. Not sure what happened to that record.
>


Can you dump output to file and look at keys around it?  Maybe you'll get a
clue as to why this key is missing?

What did you use for a partitioner running this job?

St.Ack




>
> For a working key, it gives the record properly as below
> K:
> \x00\x100003bdd08ca88ee2\x05imagevalue\x7F\xFF\xFF\xFF\xFF\xFF\xFF\xFF\x04
> V: \xFF...
>
> Please let me know if you need more information
>
>  Thanks,
> Murali Krishna
>
>
>
>
> ________________________________
> From: stack <stack@...>
> To: hbase-user@...
> Sent: Mon, 2 November, 2009 11:05:43 PM
> Subject: Re: Issue with bulk loader tool
>
> Murali:
>
> Any developments worth mentioning?
>
> St.Ack
>
>
> On Fri, Oct 30, 2009 at 10:14 AM, stack <stack@...> wrote:
>
> > That is interesting.  It'd almost point to a shell issue.  Enable DEBUG
> so
> > client can see it.  Then rerun shell.  Is it at least loading the right
> > region?  (The regions start and end keys span the asked for key?).  I
> took a
> > look at your attached .META. scan.  All looks good there.  The region
> > specifications look right.  If you want to bundle up the region that is
> > failing -- the one that the failing key comes out of, I can take a look
> > here.  You could also try playing with the HFile tool: ./bin/hbase
> > org.apache.hadoop.hbase.io.hfile.HFile.  Run the former and it'll output
> > usage.  You should be able to get it to dump content of the region (You
> need
> > to supply flags like -v to see actual keys to the HFile tool else it just
> > runs its check silently).    Check for your key.  Check things like
> > timestamp on it.  Maybe its 100 years in advance of now or something?
> >
> > Yours,
> > St.Ack
> >
> >
> > On Fri, Oct 30, 2009 at 9:01 AM, Murali Krishna. P <
> muralikpbhat@...
> > > wrote:
> >
> >> Attached ".META"
> >>
> >> Interesting, I was able to get the row from HTable via java code. But
> from
> >> the shell, still getting following
> >>
> >> hbase(main):004:0> get 'TestTable2', 'ffffef95bcbf2638'
> >> 0 row(s) in 1.2250 seconds
> >>
> >> Thanks,
> >> Murali Krishna
> >>
> >> Thanks,
> >> Murali Krishna
> >>
> >>
> >> ------------------------------
> >> *From:* stack <stack@...>
> >> *To:* hbase-user@...
> >> *Sent:* Fri, 30 October, 2009 8:39:46 PM
> >> *Subject:* Re: Issue with bulk loader tool
> >>
> >> Can you send a listing of ".META."?
> >>
> >> hbase> scan ".META."
> >>
> >> Also, can you bring a region down from hdfs, tar and gzip it, and then
> put
> >> it someplace I can pull so I can take a look?
> >>
> >> Thanks,
> >> St.Ack
> >>
> >>
> >> On Fri, Oct 30, 2009 at 3:31 AM, Murali Krishna. P
> >> <muralikpbhat@...>wrote:
> >>
> >> > Hi guys,
> >> >  I created a table according to hbase-48. A mapreduce job which
> creates
> >> > HFiles and then used loadtable.rb script to create the table.
> Everything
> >> > worked fine and i was able to scan the table. But when i do a get for
> a
> >> key
> >> > displayed in the scan output, it is not retrieving the row. shell says
> 0
> >> > row.
> >> >
> >> >  I tried using one reducer to ensure total ordering, but still same
> >> issue.
> >> >
> >> >
> >> > My mapper is like:
> >> >  context.write(new
> >> > ImmutableBytesWritable(((Text)key).toString().getBytes()), new
> >> > KeyValue(((Text)key).toString().getBytes(), "family1".getBytes(),
> >> >                    "column1".getBytes(), getValueBytes()));
> >> >
> >> >
> >> > Please help me investigate this.
> >> >
> >> > Thanks,
> >> > Murali Krishna
> >> >
> >>
> >
> >
>

Re: Issue with bulk loader tool

by stack-3 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Fri, Nov 6, 2009 at 12:58 AM, Murali Krishna. P
<muralikpbhat@...>wrote:

> Hi,
>  If I increase hbase.hregion.max.filesize so that all the records holds in
> one region (and one reducer ), all the records as retrievable. If one
> reducer creates multiple hfile or multiple reducer creates one hfile each,
> the problem occurs.
>
>

Multiple hfiles in a region?  Or are you saying if a reducer creates
multiple regions?  There is supposed to be one file per region only when
done.

Thanks for digging in,
St.Ack




> Does that give any clue?
>
> Thanks,
> Murali Krishna
>
>
>
>
> ________________________________
> From: Murali Krishna. P <muralikpbhat@...>
> To: hbase-user@...
> Sent: Thu, 5 November, 2009 6:34:20 PM
> Subject: Re: Issue with bulk loader tool
>
> Hi Stack,
> Sorry, could not look into this last week...
>
> I got problem with the Htable interface as well. Some records i am not
> retrieve from Htable as well.
> I lost the old table, but reproduced the problem with a different table.
>
> I cannot send the region since it is very huge. will try to give as much
> info as possible here :)
>
> There are total 5 regions as below in that table:
> Name
>
> Encoded Name
> Start Key
> End Key
> test1,,1257414794600
> 106817540
> fffe9c7f87c8332a
> test1,fffe9c7f87c8332a,1257414794616
> 1346846599 fffe9c7f87c8332a fffebe279c0ac4d2
> test1,fffebe279c0ac4d2,1257414794628
> 1835851728 fffebe279c0ac4d2 fffec418284d6fbc
> test1,fffec418284d6fbc,1257414794637
> 1078205908 fffec418284d6fbc fffef7a12ea22498
> test1,fffef7a12ea22498,1257414794647
> 1515378663 fffef7a12ea22498
>
> I am looking for a key, say 000011d1bc8cd6fe . This should be in the first
> region ?
>
> using hfile tool,
> org.apache.hadoop.hbase.io.hfile.HFile -k -f
> /hbase/test1/106817540/image/3828859735461759684 -v -m -p |  grep
> 000011d1bc8cd6fe
> The first region doesn't have it. Not sure what happened to that record.
>
> For a working key, it gives the record properly as below
> K:
> \x00\x100003bdd08ca88ee2\x05imagevalue\x7F\xFF\xFF\xFF\xFF\xFF\xFF\xFF\x04
> V: \xFF...
>
> Please let me know if you need more information
>
> Thanks,
> Murali Krishna
>
>
>
>
> ________________________________
> From: stack <stack@...>
> To: hbase-user@...
> Sent: Mon, 2 November, 2009 11:05:43 PM
> Subject: Re: Issue with bulk loader tool
>
> Murali:
>
> Any developments worth mentioning?
>
> St.Ack
>
>
> On Fri, Oct 30, 2009 at 10:14 AM, stack <stack@...> wrote:
>
> > That is interesting.  It'd almost point to a shell issue.  Enable DEBUG
> so
> > client can see it.  Then rerun shell.  Is it at least loading the right
> > region?  (The regions start and end keys span the asked for key?).  I
> took a
> > look at your attached .META. scan.  All looks good there.  The region
> > specifications look right.  If you want to bundle up the region that is
> > failing -- the one that the failing key comes out of, I can take a look
> > here.  You could also try playing with the HFile tool: ./bin/hbase
> > org.apache.hadoop.hbase.io.hfile.HFile.  Run the former and it'll output
> > usage.  You should be able to get it to dump content of the region (You
> need
> > to supply flags like -v to see actual keys to the HFile tool else it just
> > runs its check silently).    Check for your key.  Check things like
> > timestamp on it.  Maybe its 100 years in advance of now or something?
> >
> > Yours,
> > St.Ack
> >
> >
> > On Fri, Oct 30, 2009 at 9:01 AM, Murali Krishna. P <
> muralikpbhat@...
> > > wrote:
> >
> >> Attached ".META"
> >>
> >> Interesting, I was able to get the row from HTable via java code. But
> from
> >> the shell, still getting following
> >>
> >> hbase(main):004:0> get 'TestTable2', 'ffffef95bcbf2638'
> >> 0 row(s) in 1.2250 seconds
> >>
> >> Thanks,
> >> Murali Krishna
> >>
> >> Thanks,
> >> Murali Krishna
> >>
> >>
> >> ------------------------------
> >> *From:* stack <stack@...>
> >> *To:* hbase-user@...
> >> *Sent:* Fri, 30 October, 2009 8:39:46 PM
> >> *Subject:* Re: Issue with bulk loader tool
> >>
> >> Can you send a listing of ".META."?
> >>
> >> hbase> scan ".META."
> >>
> >> Also, can you bring a region down from hdfs, tar and gzip it, and then
> put
> >> it someplace I can pull so I can take a look?
> >>
> >> Thanks,
> >> St.Ack
> >>
> >>
> >> On Fri, Oct 30, 2009 at 3:31 AM, Murali Krishna. P
> >> <muralikpbhat@...>wrote:
> >>
> >> > Hi guys,
> >> >  I created a table according to hbase-48. A mapreduce job which
> creates
> >> > HFiles and then used loadtable.rb script to create the table.
> Everything
> >> > worked fine and i was able to scan the table. But when i do a get for
> a
> >> key
> >> > displayed in the scan output, it is not retrieving the row. shell says
> 0
> >> > row.
> >> >
> >> >  I tried using one reducer to ensure total ordering, but still same
> >> issue.
> >> >
> >> >
> >> > My mapper is like:
> >> >  context.write(new
> >> > ImmutableBytesWritable(((Text)key).toString().getBytes()), new
> >> > KeyValue(((Text)key).toString().getBytes(), "family1".getBytes(),
> >> >                    "column1".getBytes(), getValueBytes()));
> >> >
> >> >
> >> > Please help me investigate this.
> >> >
> >> > Thanks,
> >> > Murali Krishna
> >> >
> >>
> >
> >
>

Re: Issue with bulk loader tool

by Murali Krishna. P :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hi,
 I got it resolved. https://issues.apache.org/jira/browse/HADOOP-5750 was causing this, even though I supplied a custom total ordering partitioner, it didnt use that.


   Now the regions looks properly sorted, but facing a new issue. The last key of the each region is not retrievable. The table.jsp  page shows the start and end key wrongly.
for eg, take first 2 regions
region1: start : end: 333305184e0f7c3e
region2: start: 333305184e0f7c3e end: 666629fe4378c096

The end key of first region = start key of second ??

If I get the first and last key using HFile.Reader, it shows as follows:

HFileUtil /hbase/test12/98766318/image/9052388247118781160
FirstKey:00000d7d4f36c112imagevalue�������
LastKey:333305184e0f7c3eimagevalue�������

HFileUtil /hbase/test12/170637321/image/7602871928600243730
FirstKey:33338d45cc2491b8imagevalue�������
LastKey:666629fe4378c096imagevalue�������

So, according to this first key of 2nd region is 33338d45cc2491b8 not 333305184e0f7c3e which is correct!

Now when I do a get on 333305184e0f7c3e with debug on, it is loading the second region which is wrong!

Some thing went wrong with the index?

Thanks,
Murali Krishna




________________________________
From: stack <stack@...>
To: hbase-user@...
Sent: Sat, 7 November, 2009 6:26:03 AM
Subject: Re: Issue with bulk loader tool

On Fri, Nov 6, 2009 at 12:58 AM, Murali Krishna. P
<muralikpbhat@...>wrote:

> Hi,
>  If I increase hbase.hregion.max.filesize so that all the records holds in
> one region (and one reducer ), all the records as retrievable. If one
> reducer creates multiple hfile or multiple reducer creates one hfile each,
> the problem occurs.
>
>

Multiple hfiles in a region?  Or are you saying if a reducer creates
multiple regions?  There is supposed to be one file per region only when
done.

Thanks for digging in,
St.Ack




> Does that give any clue?
>
> Thanks,
> Murali Krishna
>
>
>
>
> ________________________________
> From: Murali Krishna. P <muralikpbhat@...>
> To: hbase-user@...
> Sent: Thu, 5 November, 2009 6:34:20 PM
> Subject: Re: Issue with bulk loader tool
>
> Hi Stack,
> Sorry, could not look into this last week...
>
> I got problem with the Htable interface as well. Some records i am not
> retrieve from Htable as well.
> I lost the old table, but reproduced the problem with a different table.
>
> I cannot send the region since it is very huge. will try to give as much
> info as possible here :)
>
> There are total 5 regions as below in that table:
> Name
>
> Encoded Name
> Start Key
> End Key
> test1,,1257414794600
> 106817540
> fffe9c7f87c8332a
> test1,fffe9c7f87c8332a,1257414794616
> 1346846599 fffe9c7f87c8332a fffebe279c0ac4d2
> test1,fffebe279c0ac4d2,1257414794628
> 1835851728 fffebe279c0ac4d2 fffec418284d6fbc
> test1,fffec418284d6fbc,1257414794637
> 1078205908 fffec418284d6fbc fffef7a12ea22498
> test1,fffef7a12ea22498,1257414794647
> 1515378663 fffef7a12ea22498
>
> I am looking for a key, say 000011d1bc8cd6fe . This should be in the first
> region ?
>
> using hfile tool,
> org.apache.hadoop.hbase.io.hfile.HFile -k -f
> /hbase/test1/106817540/image/3828859735461759684 -v -m -p |  grep
> 000011d1bc8cd6fe
> The first region doesn't have it. Not sure what happened to that record.
>
> For a working key, it gives the record properly as below
> K:
> \x00\x100003bdd08ca88ee2\x05imagevalue\x7F\xFF\xFF\xFF\xFF\xFF\xFF\xFF\x04
> V: \xFF...
>
> Please let me know if you need more information
>
> Thanks,
> Murali Krishna
>
>
>
>
> ________________________________
> From: stack <stack@...>
> To: hbase-user@...
> Sent: Mon, 2 November, 2009 11:05:43 PM
> Subject: Re: Issue with bulk loader tool
>
> Murali:
>
> Any developments worth mentioning?
>
> St.Ack
>
>
> On Fri, Oct 30, 2009 at 10:14 AM, stack <stack@...> wrote:
>
> > That is interesting.  It'd almost point to a shell issue.  Enable DEBUG
> so
> > client can see it.  Then rerun shell.  Is it at least loading the right
> > region?  (The regions start and end keys span the asked for key?).  I
> took a
> > look at your attached .META. scan.  All looks good there.  The region
> > specifications look right.  If you want to bundle up the region that is
> > failing -- the one that the failing key comes out of, I can take a look
> > here.  You could also try playing with the HFile tool: ./bin/hbase
> > org.apache.hadoop.hbase.io.hfile.HFile.  Run the former and it'll output
> > usage.  You should be able to get it to dump content of the region (You
> need
> > to supply flags like -v to see actual keys to the HFile tool else it just
> > runs its check silently).    Check for your key.  Check things like
> > timestamp on it.  Maybe its 100 years in advance of now or something?
> >
> > Yours,
> > St.Ack
> >
> >
> > On Fri, Oct 30, 2009 at 9:01 AM, Murali Krishna. P <
> muralikpbhat@...
> > > wrote:
> >
> >> Attached ".META"
> >>
> >> Interesting, I was able to get the row from HTable via java code. But
> from
> >> the shell, still getting following
> >>
> >> hbase(main):004:0> get 'TestTable2', 'ffffef95bcbf2638'
> >> 0 row(s) in 1.2250 seconds
> >>
> >> Thanks,
> >> Murali Krishna
> >>
> >> Thanks,
> >> Murali Krishna
> >>
> >>
> >> ------------------------------
> >> *From:* stack <stack@...>
> >> *To:* hbase-user@...
> >> *Sent:* Fri, 30 October, 2009 8:39:46 PM
> >> *Subject:* Re: Issue with bulk loader tool
> >>
> >> Can you send a listing of ".META."?
> >>
> >> hbase> scan ".META."
> >>
> >> Also, can you bring a region down from hdfs, tar and gzip it, and then
> put
> >> it someplace I can pull so I can take a look?
> >>
> >> Thanks,
> >> St.Ack
> >>
> >>
> >> On Fri, Oct 30, 2009 at 3:31 AM, Murali Krishna. P
> >> <muralikpbhat@...>wrote:
> >>
> >> > Hi guys,
> >> >  I created a table according to hbase-48. A mapreduce job which
> creates
> >> > HFiles and then used loadtable.rb script to create the table.
> Everything
> >> > worked fine and i was able to scan the table. But when i do a get for
> a
> >> key
> >> > displayed in the scan output, it is not retrieving the row. shell says
> 0
> >> > row.
> >> >
> >> >  I tried using one reducer to ensure total ordering, but still same
> >> issue.
> >> >
> >> >
> >> > My mapper is like:
> >> >  context.write(new
> >> > ImmutableBytesWritable(((Text)key).toString().getBytes()), new
> >> > KeyValue(((Text)key).toString().getBytes(), "family1".getBytes(),
> >> >                    "column1".getBytes(), getValueBytes()));
> >> >
> >> >
> >> > Please help me investigate this.
> >> >
> >> > Thanks,
> >> > Murali Krishna
> >> >
> >>
> >
> >
>

Re: Issue with bulk loader tool

by larsgeorge :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hi Murali,

What you see is normal the last keys do indeed overlap. The last key  
of a region is exclusive and marks the first key of the subsequent  
region.

Lars

On Nov 7, 2009, at 9:05, "Murali Krishna. P" <muralikpbhat@...>  
wrote:

> Hi,
> I got it resolved. https://issues.apache.org/jira/browse/HADOOP-5750 
> was causing this, even though I supplied a custom total ordering  
> partitioner, it didnt use that.
>
>
>   Now the regions looks properly sorted, but facing a new issue. The  
> last key of the each region is not retrievable. The table.jsp  page  
> shows the start and end key wrongly.
> for eg, take first 2 regions
> region1: start : end: 333305184e0f7c3e
> region2: start: 333305184e0f7c3e end: 666629fe4378c096
>
> The end key of first region = start key of second ??
>
> If I get the first and last key using HFile.Reader, it shows as  
> follows:
>
> HFileUtil /hbase/test12/98766318/image/9052388247118781160
> FirstKey:00000d7d4f36c112imagevalue�������
> LastKey:333305184e0f7c3eimagevalue�������
>
> HFileUtil /hbase/test12/170637321/image/7602871928600243730
> FirstKey:33338d45cc2491b8imagevalue�������
> LastKey:666629fe4378c096imagevalue�������
>
> So, according to this first key of 2nd region is 33338d45cc2491b8  
> not 333305184e0f7c3e which is correct!
>
> Now when I do a get on 333305184e0f7c3e with debug on, it is loading  
> the second region which is wrong!
>
> Some thing went wrong with the index?
>
> Thanks,
> Murali Krishna
>
>
>
>
> ________________________________
> From: stack <stack@...>
> To: hbase-user@...
> Sent: Sat, 7 November, 2009 6:26:03 AM
> Subject: Re: Issue with bulk loader tool
>
> On Fri, Nov 6, 2009 at 12:58 AM, Murali Krishna. P
> <muralikpbhat@...>wrote:
>
>> Hi,
>> If I increase hbase.hregion.max.filesize so that all the records  
>> holds in
>> one region (and one reducer ), all the records as retrievable. If one
>> reducer creates multiple hfile or multiple reducer creates one  
>> hfile each,
>> the problem occurs.
>>
>>
>
> Multiple hfiles in a region?  Or are you saying if a reducer creates
> multiple regions?  There is supposed to be one file per region only  
> when
> done.
>
> Thanks for digging in,
> St.Ack
>
>
>
>
>> Does that give any clue?
>>
>> Thanks,
>> Murali Krishna
>>
>>
>>
>>
>> ________________________________
>> From: Murali Krishna. P <muralikpbhat@...>
>> To: hbase-user@...
>> Sent: Thu, 5 November, 2009 6:34:20 PM
>> Subject: Re: Issue with bulk loader tool
>>
>> Hi Stack,
>> Sorry, could not look into this last week...
>>
>> I got problem with the Htable interface as well. Some records i am  
>> not
>> retrieve from Htable as well.
>> I lost the old table, but reproduced the problem with a different  
>> table.
>>
>> I cannot send the region since it is very huge. will try to give as  
>> much
>> info as possible here :)
>>
>> There are total 5 regions as below in that table:
>> Name
>>
>> Encoded Name
>> Start Key
>> End Key
>> test1,,1257414794600
>> 106817540
>> fffe9c7f87c8332a
>> test1,fffe9c7f87c8332a,1257414794616
>> 1346846599 fffe9c7f87c8332a fffebe279c0ac4d2
>> test1,fffebe279c0ac4d2,1257414794628
>> 1835851728 fffebe279c0ac4d2 fffec418284d6fbc
>> test1,fffec418284d6fbc,1257414794637
>> 1078205908 fffec418284d6fbc fffef7a12ea22498
>> test1,fffef7a12ea22498,1257414794647
>> 1515378663 fffef7a12ea22498
>>
>> I am looking for a key, say 000011d1bc8cd6fe . This should be in  
>> the first
>> region ?
>>
>> using hfile tool,
>> org.apache.hadoop.hbase.io.hfile.HFile -k -f
>> /hbase/test1/106817540/image/3828859735461759684 -v -m -p |  grep
>> 000011d1bc8cd6fe
>> The first region doesn't have it. Not sure what happened to that  
>> record.
>>
>> For a working key, it gives the record properly as below
>> K:
>> \x00\x100003bdd08ca88ee2\x05imagevalue\x7F\xFF\xFF\xFF\xFF\xFF\xFF
>> \xFF\x04
>> V: \xFF...
>>
>> Please let me know if you need more information
>>
>> Thanks,
>> Murali Krishna
>>
>>
>>
>>
>> ________________________________
>> From: stack <stack@...>
>> To: hbase-user@...
>> Sent: Mon, 2 November, 2009 11:05:43 PM
>> Subject: Re: Issue with bulk loader tool
>>
>> Murali:
>>
>> Any developments worth mentioning?
>>
>> St.Ack
>>
>>
>> On Fri, Oct 30, 2009 at 10:14 AM, stack <stack@...> wrote:
>>
>>> That is interesting.  It'd almost point to a shell issue.  Enable  
>>> DEBUG
>> so
>>> client can see it.  Then rerun shell.  Is it at least loading the  
>>> right
>>> region?  (The regions start and end keys span the asked for  
>>> key?).  I
>> took a
>>> look at your attached .META. scan.  All looks good there.  The  
>>> region
>>> specifications look right.  If you want to bundle up the region  
>>> that is
>>> failing -- the one that the failing key comes out of, I can take a  
>>> look
>>> here.  You could also try playing with the HFile tool: ./bin/hbase
>>> org.apache.hadoop.hbase.io.hfile.HFile.  Run the former and it'll  
>>> output
>>> usage.  You should be able to get it to dump content of the region  
>>> (You
>> need
>>> to supply flags like -v to see actual keys to the HFile tool else  
>>> it just
>>> runs its check silently).    Check for your key.  Check things like
>>> timestamp on it.  Maybe its 100 years in advance of now or  
>>> something?
>>>
>>> Yours,
>>> St.Ack
>>>
>>>
>>> On Fri, Oct 30, 2009 at 9:01 AM, Murali Krishna. P <
>> muralikpbhat@...
>>>> wrote:
>>>
>>>> Attached ".META"
>>>>
>>>> Interesting, I was able to get the row from HTable via java code.  
>>>> But
>> from
>>>> the shell, still getting following
>>>>
>>>> hbase(main):004:0> get 'TestTable2', 'ffffef95bcbf2638'
>>>> 0 row(s) in 1.2250 seconds
>>>>
>>>> Thanks,
>>>> Murali Krishna
>>>>
>>>> Thanks,
>>>> Murali Krishna
>>>>
>>>>
>>>> ------------------------------
>>>> *From:* stack <stack@...>
>>>> *To:* hbase-user@...
>>>> *Sent:* Fri, 30 October, 2009 8:39:46 PM
>>>> *Subject:* Re: Issue with bulk loader tool
>>>>
>>>> Can you send a listing of ".META."?
>>>>
>>>> hbase> scan ".META."
>>>>
>>>> Also, can you bring a region down from hdfs, tar and gzip it, and  
>>>> then
>> put
>>>> it someplace I can pull so I can take a look?
>>>>
>>>> Thanks,
>>>> St.Ack
>>>>
>>>>
>>>> On Fri, Oct 30, 2009 at 3:31 AM, Murali Krishna. P
>>>> <muralikpbhat@...>wrote:
>>>>
>>>>> Hi guys,
>>>>> I created a table according to hbase-48. A mapreduce job which
>> creates
>>>>> HFiles and then used loadtable.rb script to create the table.
>> Everything
>>>>> worked fine and i was able to scan the table. But when i do a  
>>>>> get for
>> a
>>>> key
>>>>> displayed in the scan output, it is not retrieving the row.  
>>>>> shell says
>> 0
>>>>> row.
>>>>>
>>>>> I tried using one reducer to ensure total ordering, but still same
>>>> issue.
>>>>>
>>>>>
>>>>> My mapper is like:
>>>>> context.write(new
>>>>> ImmutableBytesWritable(((Text)key).toString().getBytes()), new
>>>>> KeyValue(((Text)key).toString().getBytes(), "family1".getBytes(),
>>>>>                   "column1".getBytes(), getValueBytes()));
>>>>>
>>>>>
>>>>> Please help me investigate this.
>>>>>
>>>>> Thanks,
>>>>> Murali Krishna
>>>>>
>>>>
>>>
>>>
>>

Re: Issue with bulk loader tool

by Murali Krishna. P :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Thanks Lars for the clarification,
    But where does the record recide ? Is it duplicated to both the regions ?? When I use HFile.Reader, the first key in the second region is different. May be this behaviour(overlap) is only in .META. ?
    The issue is that when I request for that boundary record, it is loging the next region.

09/11/07 07:52:05 DEBUG client.HConnectionManager$TableServers: Cached location address: 76.13.20.58:60020, regioninfo: REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_META => 'true', MEMSTORE_FLUSHSIZE => '16384', FAMILIES => [{NAME => 'historian', VERSIONS => '2147483647', COMPRESSION => 'NONE', TTL => '604800', BLOCKSIZE => '8192', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', VERSIONS => '10', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '8192', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}
09/11/07 07:52:05 DEBUG client.HConnectionManager$TableServers: Cached location address: 76.13.20.114:60020, regioninfo: REGION => {NAME => 'test12,333305184e0f7c3e,1257515988652', STARTKEY => '333305184e0f7c3e', ENDKEY => '666629fe4378c096', ENCODED => 170637321, TABLE => {{NAME => 'test12', FAMILIES => [{NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}

 Thanks,
Murali Krishna




________________________________
From: Lars George <lars@...>
To: "hbase-user@..." <hbase-user@...>
Sent: Sat, 7 November, 2009 9:19:37 PM
Subject: Re: Issue with bulk loader tool

Hi Murali,

What you see is normal the last keys do indeed overlap. The last key of a region is exclusive and marks the first key of the subsequent region.

Lars

On Nov 7, 2009, at 9:05, "Murali Krishna. P" <muralikpbhat@...> wrote:

> Hi,
> I got it resolved. https://issues.apache.org/jira/browse/HADOOP-5750 was causing this, even though I supplied a custom total ordering partitioner, it didnt use that.
>
>
>   Now the regions looks properly sorted, but facing a new issue. The last key of the each region is not retrievable. The table.jsp  page shows the start and end key wrongly.
> for eg, take first 2 regions
> region1: start : end: 333305184e0f7c3e
> region2: start: 333305184e0f7c3e end: 666629fe4378c096
>
> The end key of first region = start key of second ??
>
> If I get the first and last key using HFile.Reader, it shows as follows:
>
> HFileUtil /hbase/test12/98766318/image/9052388247118781160
> FirstKey:00000d7d4f36c112imagevalue�������
> LastKey:333305184e0f7c3eimagevalue�������
>
> HFileUtil /hbase/test12/170637321/image/7602871928600243730
> FirstKey:33338d45cc2491b8imagevalue�������
> LastKey:666629fe4378c096imagevalue�������
>
> So, according to this first key of 2nd region is 33338d45cc2491b8 not 333305184e0f7c3e which is correct!
>
> Now when I do a get on 333305184e0f7c3e with debug on, it is loading the second region which is wrong!
>
> Some thing went wrong with the index?
>
> Thanks,
> Murali Krishna
>
>
>
>
> ________________________________
> From: stack <stack@...>
> To: hbase-user@...
> Sent: Sat, 7 November, 2009 6:26:03 AM
> Subject: Re: Issue with bulk loader tool
>
> On Fri, Nov 6, 2009 at 12:58 AM, Murali Krishna. P
> <muralikpbhat@...>wrote:
>
>> Hi,
>> If I increase hbase.hregion.max.filesize so that all the records holds in
>> one region (and one reducer ), all the records as retrievable. If one
>> reducer creates multiple hfile or multiple reducer creates one hfile each,
>> the problem occurs.
>>
>>
>
> Multiple hfiles in a region?  Or are you saying if a reducer creates
> multiple regions?  There is supposed to be one file per region only when
> done.
>
> Thanks for digging in,
> St.Ack
>
>
>
>
>> Does that give any clue?
>>
>> Thanks,
>> Murali Krishna
>>
>>
>>
>>
>> ________________________________
>> From: Murali Krishna. P <muralikpbhat@...>
>> To: hbase-user@...
>> Sent: Thu, 5 November, 2009 6:34:20 PM
>> Subject: Re: Issue with bulk loader tool
>>
>> Hi Stack,
>> Sorry, could not look into this last week...
>>
>> I got problem with the Htable interface as well. Some records i am not
>> retrieve from Htable as well.
>> I lost the old table, but reproduced the problem with a different table.
>>
>> I cannot send the region since it is very huge. will try to give as much
>> info as possible here :)
>>
>> There are total 5 regions as below in that table:
>> Name
>>
>> Encoded Name
>> Start Key
>> End Key
>> test1,,1257414794600
>> 106817540
>> fffe9c7f87c8332a
>> test1,fffe9c7f87c8332a,1257414794616
>> 1346846599 fffe9c7f87c8332a fffebe279c0ac4d2
>> test1,fffebe279c0ac4d2,1257414794628
>> 1835851728 fffebe279c0ac4d2 fffec418284d6fbc
>> test1,fffec418284d6fbc,1257414794637
>> 1078205908 fffec418284d6fbc fffef7a12ea22498
>> test1,fffef7a12ea22498,1257414794647
>> 1515378663 fffef7a12ea22498
>>
>> I am looking for a key, say 000011d1bc8cd6fe . This should be in the first
>> region ?
>>
>> using hfile tool,
>> org.apache.hadoop.hbase.io.hfile.HFile -k -f
>> /hbase/test1/106817540/image/3828859735461759684 -v -m -p |  grep
>> 000011d1bc8cd6fe
>> The first region doesn't have it. Not sure what happened to that record.
>>
>> For a working key, it gives the record properly as below
>> K:
>> \x00\x100003bdd08ca88ee2\x05imagevalue\x7F\xFF\xFF\xFF\xFF\xFF\xFF\xFF\x04
>> V: \xFF...
>>
>> Please let me know if you need more information
>>
>> Thanks,
>> Murali Krishna
>>
>>
>>
>>
>> ________________________________
>> From: stack <stack@...>
>> To: hbase-user@...
>> Sent: Mon, 2 November, 2009 11:05:43 PM
>> Subject: Re: Issue with bulk loader tool
>>
>> Murali:
>>
>> Any developments worth mentioning?
>>
>> St.Ack
>>
>>
>> On Fri, Oct 30, 2009 at 10:14 AM, stack <stack@...> wrote:
>>
>>> That is interesting.  It'd almost point to a shell issue.  Enable DEBUG
>> so
>>> client can see it.  Then rerun shell.  Is it at least loading the right
>>> region?  (The regions start and end keys span the asked for key?).  I
>> took a
>>> look at your attached .META. scan.  All looks good there.  The region
>>> specifications look right.  If you want to bundle up the region that is
>>> failing -- the one that the failing key comes out of, I can take a look
>>> here.  You could also try playing with the HFile tool: ./bin/hbase
>>> org.apache.hadoop.hbase.io.hfile.HFile.  Run the former and it'll output
>>> usage.  You should be able to get it to dump content of the region (You
>> need
>>> to supply flags like -v to see actual keys to the HFile tool else it just
>>> runs its check silently).    Check for your key.  Check things like
>>> timestamp on it.  Maybe its 100 years in advance of now or something?
>>>
>>> Yours,
>>> St.Ack
>>>
>>>
>>> On Fri, Oct 30, 2009 at 9:01 AM, Murali Krishna. P <
>> muralikpbhat@...
>>>> wrote:
>>>
>>>> Attached ".META"
>>>>
>>>> Interesting, I was able to get the row from HTable via java code. But
>> from
>>>> the shell, still getting following
>>>>
>>>> hbase(main):004:0> get 'TestTable2', 'ffffef95bcbf2638'
>>>> 0 row(s) in 1.2250 seconds
>>>>
>>>> Thanks,
>>>> Murali Krishna
>>>>
>>>> Thanks,
>>>> Murali Krishna
>>>>
>>>>
>>>> ------------------------------
>>>> *From:* stack <stack@...>
>>>> *To:* hbase-user@...
>>>> *Sent:* Fri, 30 October, 2009 8:39:46 PM
>>>> *Subject:* Re: Issue with bulk loader tool
>>>>
>>>> Can you send a listing of ".META."?
>>>>
>>>> hbase> scan ".META."
>>>>
>>>> Also, can you bring a region down from hdfs, tar and gzip it, and then
>> put
>>>> it someplace I can pull so I can take a look?
>>>>
>>>> Thanks,
>>>> St.Ack
>>>>
>>>>
>>>> On Fri, Oct 30, 2009 at 3:31 AM, Murali Krishna. P
>>>> <muralikpbhat@...>wrote:
>>>>
>>>>> Hi guys,
>>>>> I created a table according to hbase-48. A mapreduce job which
>> creates
>>>>> HFiles and then used loadtable.rb script to create the table.
>> Everything
>>>>> worked fine and i was able to scan the table. But when i do a get for
>> a
>>>> key
>>>>> displayed in the scan output, it is not retrieving the row. shell says
>> 0
>>>>> row.
>>>>>
>>>>> I tried using one reducer to ensure total ordering, but still same
>>>> issue.
>>>>>
>>>>>
>>>>> My mapper is like:
>>>>> context.write(new
>>>>> ImmutableBytesWritable(((Text)key).toString().getBytes()), new
>>>>> KeyValue(((Text)key).toString().getBytes(), "family1".getBytes(),
>>>>>                   "column1".getBytes(), getValueBytes()));
>>>>>
>>>>>
>>>>> Please help me investigate this.
>>>>>
>>>>> Thanks,
>>>>> Murali Krishna
>>>>>
>>>>
>>>
>>>
>>

Re: Issue with bulk loader tool

by stack-3 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Its what Lars says Murali, a region's startkey is inclusive and its endkey
exclusive.  If it exists, it should be in the region has it for a start key
(It will not be duplicated in both).

For .META., there is usually only one Region instance in a .META. table.
Its startkey will be the empty key so its not suprirising its first key is
different from the empty key.  What do you see when you look at the second
region in your just uploaded table?  I'd expect the key 666629fe4378c096 to
be first in the region whose startkey is 666629fe4378c096.

Thanks for figuring MAPREDUCE-565 could trip us up.  Your hadoop is not
0.20.1?

Yours,
St.Ack



On Sat, Nov 7, 2009 at 7:58 AM, Murali Krishna. P <muralikpbhat@...>wrote:

> Thanks Lars for the clarification,
>    But where does the record recide ? Is it duplicated to both the regions
> ?? When I use HFile.Reader, the first key in the second region is different.
> May be this behaviour(overlap) is only in .META. ?
>    The issue is that when I request for that boundary record, it is loging
> the next region.
>
> 09/11/07 07:52:05 DEBUG client.HConnectionManager$TableServers: Cached
> location address: 76.13.20.58:60020, regioninfo: REGION => {NAME =>
> '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE =>
> {{NAME => '.META.', IS_META => 'true', MEMSTORE_FLUSHSIZE => '16384',
> FAMILIES => [{NAME => 'historian', VERSIONS => '2147483647', COMPRESSION =>
> 'NONE', TTL => '604800', BLOCKSIZE => '8192', IN_MEMORY => 'false',
> BLOCKCACHE => 'false'}, {NAME => 'info', VERSIONS => '10', COMPRESSION =>
> 'NONE', TTL => '2147483647', BLOCKSIZE => '8192', IN_MEMORY => 'false',
> BLOCKCACHE => 'false'}]}}
> 09/11/07 07:52:05 DEBUG client.HConnectionManager$TableServers: Cached
> location address: 76.13.20.114:60020, regioninfo: REGION => {NAME =>
> 'test12,333305184e0f7c3e,1257515988652', STARTKEY => '333305184e0f7c3e',
> ENDKEY => '666629fe4378c096', ENCODED => 170637321, TABLE => {{NAME =>
> 'test12', FAMILIES => [{NAME => 'image', VERSIONS => '3', COMPRESSION =>
> 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false',
> BLOCKCACHE => 'true'}]}}
>
>  Thanks,
> Murali Krishna
>
>
>
>
> ________________________________
> From: Lars George <lars@...>
> To: "hbase-user@..." <hbase-user@...>
> Sent: Sat, 7 November, 2009 9:19:37 PM
> Subject: Re: Issue with bulk loader tool
>
> Hi Murali,
>
> What you see is normal the last keys do indeed overlap. The last key of a
> region is exclusive and marks the first key of the subsequent region.
>
> Lars
>
> On Nov 7, 2009, at 9:05, "Murali Krishna. P" <muralikpbhat@...>
> wrote:
>
> > Hi,
> > I got it resolved. https://issues.apache.org/jira/browse/HADOOP-5750 was
> causing this, even though I supplied a custom total ordering partitioner, it
> didnt use that.
> >
> >
> >   Now the regions looks properly sorted, but facing a new issue. The last
> key of the each region is not retrievable. The table.jsp  page shows the
> start and end key wrongly.
> > for eg, take first 2 regions
> > region1: start : end: 333305184e0f7c3e
> > region2: start: 333305184e0f7c3e end: 666629fe4378c096
> >
> > The end key of first region = start key of second ??
> >
> > If I get the first and last key using HFile.Reader, it shows as follows:
> >
> > HFileUtil /hbase/test12/98766318/image/9052388247118781160
> > FirstKey:00000d7d4f36c112imagevalue�������
> > LastKey:333305184e0f7c3eimagevalue�������
> >
> > HFileUtil /hbase/test12/170637321/image/7602871928600243730
> > FirstKey:33338d45cc2491b8imagevalue�������
> > LastKey:666629fe4378c096imagevalue�������
> >
> > So, according to this first key of 2nd region is 33338d45cc2491b8 not
> 333305184e0f7c3e which is correct!
> >
> > Now when I do a get on 333305184e0f7c3e with debug on, it is loading the
> second region which is wrong!
> >
> > Some thing went wrong with the index?
> >
> > Thanks,
> > Murali Krishna
> >
> >
> >
> >
> > ________________________________
> > From: stack <stack@...>
> > To: hbase-user@...
> > Sent: Sat, 7 November, 2009 6:26:03 AM
> > Subject: Re: Issue with bulk loader tool
> >
> > On Fri, Nov 6, 2009 at 12:58 AM, Murali Krishna. P
> > <muralikpbhat@...>wrote:
> >
> >> Hi,
> >> If I increase hbase.hregion.max.filesize so that all the records holds
> in
> >> one region (and one reducer ), all the records as retrievable. If one
> >> reducer creates multiple hfile or multiple reducer creates one hfile
> each,
> >> the problem occurs.
> >>
> >>
> >
> > Multiple hfiles in a region?  Or are you saying if a reducer creates
> > multiple regions?  There is supposed to be one file per region only when
> > done.
> >
> > Thanks for digging in,
> > St.Ack
> >
> >
> >
> >
> >> Does that give any clue?
> >>
> >> Thanks,
> >> Murali Krishna
> >>
> >>
> >>
> >>
> >> ________________________________
> >> From: Murali Krishna. P <muralikpbhat@...>
> >> To: hbase-user@...
> >> Sent: Thu, 5 November, 2009 6:34:20 PM
> >> Subject: Re: Issue with bulk loader tool
> >>
> >> Hi Stack,
> >> Sorry, could not look into this last week...
> >>
> >> I got problem with the Htable interface as well. Some records i am not
> >> retrieve from Htable as well.
> >> I lost the old table, but reproduced the problem with a different table.
> >>
> >> I cannot send the region since it is very huge. will try to give as much
> >> info as possible here :)
> >>
> >> There are total 5 regions as below in that table:
> >> Name
> >>
> >> Encoded Name
> >> Start Key
> >> End Key
> >> test1,,1257414794600
> >> 106817540
> >> fffe9c7f87c8332a
> >> test1,fffe9c7f87c8332a,1257414794616
> >> 1346846599 fffe9c7f87c8332a fffebe279c0ac4d2
> >> test1,fffebe279c0ac4d2,1257414794628
> >> 1835851728 fffebe279c0ac4d2 fffec418284d6fbc
> >> test1,fffec418284d6fbc,1257414794637
> >> 1078205908 fffec418284d6fbc fffef7a12ea22498
> >> test1,fffef7a12ea22498,1257414794647
> >> 1515378663 fffef7a12ea22498
> >>
> >> I am looking for a key, say 000011d1bc8cd6fe . This should be in the
> first
> >> region ?
> >>
> >> using hfile tool,
> >> org.apache.hadoop.hbase.io.hfile.HFile -k -f
> >> /hbase/test1/106817540/image/3828859735461759684 -v -m -p |  grep
> >> 000011d1bc8cd6fe
> >> The first region doesn't have it. Not sure what happened to that record.
> >>
> >> For a working key, it gives the record properly as below
> >> K:
> >>
> \x00\x100003bdd08ca88ee2\x05imagevalue\x7F\xFF\xFF\xFF\xFF\xFF\xFF\xFF\x04
> >> V: \xFF...
> >>
> >> Please let me know if you need more information
> >>
> >> Thanks,
> >> Murali Krishna
> >>
> >>
> >>
> >>
> >> ________________________________
> >> From: stack <stack@...>
> >> To: hbase-user@...
> >> Sent: Mon, 2 November, 2009 11:05:43 PM
> >> Subject: Re: Issue with bulk loader tool
> >>
> >> Murali:
> >>
> >> Any developments worth mentioning?
> >>
> >> St.Ack
> >>
> >>
> >> On Fri, Oct 30, 2009 at 10:14 AM, stack <stack@...> wrote:
> >>
> >>> That is interesting.  It'd almost point to a shell issue.  Enable DEBUG
> >> so
> >>> client can see it.  Then rerun shell.  Is it at least loading the right
> >>> region?  (The regions start and end keys span the asked for key?).  I
> >> took a
> >>> look at your attached .META. scan.  All looks good there.  The region
> >>> specifications look right.  If you want to bundle up the region that is
> >>> failing -- the one that the failing key comes out of, I can take a look
> >>> here.  You could also try playing with the HFile tool: ./bin/hbase
> >>> org.apache.hadoop.hbase.io.hfile.HFile.  Run the former and it'll
> output
> >>> usage.  You should be able to get it to dump content of the region (You
> >> need
> >>> to supply flags like -v to see actual keys to the HFile tool else it
> just
> >>> runs its check silently).    Check for your key.  Check things like
> >>> timestamp on it.  Maybe its 100 years in advance of now or something?
> >>>
> >>> Yours,
> >>> St.Ack
> >>>
> >>>
> >>> On Fri, Oct 30, 2009 at 9:01 AM, Murali Krishna. P <
> >> muralikpbhat@...
> >>>> wrote:
> >>>
> >>>> Attached ".META"
> >>>>
> >>>> Interesting, I was able to get the row from HTable via java code. But
> >> from
> >>>> the shell, still getting following
> >>>>
> >>>> hbase(main):004:0> get 'TestTable2', 'ffffef95bcbf2638'
> >>>> 0 row(s) in 1.2250 seconds
> >>>>
> >>>> Thanks,
> >>>> Murali Krishna
> >>>>
> >>>> Thanks,
> >>>> Murali Krishna
> >>>>
> >>>>
> >>>> ------------------------------
> >>>> *From:* stack <stack@...>
> >>>> *To:* hbase-user@...
> >>>> *Sent:* Fri, 30 October, 2009 8:39:46 PM
> >>>> *Subject:* Re: Issue with bulk loader tool
> >>>>
> >>>> Can you send a listing of ".META."?
> >>>>
> >>>> hbase> scan ".META."
> >>>>
> >>>> Also, can you bring a region down from hdfs, tar and gzip it, and then
> >> put
> >>>> it someplace I can pull so I can take a look?
> >>>>
> >>>> Thanks,
> >>>> St.Ack
> >>>>
> >>>>
> >>>> On Fri, Oct 30, 2009 at 3:31 AM, Murali Krishna. P
> >>>> <muralikpbhat@...>wrote:
> >>>>
> >>>>> Hi guys,
> >>>>> I created a table according to hbase-48. A mapreduce job which
> >> creates
> >>>>> HFiles and then used loadtable.rb script to create the table.
> >> Everything
> >>>>> worked fine and i was able to scan the table. But when i do a get for
> >> a
> >>>> key
> >>>>> displayed in the scan output, it is not retrieving the row. shell
> says
> >> 0
> >>>>> row.
> >>>>>
> >>>>> I tried using one reducer to ensure total ordering, but still same
> >>>> issue.
> >>>>>
> >>>>>
> >>>>> My mapper is like:
> >>>>> context.write(new
> >>>>> ImmutableBytesWritable(((Text)key).toString().getBytes()), new
> >>>>> KeyValue(((Text)key).toString().getBytes(), "family1".getBytes(),
> >>>>>                   "column1".getBytes(), getValueBytes()));
> >>>>>
> >>>>>
> >>>>> Please help me investigate this.
> >>>>>
> >>>>> Thanks,
> >>>>> Murali Krishna
> >>>>>
> >>>>
> >>>
> >>>
> >>
>

Re: Issue with bulk loader tool

by Murali Krishna. P :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Some parts of this message have been removed. Learn more about Nabble's security policy.
No, the first key is 6666909d611e8d7e for the region which says startKey is 666629fe4378c096.
(this is actually the next key in the order).

HFile -p:-
Scanning -> /hbase/test12/336573097/image/2362265315474952099
K: \x00\x106666909d611e8d7e\x05imagevalue\x7F\x..

 HFileUtil /hbase/test12/336573097/image/2362265315474952099 :-
FirstKey:6666909d611e8d7eimagevalue�������
LastKey:99998c8f356b0d86imagevalue�������

But the scan .META. shows the start key as 666629fe4378c096. (attached .META.)

This seems to be the case for all the regions. (the actual firstKey is next one from claimed firstKey)

I am on hadoop0.20.0
 
Thanks,
Murali Krishna



From: stack <stack@...>
To: hbase-user@...
Sent: Sun, 8 November, 2009 4:30:15 AM
Subject: Re: Issue with bulk loader tool

Its what Lars says Murali, a region's startkey is inclusive and its endkey
exclusive.  If it exists, it should be in the region has it for a start key
(It will not be duplicated in both).

For .META., there is usually only one Region instance in a .META. table.
Its startkey will be the empty key so its not suprirising its first key is
different from the empty key.  What do you see when you look at the second
region in your just uploaded table?  I'd expect the key 666629fe4378c096 to
be first in the region whose startkey is 666629fe4378c096.

Thanks for figuring MAPREDUCE-565 could trip us up.  Your hadoop is not
0.20.1?

Yours,
St.Ack



On Sat, Nov 7, 2009 at 7:58 AM, Murali Krishna. P <muralikpbhat@...>wrote:

> Thanks Lars for the clarification,
>    But where does the record recide ? Is it duplicated to both the regions
> ?? When I use HFile.Reader, the first key in the second region is different.
> May be this behaviour(overlap) is only in .META. ?
>    The issue is that when I request for that boundary record, it is loging
> the next region.
>
> 09/11/07 07:52:05 DEBUG client.HConnectionManager$TableServers: Cached
> location address: 76.13.20.58:60020, regioninfo: REGION => {NAME =>
> '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE =>
> {{NAME => '.META.', IS_META => 'true', MEMSTORE_FLUSHSIZE => '16384',
> FAMILIES => [{NAME => 'historian', VERSIONS => '2147483647', COMPRESSION =>
> 'NONE', TTL => '604800', BLOCKSIZE => '8192', IN_MEMORY => 'false',
> BLOCKCACHE => 'false'}, {NAME => 'info', VERSIONS => '10', COMPRESSION =>
> 'NONE', TTL => '2147483647', BLOCKSIZE => '8192', IN_MEMORY => 'false',
> BLOCKCACHE => 'false'}]}}
> 09/11/07 07:52:05 DEBUG client.HConnectionManager$TableServers: Cached
> location address: 76.13.20.114:60020, regioninfo: REGION => {NAME =>
> 'test12,333305184e0f7c3e,1257515988652', STARTKEY => '333305184e0f7c3e',
> ENDKEY => '666629fe4378c096', ENCODED => 170637321, TABLE => {{NAME =>
> 'test12', FAMILIES => [{NAME => 'image', VERSIONS => '3', COMPRESSION =>
> 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false',
> BLOCKCACHE => 'true'}]}}
>
>  Thanks,
> Murali Krishna
>
>
>
>
> ________________________________
> From: Lars George <lars@...>
> To: "hbase-user@..." <hbase-user@...>
> Sent: Sat, 7 November, 2009 9:19:37 PM
> Subject: Re: Issue with bulk loader tool
>
> Hi Murali,
>
> What you see is normal the last keys do indeed overlap. The last key of a
> region is exclusive and marks the first key of the subsequent region.
>
> Lars
>
> On Nov 7, 2009, at 9:05, "Murali Krishna. P" <muralikpbhat@...>
> wrote:
>
> > Hi,
> > I got it resolved. https://issues.apache.org/jira/browse/HADOOP-5750 was
> causing this, even though I supplied a custom total ordering partitioner, it
> didnt use that.
> >
> >
> >  Now the regions looks properly sorted, but facing a new issue. The last
> key of the each region is not retrievable. The table.jsp  page shows the
> start and end key wrongly.
> > for eg, take first 2 regions
> > region1: start : end: 333305184e0f7c3e
> > region2: start: 333305184e0f7c3e end: 666629fe4378c096
> >
> > The end key of first region = start key of second ??
> >
> > If I get the first and last key using HFile.Reader, it shows as follows:
> >
> > HFileUtil /hbase/test12/98766318/image/9052388247118781160
> > FirstKey:00000d7d4f36c112imagevalue�������
> > LastKey:333305184e0f7c3eimagevalue�������
> >
> > HFileUtil /hbase/test12/170637321/image/7602871928600243730
> > FirstKey:33338d45cc2491b8imagevalue�������
> > LastKey:666629fe4378c096imagevalue�������
> >
> > So, according to this first key of 2nd region is 33338d45cc2491b8 not
> 333305184e0f7c3e which is correct!
> >
> > Now when I do a get on 333305184e0f7c3e with debug on, it is loading the
> second region which is wrong!
> >
> > Some thing went wrong with the index?
> >
> > Thanks,
> > Murali Krishna
> >
> >
> >
> >
> > ________________________________
> > From: stack <stack@...>
> > To: hbase-user@...
> > Sent: Sat, 7 November, 2009 6:26:03 AM
> > Subject: Re: Issue with bulk loader tool
> >
> > On Fri, Nov 6, 2009 at 12:58 AM, Murali Krishna. P
> > <muralikpbhat@...>wrote:
> >
> >> Hi,
> >> If I increase hbase.hregion.max.filesize so that all the records holds
> in
> >> one region (and one reducer ), all the records as retrievable. If one
> >> reducer creates multiple hfile or multiple reducer creates one hfile
> each,
> >> the problem occurs.
> >>
> >>
> >
> > Multiple hfiles in a region?  Or are you saying if a reducer creates
> > multiple regions?  There is supposed to be one file per region only when
> > done.
> >
> > Thanks for digging in,
> > St.Ack
> >
> >
> >
> >
> >> Does that give any clue?
> >>
> >> Thanks,
> >> Murali Krishna
> >>
> >>
> >>
> >>
> >> ________________________________
> >> From: Murali Krishna. P <muralikpbhat@...>
> >> To: hbase-user@...
> >> Sent: Thu, 5 November, 2009 6:34:20 PM
> >> Subject: Re: Issue with bulk loader tool
> >>
> >> Hi Stack,
> >> Sorry, could not look into this last week...
> >>
> >> I got problem with the Htable interface as well. Some records i am not
> >> retrieve from Htable as well.
> >> I lost the old table, but reproduced the problem with a different table.
> >>
> >> I cannot send the region since it is very huge. will try to give as much
> >> info as possible here :)
> >>
> >> There are total 5 regions as below in that table:
> >> Name
> >>
> >> Encoded Name
> >> Start Key
> >> End Key
> >> test1,,1257414794600
> >> 106817540
> >> fffe9c7f87c8332a
> >> test1,fffe9c7f87c8332a,1257414794616
> >> 1346846599 fffe9c7f87c8332a fffebe279c0ac4d2
> >> test1,fffebe279c0ac4d2,1257414794628
> >> 1835851728 fffebe279c0ac4d2 fffec418284d6fbc
> >> test1,fffec418284d6fbc,1257414794637
> >> 1078205908 fffec418284d6fbc fffef7a12ea22498
> >> test1,fffef7a12ea22498,1257414794647
> >> 1515378663 fffef7a12ea22498
> >>
> >> I am looking for a key, say 000011d1bc8cd6fe . This should be in the
> first
> >> region ?
> >>
> >> using hfile tool,
> >> org.apache.hadoop.hbase.io.hfile.HFile -k -f
> >> /hbase/test1/106817540/image/3828859735461759684 -v -m -p |  grep
> >> 000011d1bc8cd6fe
> >> The first region doesn't have it. Not sure what happened to that record.
> >>
> >> For a working key, it gives the record properly as below
> >> K:
> >>
> \x00\x100003bdd08ca88ee2\x05imagevalue\x7F\xFF\xFF\xFF\xFF\xFF\xFF\xFF\x04
> >> V: \xFF...
> >>
> >> Please let me know if you need more information
> >>
> >> Thanks,
> >> Murali Krishna
> >>
> >>
> >>
> >>
> >> ________________________________
> >> From: stack <stack@...>
> >> To: hbase-user@...
> >> Sent: Mon, 2 November, 2009 11:05:43 PM
> >> Subject: Re: Issue with bulk loader tool
> >>
> >> Murali:
> >>
> >> Any developments worth mentioning?
> >>
> >> St.Ack
> >>
> >>
> >> On Fri, Oct 30, 2009 at 10:14 AM, stack <stack@...> wrote:
> >>
> >>> That is interesting.  It'd almost point to a shell issue.  Enable DEBUG
> >> so
> >>> client can see it.  Then rerun shell.  Is it at least loading the right
> >>> region?  (The regions start and end keys span the asked for key?).  I
> >> took a
> >>> look at your attached .META. scan.  All looks good there.  The region
> >>> specifications look right.  If you want to bundle up the region that is
> >>> failing -- the one that the failing key comes out of, I can take a look
> >>> here.  You could also try playing with the HFile tool: ./bin/hbase
> >>> org.apache.hadoop.hbase.io.hfile.HFile.  Run the former and it'll
> output
> >>> usage.  You should be able to get it to dump content of the region (You
> >> need
> >>> to supply flags like -v to see actual keys to the HFile tool else it
> just
> >>> runs its check silently).    Check for your key.  Check things like
> >>> timestamp on it.  Maybe its 100 years in advance of now or something?
> >>>
> >>> Yours,
> >>> St.Ack
> >>>
> >>>
> >>> On Fri, Oct 30, 2009 at 9:01 AM, Murali Krishna. P <
> >> muralikpbhat@...
> >>>> wrote:
> >>>
> >>>> Attached ".META"
> >>>>
> >>>> Interesting, I was able to get the row from HTable via java code. But
> >> from
> >>>> the shell, still getting following
> >>>>
> >>>> hbase(main):004:0> get 'TestTable2', 'ffffef95bcbf2638'
> >>>> 0 row(s) in 1.2250 seconds
> >>>>
> >>>> Thanks,
> >>>> Murali Krishna
> >>>>
> >>>> Thanks,
> >>>> Murali Krishna
> >>>>
> >>>>
> >>>> ------------------------------
> >>>> *From:* stack <stack@...>
> >>>> *To:* hbase-user@...
> >>>> *Sent:* Fri, 30 October, 2009 8:39:46 PM
> >>>> *Subject:* Re: Issue with bulk loader tool
> >>>>
> >>>> Can you send a listing of ".META."?
> >>>>
> >>>> hbase> scan ".META."
> >>>>
> >>>> Also, can you bring a region down from hdfs, tar and gzip it, and then
> >> put
> >>>> it someplace I can pull so I can take a look?
> >>>>
> >>>> Thanks,
> >>>> St.Ack
> >>>>
> >>>>
> >>>> On Fri, Oct 30, 2009 at 3:31 AM, Murali Krishna. P
> >>>> <muralikpbhat@...>wrote:
> >>>>
> >>>>> Hi guys,
> >>>>> I created a table according to hbase-48. A mapreduce job which
> >> creates
> >>>>> HFiles and then used loadtable.rb script to create the table.
> >> Everything
> >>>>> worked fine and i was able to scan the table. But when i do a get for
> >> a
> >>>> key
> >>>>> displayed in the scan output, it is not retrieving the row. shell
> says
> >> 0
> >>>>> row.
> >>>>>
> >>>>> I tried using one reducer to ensure total ordering, but still same
> >>>> issue.
> >>>>>
> >>>>>
> >>>>> My mapper is like:
> >>>>> context.write(new
> >>>>> ImmutableBytesWritable(((Text)key).toString().getBytes()), new
> >>>>> KeyValue(((Text)key).toString().getBytes(), "family1".getBytes(),
> >>>>>                  "column1".getBytes(), getValueBytes()));
> >>>>>
> >>>>>
> >>>>> Please help me investigate this.
> >>>>>
> >>>>> Thanks,
> >>>>> Murali Krishna
> >>>>>
> >>>>
> >>>
> >>>
> >>
>

09/11/07 21:13:58 DEBUG client.HConnectionManager$ClientZKWatcher: Got ZooKeeper event, state: SyncConnected, type: None, path: null
09/11/07 21:13:58 DEBUG zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 76.13.20.57:60020
09/11/07 21:13:59 DEBUG client.HConnectionManager$TableServers: Found ROOT at 76.13.20.57:60020
09/11/07 21:13:59 DEBUG client.HConnectionManager$TableServers: Cached location address: 76.13.20.58:60020, regioninfo: REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_META => 'true', MEMSTORE_FLUSHSIZE => '16384', FAMILIES => [{NAME => 'historian', VERSIONS => '2147483647', COMPRESSION => 'NONE', TTL => '604800', BLOCKSIZE => '8192', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', VERSIONS => '10', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '8192', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}
09/11/07 21:14:00 DEBUG client.HTable$ClientScanner: Creating scanner over .META. starting at key ''
09/11/07 21:14:00 DEBUG client.HTable$ClientScanner: Advancing internal scanner to startKey at ''
09/11/07 21:14:00 DEBUG client.HConnectionManager$TableServers: Cache hit for row <> in tableName .META.: location server 76.13.20.58:60020, location region name .META.,,1
ROW                          COLUMN+CELL                                                                      
 test12,,1257515988637       column=info:regioninfo, timestamp=1257515988651, value=REGION => {NAME => 'test12
                             ,,1257515988637', STARTKEY => '', ENDKEY => '333305184e0f7c3e', ENCODED => 987663
                             18, TABLE => {{NAME => 'test12', FAMILIES => [{NAME => 'image', VERSIONS => '3',
                             COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'f
                             alse', BLOCKCACHE => 'true'}]}}                                                  
 test12,,1257515988637       column=info:server, timestamp=1257516025194, value=76.13.20.114:60020            
 test12,,1257515988637       column=info:serverstartcode, timestamp=1257516025194, value=1257515423160        
 test12,333305184e0f7c3e,125 column=info:regioninfo, timestamp=1257515988659, value=REGION => {NAME => 'test12
 7515988652                  ,333305184e0f7c3e,1257515988652', STARTKEY => '333305184e0f7c3e', ENDKEY => '6666
                             29fe4378c096', ENCODED => 170637321, TABLE => {{NAME => 'test12', FAMILIES => [{N
                             AME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOC
                             KSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 test12,333305184e0f7c3e,125 column=info:server, timestamp=1257516025193, value=76.13.20.114:60020            
 7515988652                                                                                                  
 test12,333305184e0f7c3e,125 column=info:serverstartcode, timestamp=1257516025193, value=1257515423160        
 7515988652                                                                                                  
 test12,666629fe4378c096,125 column=info:regioninfo, timestamp=1257515988664, value=REGION => {NAME => 'test12
 7515988660                  ,666629fe4378c096,1257515988660', STARTKEY => '666629fe4378c096', ENDKEY => '9999
                             8c8f356b0d86', ENCODED => 336573097, TABLE => {{NAME => 'test12', FAMILIES => [{N
                             AME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOC
                             KSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 test12,666629fe4378c096,125 column=info:server, timestamp=1257516024481, value=76.13.20.58:60020            
 7515988660                                                                                                  
 test12,666629fe4378c096,125 column=info:serverstartcode, timestamp=1257516024481, value=1257515423025        
 7515988660                                                                                                  
 test12,99998c8f356b0d86,125 column=info:regioninfo, timestamp=1257515988672, value=REGION => {NAME => 'test12
 7515988665                  ,99998c8f356b0d86,1257515988665', STARTKEY => '99998c8f356b0d86', ENDKEY => 'cccc
                             98546ff51552', ENCODED => 1765503814, TABLE => {{NAME => 'test12', FAMILIES => [{
                             NAME => 'image', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLO
                             CKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                
 test12,99998c8f356b0d86,125 column=info:server, timestamp=1257516024377, value=76.13.20.57:60020            
 7515988665                                                                                                  
 test12,99998c8f356b0d86,125 column=info:serverstartcode, timestamp=1257516024377, value=1257515422212        
 7515988665                                                                                                  
 test12,cccc98546ff51552,125 column=info:regioninfo, timestamp=1257515988679, value=REGION => {NAME => 'test12
 7515988673                  ,cccc98546ff51552,1257515988673', STARTKEY => 'cccc98546ff51552', ENDKEY => '', E
                             NCODED => 777800369, TABLE => {{NAME => 'test12', FAMILIES => [{NAME => 'image',
                             VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536'
                             , IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}                                
 test12,cccc98546ff51552,125 column=info:server, timestamp=1257516025133, value=76.13.20.56:60020            
 7515988673                                                                                                  
 test12,cccc98546ff51552,125 column=info:serverstartcode, timestamp=1257516025133, value=1257515423001        
 7515988673                                                                                                  
09/11/07 21:14:00 DEBUG client.HTable$ClientScanner: Finished with scanning at REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_META => 'true', MEMSTORE_FLUSHSIZE => '16384', FAMILIES => [{NAME => 'historian', VERSIONS => '2147483647', COMPRESSION => 'NONE', TTL => '604800', BLOCKSIZE => '8192', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', VERSIONS => '10', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '8192', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}
5 row(s) in 0.2660 seconds

Re: Issue with bulk loader tool

by stack-3 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

So, do you think we are dropping the first key in the region?
Thanks,
St.Ack

On Sat, Nov 7, 2009 at 9:17 PM, Murali Krishna. P <muralikpbhat@...>wrote:

> No, the first key is 6666909d611e8d7e for the region which says startKey is
> 666629fe4378c096.
> (this is actually the next key in the order).
>
> HFile -p:-
> Scanning -> /hbase/test12/336573097/image/2362265315474952099
> K: \x00\x106666909d611e8d7e\x05imagevalue\x7F\x..
>
>  HFileUtil /hbase/test12/336573097/image/2362265315474952099 :-
> FirstKey:6666909d611e8d7eimagevalue�������
> LastKey:99998c8f356b0d86imagevalue�������
>
> But the scan .META. shows the start key as 666629fe4378c096. (attached
> .META.)
>
> This seems to be the case for all the regions. (the actual firstKey is next
> one from claimed firstKey)
>
> I am on hadoop0.20.0
>
> Thanks,
> Murali Krishna
>
>
> ------------------------------
> *From:* stack <stack@...>
> *To:* hbase-user@...
> *Sent:* Sun, 8 November, 2009 4:30:15 AM
>
> *Subject:* Re: Issue with bulk loader tool
>
> Its what Lars says Murali, a region's startkey is inclusive and its endkey
> exclusive.  If it exists, it should be in the region has it for a start key
> (It will not be duplicated in both).
>
> For .META., there is usually only one Region instance in a .META. table.
> Its startkey will be the empty key so its not suprirising its first key is
> different from the empty key.  What do you see when you look at the second
> region in your just uploaded table?  I'd expect the key 666629fe4378c096 to
> be first in the region whose startkey is 666629fe4378c096.
>
> Thanks for figuring MAPREDUCE-565 could trip us up.  Your hadoop is not
> 0.20.1?
>
> Yours,
> St.Ack
>
>
>
> On Sat, Nov 7, 2009 at 7:58 AM, Murali Krishna. P <muralikpbhat@...
> >wrote:
>
> > Thanks Lars for the clarification,
> >    But where does the record recide ? Is it duplicated to both the
> regions
> > ?? When I use HFile.Reader, the first key in the second region is
> different.
> > May be this behaviour(overlap) is only in .META. ?
> >    The issue is that when I request for that boundary record, it is
> loging
> > the next region.
> >
> > 09/11/07 07:52:05 DEBUG client.HConnectionManager$TableServers: Cached
> > location address: 76.13.20.58:60020, regioninfo: REGION => {NAME =>
> > '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE
> =>
> > {{NAME => '.META.', IS_META => 'true', MEMSTORE_FLUSHSIZE => '16384',
> > FAMILIES => [{NAME => 'historian', VERSIONS => '2147483647', COMPRESSION
> =>
> > 'NONE', TTL => '604800', BLOCKSIZE => '8192', IN_MEMORY => 'false',
> > BLOCKCACHE => 'false'}, {NAME => 'info', VERSIONS => '10', COMPRESSION =>
> > 'NONE', TTL => '2147483647', BLOCKSIZE => '8192', IN_MEMORY => 'false',
> > BLOCKCACHE => 'false'}]}}
> > 09/11/07 07:52:05 DEBUG client.HConnectionManager$TableServers: Cached
> > location address: 76.13.20.114:60020, regioninfo: REGION => {NAME =>
> > 'test12,333305184e0f7c3e,1257515988652', STARTKEY => '333305184e0f7c3e',
> > ENDKEY => '666629fe4378c096', ENCODED => 170637321, TABLE => {{NAME =>
> > 'test12', FAMILIES => [{NAME => 'image', VERSIONS => '3', COMPRESSION =>
> > 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false',
> > BLOCKCACHE => 'true'}]}}
> >
> >  Thanks,
> > Murali Krishna
> >
> >
> >
> >
> > ________________________________
> > From: Lars George <lars@...>
> > To: "hbase-user@..." <hbase-user@...>
> > Sent: Sat, 7 November, 2009 9:19:37 PM
> > Subject: Re: Issue with bulk loader tool
> >
> > Hi Murali,
> >
> > What you see is normal the last keys do indeed overlap. The last key of a
> > region is exclusive and marks the first key of the subsequent region.
> >
> > Lars
> >
> > On Nov 7, 2009, at 9:05, "Murali Krishna. P" <muralikpbhat@...>
> > wrote:
> >
> > > Hi,
> > > I got it resolved. https://issues.apache.org/jira/browse/HADOOP-5750was
> > causing this, even though I supplied a custom total ordering partitioner,
> it
> > didnt use that.
> > >
> > >
> > >  Now the regions looks properly sorted, but facing a new issue. The
> last
> > key of the each region is not retrievable. The table.jsp  page shows the
> > start and end key wrongly.
> > > for eg, take first 2 regions
> > > region1: start : end: 333305184e0f7c3e
> > > region2: start: 333305184e0f7c3e end: 666629fe4378c096
> > >
> > > The end key of first region = start key of second ??
> > >
> > > If I get the first and last key using HFile.Reader, it shows as
> follows:
> > >
> > > HFileUtil /hbase/test12/98766318/image/9052388247118781160
> > > FirstKey:00000d7d4f36c112imagevalue�������
> > > LastKey:333305184e0f7c3eimagevalue�������
> > >
> > > HFileUtil /hbase/test12/170637321/image/7602871928600243730
> > > FirstKey:33338d45cc2491b8imagevalue�������
> > > LastKey:666629fe4378c096imagevalue�������
> > >
> > > So, according to this first key of 2nd region is 33338d45cc2491b8 not
> > 333305184e0f7c3e which is correct!
> > >
> > > Now when I do a get on 333305184e0f7c3e with debug on, it is loading
> the
> > second region which is wrong!
> > >
> > > Some thing went wrong with the index?
> > >
> > > Thanks,
> > > Murali Krishna
> > >
> > >
> > >
> > >
> > > ________________________________
> > > From: stack <stack@...>
> > > To: hbase-user@...
> > > Sent: Sat, 7 November, 2009 6:26:03 AM
> > > Subject: Re: Issue with bulk loader tool
> > >
> > > On Fri, Nov 6, 2009 at 12:58 AM, Murali Krishna. P
> > > <muralikpbhat@...>wrote:
> > >
> > >> Hi,
> > >> If I increase hbase.hregion.max.filesize so that all the records holds
> > in
> > >> one region (and one reducer ), all the records as retrievable. If one
> > >> reducer creates multiple hfile or multiple reducer creates one hfile
> > each,
> > >> the problem occurs.
> > >>
> > >>
> > >
> > > Multiple hfiles in a region?  Or are you saying if a reducer creates
> > > multiple regions?  There is supposed to be one file per region only
> when
> > > done.
> > >
> > > Thanks for digging in,
> > > St.Ack
> > >
> > >
> > >
> > >
> > >> Does that give any clue?
> > >>
> > >> Thanks,
> > >> Murali Krishna
> > >>
> > >>
> > >>
> > >>
> > >> ________________________________
> > >> From: Murali Krishna. P <muralikpbhat@...>
> > >> To: hbase-user@...
> > >> Sent: Thu, 5 November, 2009 6:34:20 PM
> > >> Subject: Re: Issue with bulk loader tool
> > >>
> > >> Hi Stack,
> > >> Sorry, could not look into this last week...
> > >>
> > >> I got problem with the Htable interface as well. Some records i am not
> > >> retrieve from Htable as well.
> > >> I lost the old table, but reproduced the problem with a different
> table.
> > >>
> > >> I cannot send the region since it is very huge. will try to give as
> much
> > >> info as possible here :)
> > >>
> > >> There are total 5 regions as below in that table:
> > >> Name
> > >>
> > >> Encoded Name
> > >> Start Key
> > >> End Key
> > >> test1,,1257414794600
> > >> 106817540
> > >> fffe9c7f87c8332a
> > >> test1,fffe9c7f87c8332a,1257414794616
> > >> 1346846599 fffe9c7f87c8332a fffebe279c0ac4d2
> > >> test1,fffebe279c0ac4d2,1257414794628
> > >> 1835851728 fffebe279c0ac4d2 fffec418284d6fbc
> > >> test1,fffec418284d6fbc,1257414794637
> > >> 1078205908 fffec418284d6fbc fffef7a12ea22498
> > >> test1,fffef7a12ea22498,1257414794647
> > >> 1515378663 fffef7a12ea22498
> > >>
> > >> I am looking for a key, say 000011d1bc8cd6fe . This should be in the
> > first
> > >> region ?
> > >>
> > >> using hfile tool,
> > >> org.apache.hadoop.hbase.io.hfile.HFile -k -f
> > >> /hbase/test1/106817540/image/3828859735461759684 -v -m -p |  grep
> > >> 000011d1bc8cd6fe
> > >> The first region doesn't have it. Not sure what happened to that
> record.
> > >>
> > >> For a working key, it gives the record properly as below
> > >> K:
> > >>
> >
> \x00\x100003bdd08ca88ee2\x05imagevalue\x7F\xFF\xFF\xFF\xFF\xFF\xFF\xFF\x04
> > >> V: \xFF...
> > >>
> > >> Please let me know if you need more information
> > >>
> > >> Thanks,
> > >> Murali Krishna
> > >>
> > >>
> > >>
> > >>
> > >> ________________________________
> > >> From: stack <stack@...>
> > >> To: hbase-user@...
> > >> Sent: Mon, 2 November, 2009 11:05:43 PM
> > >> Subject: Re: Issue with bulk loader tool
> > >>
> > >> Murali:
> > >>
> > >> Any developments worth mentioning?
> > >>
> > >> St.Ack
> > >>
> > >>
> > >> On Fri, Oct 30, 2009 at 10:14 AM, stack <stack@...> wrote:
> > >>
> > >>> That is interesting.  It'd almost point to a shell issue.  Enable
> DEBUG
> > >> so
> > >>> client can see it.  Then rerun shell.  Is it at least loading the
> right
> > >>> region?  (The regions start and end keys span the asked for key?).  I
> > >> took a
> > >>> look at your attached .META. scan.  All looks good there.  The region
> > >>> specifications look right.  If you want to bundle up the region that
> is
> > >>> failing -- the one that the failing key comes out of, I can take a
> look
> > >>> here.  You could also try playing with the HFile tool: ./bin/hbase
> > >>> org.apache.hadoop.hbase.io.hfile.HFile.  Run the former and it'll
> > output
> > >>> usage.  You should be able to get it to dump content of the region
> (You
> > >> need
> > >>> to supply flags like -v to see actual keys to the HFile tool else it
> > just
> > >>> runs its check silently).    Check for your key.  Check things like
> > >>> timestamp on it.  Maybe its 100 years in advance of now or something?
> > >>>
> > >>> Yours,
> > >>> St.Ack
> > >>>
> > >>>
> > >>> On Fri, Oct 30, 2009 at 9:01 AM, Murali Krishna. P <
> > >> muralikpbhat@...
> > >>>> wrote:
> > >>>
> > >>>> Attached ".META"
> > >>>>
> > >>>> Interesting, I was able to get the row from HTable via java code.
> But
> > >> from
> > >>>> the shell, still getting following
> > >>>>
> > >>>> hbase(main):004:0> get 'TestTable2', 'ffffef95bcbf2638'
> > >>>> 0 row(s) in 1.2250 seconds
> > >>>>
> > >>>> Thanks,
> > >>>> Murali Krishna
> > >>>>
> > >>>> Thanks,
> > >>>> Murali Krishna
> > >>>>
> > >>>>
> > >>>> ------------------------------
> > >>>> *From:* stack <stack@...>
> > >>>> *To:* hbase-user@...
> > >>>> *Sent:* Fri, 30 October, 2009 8:39:46 PM
> > >>>> *Subject:* Re: Issue with bulk loader tool
> > >>>>
> > >>>> Can you send a listing of ".META."?
> > >>>>
> > >>>> hbase> scan ".META."
> > >>>>
> > >>>> Also, can you bring a region down from hdfs, tar and gzip it, and
> then
> > >> put
> > >>>> it someplace I can pull so I can take a look?
> > >>>>
> > >>>> Thanks,
> > >>>> St.Ack
> > >>>>
> > >>>>
> > >>>> On Fri, Oct 30, 2009 at 3:31 AM, Murali Krishna. P
> > >>>> <muralikpbhat@...>wrote:
> > >>>>
> > >>>>> Hi guys,
> > >>>>> I created a table according to hbase-48. A mapreduce job which
> > >> creates
> > >>>>> HFiles and then used loadtable.rb script to create the table.
> > >> Everything
> > >>>>> worked fine and i was able to scan the table. But when i do a get
> for
> > >> a
> > >>>> key
> > >>>>> displayed in the scan output, it is not retrieving the row. shell
> > says
> > >> 0
> > >>>>> row.
> > >>>>>
> > >>>>> I tried using one reducer to ensure total ordering, but still same
> > >>>> issue.
> > >>>>>
> > >>>>>
> > >>>>> My mapper is like:
> > >>>>> context.write(new
> > >>>>> ImmutableBytesWritable(((Text)key).toString().getBytes()), new
> > >>>>> KeyValue(((Text)key).toString().getBytes(), "family1".getBytes(),
> > >>>>>                  "column1".getBytes(), getValueBytes()));
> > >>>>>
> > >>>>>
> > >>>>> Please help me investigate this.
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Murali Krishna
> > >>>>>
> > >>>>
> > >>>
> > >>>
> > >>
> >
>

Re: Issue with bulk loader tool

by Murali Krishna. P :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

no, we are not dropping it. It is going to the previous region's last entry. So,the  last key in inclusive but firstkey is exclusive.

look at my test code:
                        HFile.Reader reader = new HFile.Reader(fs, new Path(args[0]), null, true);
                        reader.loadFileInfo();
                        System.out.println("FirstKey:" + new String(reader.getFirstKey()));
                        System.out.println("LastKey:" + new String(reader.getLastKey()));
                        HFileScanner l = reader.getScanner();
                        l.seekTo(reader.getLastKey());
                        KeyValue t = l.getKeyValue();
                        System.out.println("last key:" + t.getKeyString() + " last value length:" + t.getValueLength() + " value:" + t.getValue());
and output is:

FirstKey:00000d7d4f36c112imagevalue�������
LastKey:333305184e0f7c3eimagevalue�������
last key:\x00\x10333305184e0f7c3e\x05imagevalue\x7F\xFF\xFF\xFF\xFF\xFF\xFF\xFF\x04 last value length:3398 value:[B@8888e6c

 Thanks,
Murali Krishna




________________________________
From: stack <stack@...>
To: hbase-user@...
Sent: Sun, 8 November, 2009 12:21:48 PM
Subject: Re: Issue with bulk loader tool

So, do you think we are dropping the first key in the region?
Thanks,
St.Ack

On Sat, Nov 7, 2009 at 9:17 PM, Murali Krishna. P <muralikpbhat@...>wrote:

> No, the first key is 6666909d611e8d7e for the region which says startKey is
> 666629fe4378c096.
> (this is actually the next key in the order).
>
> HFile -p:-
> Scanning -> /hbase/test12/336573097/image/2362265315474952099
> K: \x00\x106666909d611e8d7e\x05imagevalue\x7F\x..
>
>  HFileUtil /hbase/test12/336573097/image/2362265315474952099 :-
> FirstKey:6666909d611e8d7eimagevalue�������
> LastKey:99998c8f356b0d86imagevalue�������
>
> But the scan .META. shows the start key as 666629fe4378c096. (attached
> .META.)
>
> This seems to be the case for all the regions. (the actual firstKey is next
> one from claimed firstKey)
>
> I am on hadoop0.20.0
>
> Thanks,
> Murali Krishna
>
>
> ------------------------------
> *From:* stack <stack@...>
> *To:* hbase-user@...
> *Sent:* Sun, 8 November, 2009 4:30:15 AM
>
> *Subject:* Re: Issue with bulk loader tool
>
> Its what Lars says Murali, a region's startkey is inclusive and its endkey
> exclusive.  If it exists, it should be in the region has it for a start key
> (It will not be duplicated in both).
>
> For .META., there is usually only one Region instance in a .META. table.
> Its startkey will be the empty key so its not suprirising its first key is
> different from the empty key.  What do you see when you look at the second
> region in your just uploaded table?  I'd expect the key 666629fe4378c096 to
> be first in the region whose startkey is 666629fe4378c096.
>
> Thanks for figuring MAPREDUCE-565 could trip us up.  Your hadoop is not
> 0.20.1?
>
> Yours,
> St.Ack
>
>
>
> On Sat, Nov 7, 2009 at 7:58 AM, Murali Krishna. P <muralikpbhat@...
> >wrote:
>
> > Thanks Lars for the clarification,
> >    But where does the record recide ? Is it duplicated to both the
> regions
> > ?? When I use HFile.Reader, the first key in the second region is
> different.
> > May be this behaviour(overlap) is only in .META. ?
> >    The issue is that when I request for that boundary record, it is
> loging
> > the next region.
> >
> > 09/11/07 07:52:05 DEBUG client.HConnectionManager$TableServers: Cached
> > location address: 76.13.20.58:60020, regioninfo: REGION => {NAME =>
> > '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE
> =>
> > {{NAME => '.META.', IS_META => 'true', MEMSTORE_FLUSHSIZE => '16384',
> > FAMILIES => [{NAME => 'historian', VERSIONS => '2147483647', COMPRESSION
> =>
> > 'NONE', TTL => '604800', BLOCKSIZE => '8192', IN_MEMORY => 'false',
> > BLOCKCACHE => 'false'}, {NAME => 'info', VERSIONS => '10', COMPRESSION =>
> > 'NONE', TTL => '2147483647', BLOCKSIZE => '8192', IN_MEMORY => 'false',
> > BLOCKCACHE => 'false'}]}}
> > 09/11/07 07:52:05 DEBUG client.HConnectionManager$TableServers: Cached
> > location address: 76.13.20.114:60020, regioninfo: REGION => {NAME =>
> > 'test12,333305184e0f7c3e,1257515988652', STARTKEY => '333305184e0f7c3e',
> > ENDKEY => '666629fe4378c096', ENCODED => 170637321, TABLE => {{NAME =>
> > 'test12', FAMILIES => [{NAME => 'image', VERSIONS => '3', COMPRESSION =>
> > 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false',
> > BLOCKCACHE => 'true'}]}}
> >
> >  Thanks,
> > Murali Krishna
> >
> >
> >
> >
> > ________________________________
> > From: Lars George <lars@...>
> > To: "hbase-user@..." <hbase-user@...>
> > Sent: Sat, 7 November, 2009 9:19:37 PM
> > Subject: Re: Issue with bulk loader tool
> >
> > Hi Murali,
> >
> > What you see is normal the last keys do indeed overlap. The last key of a
> > region is exclusive and marks the first key of the subsequent region.
> >
> > Lars
> >
> > On Nov 7, 2009, at 9:05, "Murali Krishna. P" <muralikpbhat@...>
> > wrote:
> >
> > > Hi,
> > > I got it resolved. https://issues.apache.org/jira/browse/HADOOP-5750was
> > causing this, even though I supplied a custom total ordering partitioner,
> it
> > didnt use that.
> > >
> > >
> > >  Now the regions looks properly sorted, but facing a new issue. The
> last
> > key of the each region is not retrievable. The table.jsp  page shows the
> > start and end key wrongly.
> > > for eg, take first 2 regions
> > > region1: start : end: 333305184e0f7c3e
> > > region2: start: 333305184e0f7c3e end: 666629fe4378c096
> > >
> > > The end key of first region = start key of second ??
> > >
> > > If I get the first and last key using HFile.Reader, it shows as
> follows:
> > >
> > > HFileUtil /hbase/test12/98766318/image/9052388247118781160
> > > FirstKey:00000d7d4f36c112imagevalue�������
> > > LastKey:333305184e0f7c3eimagevalue�������
> > >
> > > HFileUtil /hbase/test12/170637321/image/7602871928600243730
> > > FirstKey:33338d45cc2491b8imagevalue�������
> > > LastKey:666629fe4378c096imagevalue�������
> > >
> > > So, according to this first key of 2nd region is 33338d45cc2491b8 not
> > 333305184e0f7c3e which is correct!
> > >
> > > Now when I do a get on 333305184e0f7c3e with debug on, it is loading
> the
> > second region which is wrong!
> > >
> > > Some thing went wrong with the index?
> > >
> > > Thanks,
> > > Murali Krishna
> > >
> > >
> > >
> > >
> > > ________________________________
> > > From: stack <stack@...>
> > > To: hbase-user@...
> > > Sent: Sat, 7 November, 2009 6:26:03 AM
> > > Subject: Re: Issue with bulk loader tool
> > >
> > > On Fri, Nov 6, 2009 at 12:58 AM, Murali Krishna. P
> > > <muralikpbhat@...>wrote:
> > >
> > >> Hi,
> > >> If I increase hbase.hregion.max.filesize so that all the records holds
> > in
> > >> one region (and one reducer ), all the records as retrievable. If one
> > >> reducer creates multiple hfile or multiple reducer creates one hfile
> > each,
> > >> the problem occurs.
> > >>
> > >>
> > >
> > > Multiple hfiles in a region?  Or are you saying if a reducer creates
> > > multiple regions?  There is supposed to be one file per region only
> when
> > > done.
> > >
> > > Thanks for digging in,
> > > St.Ack
> > >
> > >
> > >
> > >
> > >> Does that give any clue?
> > >>
> > >> Thanks,
> > >> Murali Krishna
> > >>
> > >>
> > >>
> > >>
> > >> ________________________________
> > >> From: Murali Krishna. P <muralikpbhat@...>
> > >> To: hbase-user@...
> > >> Sent: Thu, 5 November, 2009 6:34:20 PM
> > >> Subject: Re: Issue with bulk loader tool
> > >>
> > >> Hi Stack,
> > >> Sorry, could not look into this last week...
> > >>
> > >> I got problem with the Htable interface as well. Some records i am not
> > >> retrieve from Htable as well.
> > >> I lost the old table, but reproduced the problem with a different
> table.
> > >>
> > >> I cannot send the region since it is very huge. will try to give as
> much
> > >> info as possible here :)
> > >>
> > >> There are total 5 regions as below in that table:
> > >> Name
> > >>
> > >> Encoded Name
> > >> Start Key
> > >> End Key
> > >> test1,,1257414794600
> > >> 106817540
> > >> fffe9c7f87c8332a
> > >> test1,fffe9c7f87c8332a,1257414794616
> > >> 1346846599 fffe9c7f87c8332a fffebe279c0ac4d2
> > >> test1,fffebe279c0ac4d2,1257414794628
> > >> 1835851728 fffebe279c0ac4d2 fffec418284d6fbc
> > >> test1,fffec418284d6fbc,1257414794637
> > >> 1078205908 fffec418284d6fbc fffef7a12ea22498
> > >> test1,fffef7a12ea22498,1257414794647
> > >> 1515378663 fffef7a12ea22498
> > >>
> > >> I am looking for a key, say 000011d1bc8cd6fe . This should be in the
> > first
> > >> region ?
> > >>
> > >> using hfile tool,
> > >> org.apache.hadoop.hbase.io.hfile.HFile -k -f
> > >> /hbase/test1/106817540/image/3828859735461759684 -v -m -p |  grep
> > >> 000011d1bc8cd6fe
> > >> The first region doesn't have it. Not sure what happened to that
> record.
> > >>
> > >> For a working key, it gives the record properly as below
> > >> K:
> > >>
> >
> \x00\x100003bdd08ca88ee2\x05imagevalue\x7F\xFF\xFF\xFF\xFF\xFF\xFF\xFF\x04
> > >> V: \xFF...
> > >>
> > >> Please let me know if you need more information
> > >>
> > >> Thanks,
> > >> Murali Krishna
> > >>
> > >>
> > >>
> > >>
> > >> ________________________________
> > >> From: stack <stack@...>
> > >> To: hbase-user@...
> > >> Sent: Mon, 2 November, 2009 11:05:43 PM
> > >> Subject: Re: Issue with bulk loader tool
> > >>
> > >> Murali:
> > >>
> > >> Any developments worth mentioning?
> > >>
> > >> St.Ack
> > >>
> > >>
> > >> On Fri, Oct 30, 2009 at 10:14 AM, stack <stack@...> wrote:
> > >>
> > >>> That is interesting.  It'd almost point to a shell issue.  Enable
> DEBUG
> > >> so
> > >>> client can see it.  Then rerun shell.  Is it at least loading the
> right
> > >>> region?  (The regions start and end keys span the asked for key?).  I
> > >> took a
> > >>> look at your attached .META. scan.  All looks good there.  The region
> > >>> specifications look right.  If you want to bundle up the region that
> is
> > >>> failing -- the one that the failing key comes out of, I can take a
> look
> > >>> here.  You could also try playing with the HFile tool: ./bin/hbase
> > >>> org.apache.hadoop.hbase.io.hfile.HFile.  Run the former and it'll
> > output
> > >>> usage.  You should be able to get it to dump content of the region
> (You
> > >> need
> > >>> to supply flags like -v to see actual keys to the HFile tool else it
> > just
> > >>> runs its check silently).    Check for your key.  Check things like
> > >>> timestamp on it.  Maybe its 100 years in advance of now or something?
> > >>>
> > >>> Yours,
> > >>> St.Ack
> > >>>
> > >>>
> > >>> On Fri, Oct 30, 2009 at 9:01 AM, Murali Krishna. P <
> > >> muralikpbhat@...
> > >>>> wrote:
> > >>>
> > >>>> Attached ".META"
> > >>>>
> > >>>> Interesting, I was able to get the row from HTable via java code.
> But
> > >> from
> > >>>> the shell, still getting following
> > >>>>
> > >>>> hbase(main):004:0> get 'TestTable2', 'ffffef95bcbf2638'
> > >>>> 0 row(s) in 1.2250 seconds
> > >>>>
> > >>>> Thanks,
> > >>>> Murali Krishna
> > >>>>
> > >>>> Thanks,
> > >>>> Murali Krishna
> > >>>>
> > >>>>
> > >>>> ------------------------------
> > >>>> *From:* stack <stack@...>
> > >>>> *To:* hbase-user@...
> > >>>> *Sent:* Fri, 30 October, 2009 8:39:46 PM
> > >>>> *Subject:* Re: Issue with bulk loader tool
> > >>>>
> > >>>> Can you send a listing of ".META."?
> > >>>>
> > >>>> hbase> scan ".META."
> > >>>>
> > >>>> Also, can you bring a region down from hdfs, tar and gzip it, and
> then
> > >> put
> > >>>> it someplace I can pull so I can take a look?
> > >>>>
> > >>>> Thanks,
> > >>>> St.Ack
> > >>>>
> > >>>>
> > >>>> On Fri, Oct 30, 2009 at 3:31 AM, Murali Krishna. P
> > >>>> <muralikpbhat@...>wrote:
> > >>>>
> > >>>>> Hi guys,
> > >>>>> I created a table according to hbase-48. A mapreduce job which
> > >> creates
> > >>>>> HFiles and then used loadtable.rb script to create the table.
> > >> Everything
> > >>>>> worked fine and i was able to scan the table. But when i do a get
> for
> > >> a
> > >>>> key
> > >>>>> displayed in the scan output, it is not retrieving the row. shell
> > says
> > >> 0
> > >>>>> row.
> > >>>>>
> > >>>>> I tried using one reducer to ensure total ordering, but still same
> > >>>> issue.
> > >>>>>
> > >>>>>
> > >>>>> My mapper is like:
> > >>>>> context.write(new
> > >>>>> ImmutableBytesWritable(((Text)key).toString().getBytes()), new
> > >>>>> KeyValue(((Text)key).toString().getBytes(), "family1".getBytes(),
> > >>>>>                  "column1".getBytes(), getValueBytes()));
> > >>>>>
> > >>>>>
> > >>>>> Please help me investigate this.
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Murali Krishna
> > >>>>>
> > >>>>
> > >>>
> > >>>
> > >>
> >
>

Re: Issue with bulk loader tool

by stack-3 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Ok. Thanks.  Looks like a dumb bug in the HFileOutputFormat.  I'll check
tomorrow.  Thanks for your patience.
St.Ack

On Sat, Nov 7, 2009 at 11:06 PM, Murali Krishna. P
<muralikpbhat@...>wrote:

> no, we are not dropping it. It is going to the previous region's last
> entry. So,the  last key in inclusive but firstkey is exclusive.
>
> look at my test code:
>                        HFile.Reader reader = new HFile.Reader(fs, new
> Path(args[0]), null, true);
>                        reader.loadFileInfo();
>                        System.out.println("FirstKey:" + new
> String(reader.getFirstKey()));
>                        System.out.println("LastKey:" + new
> String(reader.getLastKey()));
>                        HFileScanner l = reader.getScanner();
>                        l.seekTo(reader.getLastKey());
>                        KeyValue t = l.getKeyValue();
>                        System.out.println("last key:" + t.getKeyString() +
> " last value length:" + t.getValueLength() + " value:" + t.getValue());
> and output is:
>
> FirstKey:00000d7d4f36c112imagevalue�������
> LastKey:333305184e0f7c3eimagevalue�������
> last
> key:\x00\x10333305184e0f7c3e\x05imagevalue\x7F\xFF\xFF\xFF\xFF\xFF\xFF\xFF\x04
> last value length:3398 value:[B@8888e6c
>
>  Thanks,
> Murali Krishna
>
>
>
>
> ________________________________
> From: stack <stack@...>
> To: hbase-user@...
> Sent: Sun, 8 November, 2009 12:21:48 PM
> Subject: Re: Issue with bulk loader tool
>
> So, do you think we are dropping the first key in the region?
> Thanks,
> St.Ack
>
> On Sat, Nov 7, 2009 at 9:17 PM, Murali Krishna. P <muralikpbhat@...
> >wrote:
>
> > No, the first key is 6666909d611e8d7e for the region which says startKey
> is
> > 666629fe4378c096.
> > (this is actually the next key in the order).
> >
> > HFile -p:-
> > Scanning -> /hbase/test12/336573097/image/2362265315474952099
> > K: \x00\x106666909d611e8d7e\x05imagevalue\x7F\x..
> >
> >  HFileUtil /hbase/test12/336573097/image/2362265315474952099 :-
> > FirstKey:6666909d611e8d7eimagevalue�������
> > LastKey:99998c8f356b0d86imagevalue�������
> >
> > But the scan .META. shows the start key as 666629fe4378c096. (attached
> > .META.)
> >
> > This seems to be the case for all the regions. (the actual firstKey is
> next
> > one from claimed firstKey)
> >
> > I am on hadoop0.20.0
> >
> > Thanks,
> > Murali Krishna
> >
> >
> > ------------------------------
> > *From:* stack <stack@...>
> > *To:* hbase-user@...
> > *Sent:* Sun, 8 November, 2009 4:30:15 AM
> >
> > *Subject:* Re: Issue with bulk loader tool
> >
> > Its what Lars says Murali, a region's startkey is inclusive and its
> endkey
> > exclusive.  If it exists, it should be in the region has it for a start
> key
> > (It will not be duplicated in both).
> >
> > For .META., there is usually only one Region instance in a .META. table.
> > Its startkey will be the empty key so its not suprirising its first key
> is
> > different from the empty key.  What do you see when you look at the
> second
> > region in your just uploaded table?  I'd expect the key 666629fe4378c096
> to
> > be first in the region whose startkey is 666629fe4378c096.
> >
> > Thanks for figuring MAPREDUCE-565 could trip us up.  Your hadoop is not
> > 0.20.1?
> >
> > Yours,
> > St.Ack
> >
> >
> >
> > On Sat, Nov 7, 2009 at 7:58 AM, Murali Krishna. P <
> muralikpbhat@...
> > >wrote:
> >
> > > Thanks Lars for the clarification,
> > >    But where does the record recide ? Is it duplicated to both the
> > regions
> > > ?? When I use HFile.Reader, the first key in the second region is
> > different.
> > > May be this behaviour(overlap) is only in .META. ?
> > >    The issue is that when I request for that boundary record, it is
> > loging
> > > the next region.
> > >
> > > 09/11/07 07:52:05 DEBUG client.HConnectionManager$TableServers: Cached
> > > location address: 76.13.20.58:60020, regioninfo: REGION => {NAME =>
> > > '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE
> > =>
> > > {{NAME => '.META.', IS_META => 'true', MEMSTORE_FLUSHSIZE => '16384',
> > > FAMILIES => [{NAME => 'historian', VERSIONS => '2147483647',
> COMPRESSION
> > =>
> > > 'NONE', TTL => '604800', BLOCKSIZE => '8192', IN_MEMORY => 'false',
> > > BLOCKCACHE => 'false'}, {NAME => 'info', VERSIONS => '10', COMPRESSION
> =>
> > > 'NONE', TTL => '2147483647', BLOCKSIZE => '8192', IN_MEMORY => 'false',
> > > BLOCKCACHE => 'false'}]}}
> > > 09/11/07 07:52:05 DEBUG client.HConnectionManager$TableServers: Cached
> > > location address: 76.13.20.114:60020, regioninfo: REGION => {NAME =>
> > > 'test12,333305184e0f7c3e,1257515988652', STARTKEY =>
> '333305184e0f7c3e',
> > > ENDKEY => '666629fe4378c096', ENCODED => 170637321, TABLE => {{NAME =>
> > > 'test12', FAMILIES => [{NAME => 'image', VERSIONS => '3', COMPRESSION
> =>
> > > 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY =>
> 'false',
> > > BLOCKCACHE => 'true'}]}}
> > >
> > >  Thanks,
> > > Murali Krishna
> > >
> > >
> > >
> > >
> > > ________________________________
> > > From: Lars George <lars@...>
> > > To: "hbase-user@..." <hbase-user@...>
> > > Sent: Sat, 7 November, 2009 9:19:37 PM
> > > Subject: Re: Issue with bulk loader tool
> > >
> > > Hi Murali,
> > >
> > > What you see is normal the last keys do indeed overlap. The last key of
> a
> > > region is exclusive and marks the first key of the subsequent region.
> > >
> > > Lars
> > >
> > > On Nov 7, 2009, at 9:05, "Murali Krishna. P" <muralikpbhat@...>
> > > wrote:
> > >
> > > > Hi,
> > > > I got it resolved.
> https://issues.apache.org/jira/browse/HADOOP-5750was
> > > causing this, even though I supplied a custom total ordering
> partitioner,
> > it
> > > didnt use that.
> > > >
> > > >
> > > >  Now the regions looks properly sorted, but facing a new issue. The
> > last
> > > key of the each region is not retrievable. The table.jsp  page shows
> the
> > > start and end key wrongly.
> > > > for eg, take first 2 regions
> > > > region1: start : end: 333305184e0f7c3e
> > > > region2: start: 333305184e0f7c3e end: 666629fe4378c096
> > > >
> > > > The end key of first region = start key of second ??
> > > >
> > > > If I get the first and last key using HFile.Reader, it shows as
> > follows:
> > > >
> > > > HFileUtil /hbase/test12/98766318/image/9052388247118781160
> > > > FirstKey:00000d7d4f36c112imagevalue�������
> > > > LastKey:333305184e0f7c3eimagevalue�������
> > > >
> > > > HFileUtil /hbase/test12/170637321/image/7602871928600243730
> > > > FirstKey:33338d45cc2491b8imagevalue�������
> > > > LastKey:666629fe4378c096imagevalue�������
> > > >
> > > > So, according to this first key of 2nd region is 33338d45cc2491b8 not
> > > 333305184e0f7c3e which is correct!
> > > >
> > > > Now when I do a get on 333305184e0f7c3e with debug on, it is loading
> > the
> > > second region which is wrong!
> > > >
> > > > Some thing went wrong with the index?
> > > >
> > > > Thanks,
> > > > Murali Krishna
> > > >
> > > >
> > > >
> > > >
> > > > ________________________________
> > > > From: stack <stack@...>
> > > > To: hbase-user@...
> > > > Sent: Sat, 7 November, 2009 6:26:03 AM
> > > > Subject: Re: Issue with bulk loader tool
> > > >
> > > > On Fri, Nov 6, 2009 at 12:58 AM, Murali Krishna. P
> > > > <muralikpbhat@...>wrote:
> > > >
> > > >> Hi,
> > > >> If I increase hbase.hregion.max.filesize so that all the records
> holds
> > > in
> > > >> one region (and one reducer ), all the records as retrievable. If
> one
> > > >> reducer creates multiple hfile or multiple reducer creates one hfile
> > > each,
> > > >> the problem occurs.
> > > >>
> > > >>
> > > >
> > > > Multiple hfiles in a region?  Or are you saying if a reducer creates
> > > > multiple regions?  There is supposed to be one file per region only
> > when
> > > > done.
> > > >
> > > > Thanks for digging in,
> > > > St.Ack
> > > >
> > > >
> > > >
> > > >
> > > >> Does that give any clue?
> > > >>
> > > >> Thanks,
> > > >> Murali Krishna
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> ________________________________
> > > >> From: Murali Krishna. P <muralikpbhat@...>
> > > >> To: hbase-user@...
> > > >> Sent: Thu, 5 November, 2009 6:34:20 PM
> > > >> Subject: Re: Issue with bulk loader tool
> > > >>
> > > >> Hi Stack,
> > > >> Sorry, could not look into this last week...
> > > >>
> > > >> I got problem with the Htable interface as well. Some records i am
> not
> > > >> retrieve from Htable as well.
> > > >> I lost the old table, but reproduced the problem with a different
> > table.
> > > >>
> > > >> I cannot send the region since it is very huge. will try to give as
> > much
> > > >> info as possible here :)
> > > >>
> > > >> There are total 5 regions as below in that table:
> > > >> Name
> > > >>
> > > >> Encoded Name
> > > >> Start Key
> > > >> End Key
> > > >> test1,,1257414794600
> > > >> 106817540
> > > >> fffe9c7f87c8332a
> > > >> test1,fffe9c7f87c8332a,1257414794616
> > > >> 1346846599 fffe9c7f87c8332a fffebe279c0ac4d2
> > > >> test1,fffebe279c0ac4d2,1257414794628
> > > >> 1835851728 fffebe279c0ac4d2 fffec418284d6fbc
> > > >> test1,fffec418284d6fbc,1257414794637
> > > >> 1078205908 fffec418284d6fbc fffef7a12ea22498
> > > >> test1,fffef7a12ea22498,1257414794647
> > > >> 1515378663 fffef7a12ea22498
> > > >>
> > > >> I am looking for a key, say 000011d1bc8cd6fe . This should be in the
> > > first
> > > >> region ?
> > > >>
> > > >> using hfile tool,
> > > >> org.apache.hadoop.hbase.io.hfile.HFile -k -f
> > > >> /hbase/test1/106817540/image/3828859735461759684 -v -m -p |  grep
> > > >> 000011d1bc8cd6fe
> > > >> The first region doesn't have it. Not sure what happened to that
> > record.
> > > >>
> > > >> For a working key, it gives the record properly as below
> > > >> K:
> > > >>
> > >
> >
> \x00\x100003bdd08ca88ee2\x05imagevalue\x7F\xFF\xFF\xFF\xFF\xFF\xFF\xFF\x04
> > > >> V: \xFF...
> > > >>
> > > >> Please let me know if you need more information
> > > >>
> > > >> Thanks,
> > > >> Murali Krishna
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> ________________________________
> > > >> From: stack <stack@...>
> > > >> To: hbase-user@...
> > > >> Sent: Mon, 2 November, 2009 11:05:43 PM
> > > >> Subject: Re: Issue with bulk loader tool
> > > >>
> > > >> Murali:
> > > >>
> > > >> Any developments worth mentioning?
> > > >>
> > > >> St.Ack
> > > >>
> > > >>
> > > >> On Fri, Oct 30, 2009 at 10:14 AM, stack <stack@...> wrote:
> > > >>
> > > >>> That is interesting.  It'd almost point to a shell issue.  Enable
> > DEBUG
> > > >> so
> > > >>> client can see it.  Then rerun shell.  Is it at least loading the
> > right
> > > >>> region?  (The regions start and end keys span the asked for key?).
>  I
> > > >> took a
> > > >>> look at your attached .META. scan.  All looks good there.  The
> region
> > > >>> specifications look right.  If you want to bundle up the region
> that
> > is
> > > >>> failing -- the one that the failing key comes out of, I can take a
> > look
> > > >>> here.  You could also try playing with the HFile tool: ./bin/hbase
> > > >>> org.apache.hadoop.hbase.io.hfile.HFile.  Run the former and it'll
> > > output
> > > >>> usage.  You should be able to get it to dump content of the region
> > (You
> > > >> need
> > > >>> to supply flags like -v to see actual keys to the HFile tool else
> it
> > > just
> > > >>> runs its check silently).    Check for your key.  Check things like
> > > >>> timestamp on it.  Maybe its 100 years in advance of now or
> something?
> > > >>>
> > > >>> Yours,
> > > >>> St.Ack
> > > >>>
> > > >>>
> > > >>> On Fri, Oct 30, 2009 at 9:01 AM, Murali Krishna. P <
> > > >> muralikpbhat@...
> > > >>>> wrote:
> > > >>>
> > > >>>> Attached ".META"
> > > >>>>
> > > >>>> Interesting, I was able to get the row from HTable via java code.
> > But
> > > >> from
> > > >>>> the shell, still getting following
> > > >>>>
> > > >>>> hbase(main):004:0> get 'TestTable2', 'ffffef95bcbf2638'
> > > >>>> 0 row(s) in 1.2250 seconds
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Murali Krishna
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Murali Krishna
> > > >>>>
> > > >>>>
> > > >>>> ------------------------------
> > > >>>> *From:* stack <stack@...>
> > > >>>> *To:* hbase-user@...
> > > >>>> *Sent:* Fri, 30 October, 2009 8:39:46 PM
> > > >>>> *Subject:* Re: Issue with bulk loader tool
> > > >>>>
> > > >>>> Can you send a listing of ".META."?
> > > >>>>
> > > >>>> hbase> scan ".META."
> > > >>>>
> > > >>>> Also, can you bring a region down from hdfs, tar and gzip it, and
> > then
> > > >> put
> > > >>>> it someplace I can pull so I can take a look?
> > > >>>>
> > > >>>> Thanks,
> > > >>>> St.Ack
> > > >>>>
> > > >>>>
> > > >>>> On Fri, Oct 30, 2009 at 3:31 AM, Murali Krishna. P
> > > >>>> <muralikpbhat@...>wrote:
> > > >>>>
> > > >>>>> Hi guys,
> > > >>>>> I created a table according to hbase-48. A mapreduce job which
> > > >> creates
> > > >>>>> HFiles and then used loadtable.rb script to create the table.
> > > >> Everything
> > > >>>>> worked fine and i was able to scan the table. But when i do a get
> > for
> > > >> a
> > > >>>> key
> > > >>>>> displayed in the scan output, it is not retrieving the row. shell
> > > says
> > > >> 0
> > > >>>>> row.
> > > >>>>>
> > > >>>>> I tried using one reducer to ensure total ordering, but still
> same
> > > >>>> issue.
> > > >>>>>
> > > >>>>>
> > > >>>>> My mapper is like:
> > > >>>>> context.write(new
> > > >>>>> ImmutableBytesWritable(((Text)key).toString().getBytes()), new
> > > >>>>> KeyValue(((Text)key).toString().getBytes(), "family1".getBytes(),
> > > >>>>>                  "column1".getBytes(), getValueBytes()));
> > > >>>>>
> > > >>>>>
> > > >>>>> Please help me investigate this.
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>> Murali Krishna
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >>
> > >
> >
>

Re: Issue with bulk loader tool

by Murali Krishna. P :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Thanks Stack,
  My guess is that loadtable script assigning the first key wrongly, not an issue with HFileOuputFormat. May the first key should be 33338d45cc2491b8 not 333305184e0f7c3e
 Anyway, will wait for your analysis tomorrow

Thanks for the support,
Murali Krishna




________________________________
From: stack <stack@...>
To: hbase-user@...
Sent: Sun, 8 November, 2009 12:47:25 PM
Subject: Re: Issue with bulk loader tool

Ok. Thanks.  Looks like a dumb bug in the HFileOutputFormat.  I'll check
tomorrow.  Thanks for your patience.
St.Ack

On Sat, Nov 7, 2009 at 11:06 PM, Murali Krishna. P
<muralikpbhat@...>wrote:

> no, we are not dropping it. It is going to the previous region's last
> entry. So,the  last key in inclusive but firstkey is exclusive.
>
> look at my test code:
>                        HFile.Reader reader = new HFile.Reader(fs, new
> Path(args[0]), null, true);
>                        reader.loadFileInfo();
>                        System.out.println("FirstKey:" + new
> String(reader.getFirstKey()));
>                        System.out.println("LastKey:" + new
> String(reader.getLastKey()));
>                        HFileScanner l = reader.getScanner();
>                        l.seekTo(reader.getLastKey());
>                        KeyValue t = l.getKeyValue();
>                        System.out.println("last key:" + t.getKeyString() +
> " last value length:" + t.getValueLength() + " value:" + t.getValue());
> and output is:
>
> FirstKey:00000d7d4f36c112imagevalue�������
> LastKey:333305184e0f7c3eimagevalue�������
> last
> key:\x00\x10333305184e0f7c3e\x05imagevalue\x7F\xFF\xFF\xFF\xFF\xFF\xFF\xFF\x04
> last value length:3398 value:[B@8888e6c
>
>  Thanks,
> Murali Krishna
>
>
>
>
> ________________________________
> From: stack <stack@...>
> To: hbase-user@...
> Sent: Sun, 8 November, 2009 12:21:48 PM
> Subject: Re: Issue with bulk loader tool
>
> So, do you think we are dropping the first key in the region?
> Thanks,
> St.Ack
>
> On Sat, Nov 7, 2009 at 9:17 PM, Murali Krishna. P <muralikpbhat@...
> >wrote:
>
> > No, the first key is 6666909d611e8d7e for the region which says startKey
> is
> > 666629fe4378c096.
> > (this is actually the next key in the order).
> >
> > HFile -p:-
> > Scanning -> /hbase/test12/336573097/image/2362265315474952099
> > K: \x00\x106666909d611e8d7e\x05imagevalue\x7F\x..
> >
> >  HFileUtil /hbase/test12/336573097/image/2362265315474952099 :-
> > FirstKey:6666909d611e8d7eimagevalue�������
> > LastKey:99998c8f356b0d86imagevalue�������
> >
> > But the scan .META. shows the start key as 666629fe4378c096. (attached
> > .META.)
> >
> > This seems to be the case for all the regions. (the actual firstKey is
> next
> > one from claimed firstKey)
> >
> > I am on hadoop0.20.0
> >
> > Thanks,
> > Murali Krishna
> >
> >
> > ------------------------------
> > *From:* stack <stack@...>
> > *To:* hbase-user@...
> > *Sent:* Sun, 8 November, 2009 4:30:15 AM
> >
> > *Subject:* Re: Issue with bulk loader tool
> >
> > Its what Lars says Murali, a region's startkey is inclusive and its
> endkey
> > exclusive.  If it exists, it should be in the region has it for a start
> key
> > (It will not be duplicated in both).
> >
> > For .META., there is usually only one Region instance in a .META. table.
> > Its startkey will be the empty key so its not suprirising its first key
> is
> > different from the empty key.  What do you see when you look at the
> second
> > region in your just uploaded table?  I'd expect the key 666629fe4378c096
> to
> > be first in the region whose startkey is 666629fe4378c096.
> >
> > Thanks for figuring MAPREDUCE-565 could trip us up.  Your hadoop is not
> > 0.20.1?
> >
> > Yours,
> > St.Ack
> >
> >
> >
> > On Sat, Nov 7, 2009 at 7:58 AM, Murali Krishna. P <
> muralikpbhat@...
> > >wrote:
> >
> > > Thanks Lars for the clarification,
> > >    But where does the record recide ? Is it duplicated to both the
> > regions
> > > ?? When I use HFile.Reader, the first key in the second region is
> > different.
> > > May be this behaviour(overlap) is only in .META. ?
> > >    The issue is that when I request for that boundary record, it is
> > loging
> > > the next region.
> > >
> > > 09/11/07 07:52:05 DEBUG client.HConnectionManager$TableServers: Cached
> > > location address: 76.13.20.58:60020, regioninfo: REGION => {NAME =>
> > > '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE
> > =>
> > > {{NAME => '.META.', IS_META => 'true', MEMSTORE_FLUSHSIZE => '16384',
> > > FAMILIES => [{NAME => 'historian', VERSIONS => '2147483647',
> COMPRESSION
> > =>
> > > 'NONE', TTL => '604800', BLOCKSIZE => '8192', IN_MEMORY => 'false',
> > > BLOCKCACHE => 'false'}, {NAME => 'info', VERSIONS => '10', COMPRESSION
> =>
> > > 'NONE', TTL => '2147483647', BLOCKSIZE => '8192', IN_MEMORY => 'false',
> > > BLOCKCACHE => 'false'}]}}
> > > 09/11/07 07:52:05 DEBUG client.HConnectionManager$TableServers: Cached
> > > location address: 76.13.20.114:60020, regioninfo: REGION => {NAME =>
> > > 'test12,333305184e0f7c3e,1257515988652', STARTKEY =>
> '333305184e0f7c3e',
> > > ENDKEY => '666629fe4378c096', ENCODED => 170637321, TABLE => {{NAME =>
> > > 'test12', FAMILIES => [{NAME => 'image', VERSIONS => '3', COMPRESSION
> =>
> > > 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY =>
> 'false',
> > > BLOCKCACHE => 'true'}]}}
> > >
> > >  Thanks,
> > > Murali Krishna
> > >
> > >
> > >
> > >
> > > ________________________________
> > > From: Lars George <lars@...>
> > > To: "hbase-user@..." <hbase-user@...>
> > > Sent: Sat, 7 November, 2009 9:19:37 PM
> > > Subject: Re: Issue with bulk loader tool
> > >
> > > Hi Murali,
> > >
> > > What you see is normal the last keys do indeed overlap. The last key of
> a
> > > region is exclusive and marks the first key of the subsequent region.
> > >
> > > Lars
> > >
> > > On Nov 7, 2009, at 9:05, "Murali Krishna. P" <muralikpbhat@...>
> > > wrote:
> > >
> > > > Hi,
> > > > I got it resolved.
> https://issues.apache.org/jira/browse/HADOOP-5750was
> > > causing this, even though I supplied a custom total ordering
> partitioner,
> > it
> > > didnt use that.
> > > >
> > > >
> > > >  Now the regions looks properly sorted, but facing a new issue. The
> > last
> > > key of the each region is not retrievable. The table.jsp  page shows
> the
> > > start and end key wrongly.
> > > > for eg, take first 2 regions
> > > > region1: start : end: 333305184e0f7c3e
> > > > region2: start: 333305184e0f7c3e end: 666629fe4378c096
> > > >
> > > > The end key of first region = start key of second ??
> > > >
> > > > If I get the first and last key using HFile.Reader, it shows as
> > follows:
> > > >
> > > > HFileUtil /hbase/test12/98766318/image/9052388247118781160
> > > > FirstKey:00000d7d4f36c112imagevalue�������
> > > > LastKey:333305184e0f7c3eimagevalue�������
> > > >
> > > > HFileUtil /hbase/test12/170637321/image/7602871928600243730
> > > > FirstKey:33338d45cc2491b8imagevalue�������
> > > > LastKey:666629fe4378c096imagevalue�������
> > > >
> > > > So, according to this first key of 2nd region is 33338d45cc2491b8 not
> > > 333305184e0f7c3e which is correct!
> > > >
> > > > Now when I do a get on 333305184e0f7c3e with debug on, it is loading
> > the
> > > second region which is wrong!
> > > >
> > > > Some thing went wrong with the index?
> > > >
> > > > Thanks,
> > > > Murali Krishna
> > > >
> > > >
> > > >
> > > >
> > > > ________________________________
> > > > From: stack <stack@...>
> > > > To: hbase-user@...
> > > > Sent: Sat, 7 November, 2009 6:26:03 AM
> > > > Subject: Re: Issue with bulk loader tool
> > > >
> > > > On Fri, Nov 6, 2009 at 12:58 AM, Murali Krishna. P
> > > > <muralikpbhat@...>wrote:
> > > >
> > > >> Hi,
> > > >> If I increase hbase.hregion.max.filesize so that all the records
> holds
> > > in
> > > >> one region (and one reducer ), all the records as retrievable. If
> one
> > > >> reducer creates multiple hfile or multiple reducer creates one hfile
> > > each,
> > > >> the problem occurs.
> > > >>
> > > >>
> > > >
> > > > Multiple hfiles in a region?  Or are you saying if a reducer creates
> > > > multiple regions?  There is supposed to be one file per region only
> > when
> > > > done.
> > > >
> > > > Thanks for digging in,
> > > > St.Ack
> > > >
> > > >
> > > >
> > > >
> > > >> Does that give any clue?
> > > >>
> > > >> Thanks,
> > > >> Murali Krishna
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> ________________________________
> > > >> From: Murali Krishna. P <muralikpbhat@...>
> > > >> To: hbase-user@...
> > > >> Sent: Thu, 5 November, 2009 6:34:20 PM
> > > >> Subject: Re: Issue with bulk loader tool
> > > >>
> > > >> Hi Stack,
> > > >> Sorry, could not look into this last week...
> > > >>
> > > >> I got problem with the Htable interface as well. Some records i am
> not
> > > >> retrieve from Htable as well.
> > > >> I lost the old table, but reproduced the problem with a different
> > table.
> > > >>
> > > >> I cannot send the region since it is very huge. will try to give as
> > much
> > > >> info as possible here :)
> > > >>
> > > >> There are total 5 regions as below in that table:
> > > >> Name
> > > >>
> > > >> Encoded Name
> > > >> Start Key
> > > >> End Key
> > > >> test1,,1257414794600
> > > >> 106817540
> > > >> fffe9c7f87c8332a
> > > >> test1,fffe9c7f87c8332a,1257414794616
> > > >> 1346846599 fffe9c7f87c8332a fffebe279c0ac4d2
> > > >> test1,fffebe279c0ac4d2,1257414794628
> > > >> 1835851728 fffebe279c0ac4d2 fffec418284d6fbc
> > > >> test1,fffec418284d6fbc,1257414794637
> > > >> 1078205908 fffec418284d6fbc fffef7a12ea22498
> > > >> test1,fffef7a12ea22498,1257414794647
> > > >> 1515378663 fffef7a12ea22498
> > > >>
> > > >> I am looking for a key, say 000011d1bc8cd6fe . This should be in the
> > > first
> > > >> region ?
> > > >>
> > > >> using hfile tool,
> > > >> org.apache.hadoop.hbase.io.hfile.HFile -k -f
> > > >> /hbase/test1/106817540/image/3828859735461759684 -v -m -p |  grep
> > > >> 000011d1bc8cd6fe
> > > >> The first region doesn't have it. Not sure what happened to that
> > record.
> > > >>
> > > >> For a working key, it gives the record properly as below
> > > >> K:
> > > >>
> > >
> >
> \x00\x100003bdd08ca88ee2\x05imagevalue\x7F\xFF\xFF\xFF\xFF\xFF\xFF\xFF\x04
> > > >> V: \xFF...
> > > >>
> > > >> Please let me know if you need more information
> > > >>
> > > >> Thanks,
> > > >> Murali Krishna
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> ________________________________
> > > >> From: stack <stack@...>
> > > >> To: hbase-user@...
> > > >> Sent: Mon, 2 November, 2009 11:05:43 PM
> > > >> Subject: Re: Issue with bulk loader tool
> > > >>
> > > >> Murali:
> > > >>
> > > >> Any developments worth mentioning?
> > > >>
> > > >> St.Ack
> > > >>
> > > >>
> > > >> On Fri, Oct 30, 2009 at 10:14 AM, stack <stack@...> wrote:
> > > >>
> > > >>> That is interesting.  It'd almost point to a shell issue.  Enable
> > DEBUG
> > > >> so
> > > >>> client can see it.  Then rerun shell.  Is it at least loading the
> > right
> > > >>> region?  (The regions start and end keys span the asked for key?).
>  I
> > > >> took a
> > > >>> look at your attached .META. scan.  All looks good there.  The
> region
> > > >>> specifications look right.  If you want to bundle up the region
> that
> > is
> > > >>> failing -- the one that the failing key comes out of, I can take a
> > look
> > > >>> here.  You could also try playing with the HFile tool: ./bin/hbase
> > > >>> org.apache.hadoop.hbase.io.hfile.HFile.  Run the former and it'll
> > > output
> > > >>> usage.  You should be able to get it to dump content of the region
> > (You
> > > >> need
> > > >>> to supply flags like -v to see actual keys to the HFile tool else
> it
> > > just
> > > >>> runs its check silently).    Check for your key.  Check things like
> > > >>> timestamp on it.  Maybe its 100 years in advance of now or
> something?
> > > >>>
> > > >>> Yours,
> > > >>> St.Ack
> > > >>>
> > > >>>
> > > >>> On Fri, Oct 30, 2009 at 9:01 AM, Murali Krishna. P <
> > > >> muralikpbhat@...
> > > >>>> wrote:
> > > >>>
> > > >>>> Attached ".META"
> > > >>>>
> > > >>>> Interesting, I was able to get the row from HTable via java code.
> > But
> > > >> from
> > > >>>> the shell, still getting following
> > > >>>>
> > > >>>> hbase(main):004:0> get 'TestTable2', 'ffffef95bcbf2638'
> > > >>>> 0 row(s) in 1.2250 seconds
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Murali Krishna
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Murali Krishna
> > > >>>>
> > > >>>>
> > > >>>> ------------------------------
> > > >>>> *From:* stack <stack@...>
> > > >>>> *To:* hbase-user@...
> > > >>>> *Sent:* Fri, 30 October, 2009 8:39:46 PM
> > > >>>> *Subject:* Re: Issue with bulk loader tool
> > > >>>>
> > > >>>> Can you send a listing of ".META."?
> > > >>>>
> > > >>>> hbase> scan ".META."
> > > >>>>
> > > >>>> Also, can you bring a region down from hdfs, tar and gzip it, and
> > then
> > > >> put
> > > >>>> it someplace I can pull so I can take a look?
> > > >>>>
> > > >>>> Thanks,
> > > >>>> St.Ack
> > > >>>>
> > > >>>>
> > > >>>> On Fri, Oct 30, 2009 at 3:31 AM, Murali Krishna. P
> > > >>>> <muralikpbhat@...>wrote:
> > > >>>>
> > > >>>>> Hi guys,
> > > >>>>> I created a table according to hbase-48. A mapreduce job which
> > > >> creates
> > > >>>>> HFiles and then used loadtable.rb script to create the table.
> > > >> Everything
> > > >>>>> worked fine and i was able to scan the table. But when i do a get
> > for
> > > >> a
> > > >>>> key
> > > >>>>> displayed in the scan output, it is not retrieving the row. shell
> > > says
> > > >> 0
> > > >>>>> row.
> > > >>>>>
> > > >>>>> I tried using one reducer to ensure total ordering, but still
> same
> > > >>>> issue.
> > > >>>>>
> > > >>>>>
> > > >>>>> My mapper is like:
> > > >>>>> context.write(new
> > > >>>>> ImmutableBytesWritable(((Text)key).toString().getBytes()), new
> > > >>>>> KeyValue(((Text)key).toString().getBytes(), "family1".getBytes(),
> > > >>>>>                  "column1".getBytes(), getValueBytes()));
> > > >>>>>
> > > >>>>>
> > > >>>>> Please help me investigate this.
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>> Murali Krishna
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >>
> > >
> >
>

Re: Issue with bulk loader tool

by stack-3 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Sat, Nov 7, 2009 at 11:24 PM, Murali Krishna. P
<muralikpbhat@...>wrote:

> Thanks Stack,
>  My guess is that loadtable script assigning the first key wrongly....


Yes.  You are right.  See https://issues.apache.org/jira/browse/HBASE-1962.
loadtable.rb was making regions with key ranges that spanned lastrows rather
than startrows.  HBASE-1962 has a patch that has not been tested other than
by my messing with ruby.  I'll test better tomorrow but maybe you want to
try it meantime?

Thanks for your patience Murali.
St.Ack

Re: Issue with bulk loader tool

by Murali Krishna. P :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

hi Stack,
  I attached the changes to that patch to the jira. It is working now. Is it possible correct an existing table created via old loadtable? I have a table with few billon records and 40k regions, took almost 2 days for the MR to create this. Don't want to do it again :)

  Is it possible to write a script which iterates though regions of a given table and fixes the border keys in .META.. Will it work?

 Thanks,
Murali Krishna




________________________________
From: stack <stack@...>
To: hbase-user@...
Sent: Mon, 9 November, 2009 6:03:19 AM
Subject: Re: Issue with bulk loader tool

On Sat, Nov 7, 2009 at 11:24 PM, Murali Krishna. P
<muralikpbhat@...>wrote:

> Thanks Stack,
>  My guess is that loadtable script assigning the first key wrongly....


Yes.  You are right.  See https://issues.apache.org/jira/browse/HBASE-1962.
loadtable.rb was making regions with key ranges that spanned lastrows rather
than startrows.  HBASE-1962 has a patch that has not been tested other than
by my messing with ruby.  I'll test better tomorrow but maybe you want to
try it meantime?

Thanks for your patience Murali.
St.Ack
< Prev | 1 - 2 | Next >