Nabble - HBase User

Re: Insert, update performance

2009-12-16T17:19:58Z

On Wed, Dec 16, 2009 at 7:52 AM, Adam Silberstein <silberst@...>wrote:

> I was expecting performance to be similar, but I¹m having problems with
> Case
> B. In particular, after running continuously for 5-10, the latencies go
> way
> up and my throughput goes way down to a couple of operations a second. I
> left it like this for a while in case there were compactions going on, but
> it never returned to its original throughput. I have no problems with Case
> A.
>

So, if you look in the logs on the regionservers, do you see anything? In
particular, there is a writes barrier that we'll bring down if we are not
flushing fast enough. The default is pretty conservative so the barrier
comes down pretty quick. The read load might be slowing the flushing?

Or are splits happening at this time? Are number of regions climbing during
test B?

If you want, post a few regionserver logs somewhere we can pull it and we'll
take a look at it for you.

> Any ideas on what¹s going on here? It may be related to the concurrent
> read
> workload, since I successfully loaded 120 million records originally, and
> that took several hours.

"serveral hours" is good or bad by your estimation? Should it be faster?

Thanks,
St.Ack

Re: Hbase store format

2009-12-16T17:02:23Z

Oh, this is what I am looking for. The wiki of hbase seems a bit out of
date. Maybe some updates need to do to let new comers like me to know more
about hbase.

Jeff Zhang

On Wed, Dec 16, 2009 at 2:21 PM, Ryan Rawson <ryanobjc@...> wrote:

> I concur, the article is really core documentation right now - as in
> it accurately describes the architecture and bits and bytes of the
> system.
>
> It's always gratifying to see your code being diagrammed as your
> originally intended/wrote it.
>
> On Wed, Dec 16, 2009 at 2:00 PM, Wim Van Leuven
> <wim.vanleuven@...> wrote:
> > Or just add the full article to the wiki? Isn't this terrific article
> just
> > core architecture stuff for HBase?
> >
> > -----Original Message-----
> > From: Stack [mailto:saint.ack@...]
> > Sent: woensdag 16 december 2009 22:46
> > To: hbase-user@...
> > Subject: Re: Hbase store format
> >
> > +1
> >
> >
> >
> > On Dec 16, 2009, at 1:39 PM, Lars George <lars.george@...> wrote:
> >
> >> Maybe we should add this to the article page up on wiki? ;)
> >>
> >> On Wed, Dec 16, 2009 at 10:12 PM, Andrew Purtell
> >> <apurtell@...>wrote:
> >>
> >>> http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html
> >>>
> >>>
> >>>
> >>> ----- Original Message ----
> >>>> From: kishore g <g.kishore@...>
> >>>> To: hbase-user@...
> >>>> Sent: Wed, December 16, 2009 1:03:05 PM
> >>>> Subject: Hbase store format
> >>>>
> >>>> Hi,
> >>>>
> >>>> Whats the current implementation in hbase 0.20.2 for storing data.
> >>> MapFile,
> >>>> HFile, TFile.
> >>>>
> >>>> Also any pointers to links explaining the format will be helpful
> >>>>
> >>>> thanks,
> >>>> Kishore G
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >
> >
>

Re: Hbase store format

2009-12-16T14:21:45Z

I concur, the article is really core documentation right now - as in
it accurately describes the architecture and bits and bytes of the
system.

It's always gratifying to see your code being diagrammed as your
originally intended/wrote it.

On Wed, Dec 16, 2009 at 2:00 PM, Wim Van Leuven
<wim.vanleuven@...> wrote:

> Or just add the full article to the wiki? Isn't this terrific article just
> core architecture stuff for HBase?
>
> -----Original Message-----
> From: Stack [mailto:saint.ack@...]
> Sent: woensdag 16 december 2009 22:46
> To: hbase-user@...
> Subject: Re: Hbase store format
>
> +1
>
>
>
> On Dec 16, 2009, at 1:39 PM, Lars George <lars.george@...> wrote:
>
>> Maybe we should add this to the article page up on wiki? ;)
>>
>> On Wed, Dec 16, 2009 at 10:12 PM, Andrew Purtell
>> <apurtell@...>wrote:
>>
>>> http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html
>>>
>>>
>>>
>>> ----- Original Message ----
>>>> From: kishore g <g.kishore@...>
>>>> To: hbase-user@...
>>>> Sent: Wed, December 16, 2009 1:03:05 PM
>>>> Subject: Hbase store format
>>>>
>>>> Hi,
>>>>
>>>> Whats the current implementation in hbase 0.20.2 for storing data.
>>> MapFile,
>>>> HFile, TFile.
>>>>
>>>> Also any pointers to links explaining the format will be helpful
>>>>
>>>> thanks,
>>>> Kishore G
>>>
>>>
>>>
>>>
>>>
>>>
>
>

RE: Hbase store format

2009-12-16T14:00:55Z

Or just add the full article to the wiki? Isn't this terrific article just
core architecture stuff for HBase?

-----Original Message-----
From: Stack [mailto:saint.ack@...]
Sent: woensdag 16 december 2009 22:46
To: hbase-user@...
Subject: Re: Hbase store format

+1

On Dec 16, 2009, at 1:39 PM, Lars George <lars.george@...> wrote:

> Maybe we should add this to the article page up on wiki? ;)
>
> On Wed, Dec 16, 2009 at 10:12 PM, Andrew Purtell
> <apurtell@...>wrote:
>
>> http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html
>>
>>
>>
>> ----- Original Message ----
>>> From: kishore g <g.kishore@...>
>>> To: hbase-user@...
>>> Sent: Wed, December 16, 2009 1:03:05 PM
>>> Subject: Hbase store format
>>>
>>> Hi,
>>>
>>> Whats the current implementation in hbase 0.20.2 for storing data.
>> MapFile,
>>> HFile, TFile.
>>>
>>> Also any pointers to links explaining the format will be helpful
>>>
>>> thanks,
>>> Kishore G
>>
>>
>>
>>
>>
>>

Re: Hbase store format

2009-12-16T13:58:16Z

Yes, its very useful. Also the twiki
http://wiki.apache.org/hadoop/Hbase/HbaseArchitecture should have a diagram.

On Wed, Dec 16, 2009 at 1:45 PM, Stack <saint.ack@...> wrote:

> +1
>
>
>
>
> On Dec 16, 2009, at 1:39 PM, Lars George <lars.george@...> wrote:
>
> Maybe we should add this to the article page up on wiki? ;)
>>
>> On Wed, Dec 16, 2009 at 10:12 PM, Andrew Purtell <apurtell@...
>> >wrote:
>>
>> http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html
>>>
>>>
>>>
>>> ----- Original Message ----
>>>
>>>> From: kishore g <g.kishore@...>
>>>> To: hbase-user@...
>>>> Sent: Wed, December 16, 2009 1:03:05 PM
>>>> Subject: Hbase store format
>>>>
>>>> Hi,
>>>>
>>>> Whats the current implementation in hbase 0.20.2 for storing data.
>>>>
>>> MapFile,
>>>
>>>> HFile, TFile.
>>>>
>>>> Also any pointers to links explaining the format will be helpful
>>>>
>>>> thanks,
>>>> Kishore G
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>

Re: Hbase store format

2009-12-16T13:45:32Z

+1

On Dec 16, 2009, at 1:39 PM, Lars George <lars.george@...> wrote:

Re: Hbase store format

2009-12-16T13:39:22Z

Maybe we should add this to the article page up on wiki? ;)

On Wed, Dec 16, 2009 at 10:12 PM, Andrew Purtell <apurtell@...>wrote:

> http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html
>
>
>
> ----- Original Message ----
> > From: kishore g <g.kishore@...>
> > To: hbase-user@...
> > Sent: Wed, December 16, 2009 1:03:05 PM
> > Subject: Hbase store format
> >
> > Hi,
> >
> > Whats the current implementation in hbase 0.20.2 for storing data.
> MapFile,
> > HFile, TFile.
> >
> > Also any pointers to links explaining the format will be helpful
> >
> > thanks,
> > Kishore G
>
>
>
>
>
>

Re: Hbase store format

2009-12-16T13:16:45Z

StoreFile.java:269 is the line where the tie in between HFile and the
rest of HBase happens.

-ryan

On Wed, Dec 16, 2009 at 1:03 PM, kishore g <g.kishore@...> wrote:

> Hi,
>
> Whats the current implementation in hbase 0.20.2 for storing data. MapFile,
> HFile, TFile.
>
> Also any pointers to links explaining the format will be helpful
>
> thanks,
> Kishore G
>

Re: Hbase store format

2009-12-16T13:12:42Z

http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html

----- Original Message ----

> From: kishore g <g.kishore@...>
> To: hbase-user@...
> Sent: Wed, December 16, 2009 1:03:05 PM
> Subject: Hbase store format
>
> Hi,
>
> Whats the current implementation in hbase 0.20.2 for storing data. MapFile,
> HFile, TFile.
>
> Also any pointers to links explaining the format will be helpful
>
> thanks,
> Kishore G

Hbase store format

2009-12-16T13:03:05Z

Hi,

Whats the current implementation in hbase 0.20.2 for storing data. MapFile,
HFile, TFile.

Also any pointers to links explaining the format will be helpful

thanks,
Kishore G

Re: read Filtered Rows From Secondary Index

2009-12-16T08:16:52Z

Apologies if this reply us not relevant, I'm posting just in case my approach to this problem is useful.

I'm using 20.1 and I'm looking to solve this sane problem by applying the filter to a scan on the root table and using the results to create RowFilter instances which I use in a second scan. The second scan is against the indexed table.

y_823910 wrote:

Hi there,

I don't know how to use ColumnValueFilter in hbase-0.20.2
to read some rows through my built Secondary Index.
Any suggestions or samples?

public static void readFilteredRowsFromSecondaryIndex() throws IOException
{

IndexedTable table = new IndexedTable(config,
Bytes.toBytes("test_table"));

Filter filter = new
ColumnValueFilter(Bytes.toBytes("columnfamily1:column1"),
CompareOp.EQUAL,
Bytes.toBytes("ray1-40"));

ResultScanner scanner = table.getIndexedScanner("column1",

HConstants.EMPTY_START_ROW,
null,
new byte[][] {
Bytes.toBytes("columnfamily1:column1"),

Bytes.toBytes("columnfamily1:column1")},
filter,
new byte[][] {
Bytes.toBytes("columnfamily1:column1"),

Bytes.toBytes("columnfamily1:column1")});
for (Result rowResult : scanner) {

System.out.println(Bytes.toString(rowResult.getValue(Bytes.toBytes("columnfamily1:column1"))));
}

table.close();
}

---------------------------------------------------------------------------------------------------------------------------------------------------
public static void createTableWithSecondaryIndexes() throws IOException {
//HBaseConfiguration conf = new HBaseConfiguration();
HTableDescriptor desc = new HTableDescriptor("test_table");
desc.addFamily(new HColumnDescriptor("columnfamily1"));

IndexedTableDescriptor idx_desc = new IndexedTableDescriptor(desc);
idx_desc.addIndex(new IndexSpecification("column1",
Bytes.toBytes("columnfamily1:column1")));

IndexedTableAdmin admin = null;
admin = new IndexedTableAdmin(config);

if (admin.tableExists(Bytes.toBytes("test_table"))) {
if (admin.isTableEnabled("test_table")) {
admin.disableTable(Bytes.toBytes("test_table"));
}

admin.deleteTable(Bytes.toBytes("test_table"));
}

if (admin.tableExists(Bytes.toBytes("test_table-column1"))) {
if (admin.isTableEnabled("test_table-column1")) {
admin.disableTable(Bytes.toBytes("test_table-column1"));
}

admin.deleteTable(Bytes.toBytes("test_table-column1"));
}

admin.createIndexedTable(idx_desc);
}

-----------------------------------------------------------------------------------------------------------------------------------------

public static void writeToTable() throws IOException {
//HBaseConfiguration conf = new HBaseConfiguration();

IndexedTable table = new IndexedTable(config,
Bytes.toBytes("test_table"));
String row = "test_row";
Put row_update = null;
byte[] column = Bytes.toBytes("columnfamily1");
byte[][] qualifier = new byte[][] {
Bytes.toBytes("column1"),
Bytes.toBytes("column2"),
Bytes.toBytes("column3"),
};

for (int i = 0; i < 100; i++) {
row_update = new Put(Bytes.toBytes(row + i));
row_update.add(column, qualifier[0], Bytes.toBytes("ray1-" +
(i/4)));
row_update.add(column, qualifier[1], Bytes.toBytes("ray2-" +
i));
row_update.add(column, qualifier[2], Bytes.toBytes("ray3-" +
i));
table.put(row_update);
}

table.close();
}

Fleming Chiu(邱宏明)
707-6128
y_823910@tsmc.com
週一無肉日吃素救地球(Meat Free Monday Taiwan)

---------------------------------------------------------------------------
TSMC PROPERTY
This email communication (and any attachments) is proprietary information
for the sole use of its
intended recipient. Any unauthorized review, use or distribution by anyone
other than the intended
recipient is strictly prohibited. If you are not the intended recipient,
please notify the sender by
replying to this email, and then delete this email and any copies of it
immediately. Thank you.
---------------------------------------------------------------------------

Insert, update performance

2009-12-16T07:52:11Z

Hi,
I¹m currently doing some tests comparing 2 use cases:
6 servers
2400 ops per second from 100 clients
Enough ops total to take 30 minutes (2400 * 60 * 30)
120 million 1KB records to start

Case A: Mix of 95% reads, 5% updates
Case B: Mix of 95% reads, 5% inserts

From a client API standpoint, these cases are identical. The only difference
is that in Case A I only do operations on the 120 million keys I already
inserted. In Case B, I do operations on new keys.

I was expecting performance to be similar, but I¹m having problems with Case
B. In particular, after running continuously for 5-10, the latencies go way
up and my throughput goes way down to a couple of operations a second. I
left it like this for a while in case there were compactions going on, but
it never returned to its original throughput. I have no problems with Case
A.

Any ideas on what¹s going on here? It may be related to the concurrent read
workload, since I successfully loaded 120 million records originally, and
that took several hours.

Thanks,
Adam

read Filtered Rows From Secondary Index

2009-12-16T00:27:00Z

Hi there,

I don't know how to use ColumnValueFilter in hbase-0.20.2
to read some rows through my built Secondary Index.
Any suggestions or samples?

public static void readFilteredRowsFromSecondaryIndex() throws IOException
{

IndexedTable table = new IndexedTable(config,
Bytes.toBytes("test_table"));

Filter filter = new
ColumnValueFilter(Bytes.toBytes("columnfamily1:column1"),
CompareOp.EQUAL,
Bytes.toBytes("ray1-40"));

ResultScanner scanner = table.getIndexedScanner("column1",

HConstants.EMPTY_START_ROW,
null,
new byte[][] {
Bytes.toBytes("columnfamily1:column1"),

Bytes.toBytes("columnfamily1:column1")},
filter,
new byte[][] {
Bytes.toBytes("columnfamily1:column1"),

Bytes.toBytes("columnfamily1:column1")});
for (Result rowResult : scanner) {

System.out.println(Bytes.toString(rowResult.getValue(Bytes.toBytes("columnfamily1:column1"))));
}

table.close();
}

---------------------------------------------------------------------------------------------------------------------------------------------------
public static void createTableWithSecondaryIndexes() throws IOException {
//HBaseConfiguration conf = new HBaseConfiguration();
HTableDescriptor desc = new HTableDescriptor("test_table");
desc.addFamily(new HColumnDescriptor("columnfamily1"));

IndexedTableDescriptor idx_desc = new IndexedTableDescriptor(desc);
idx_desc.addIndex(new IndexSpecification("column1",
Bytes.toBytes("columnfamily1:column1")));

IndexedTableAdmin admin = null;
admin = new IndexedTableAdmin(config);

if (admin.tableExists(Bytes.toBytes("test_table"))) {
if (admin.isTableEnabled("test_table")) {
admin.disableTable(Bytes.toBytes("test_table"));
}

admin.deleteTable(Bytes.toBytes("test_table"));
}

if (admin.tableExists(Bytes.toBytes("test_table-column1"))) {
if (admin.isTableEnabled("test_table-column1")) {
admin.disableTable(Bytes.toBytes("test_table-column1"));
}

admin.deleteTable(Bytes.toBytes("test_table-column1"));
}

admin.createIndexedTable(idx_desc);
}

-----------------------------------------------------------------------------------------------------------------------------------------

public static void writeToTable() throws IOException {
//HBaseConfiguration conf = new HBaseConfiguration();

IndexedTable table = new IndexedTable(config,
Bytes.toBytes("test_table"));
String row = "test_row";
Put row_update = null;
byte[] column = Bytes.toBytes("columnfamily1");
byte[][] qualifier = new byte[][] {
Bytes.toBytes("column1"),
Bytes.toBytes("column2"),
Bytes.toBytes("column3"),
};

for (int i = 0; i < 100; i++) {
row_update = new Put(Bytes.toBytes(row + i));
row_update.add(column, qualifier[0], Bytes.toBytes("ray1-" +
(i/4)));
row_update.add(column, qualifier[1], Bytes.toBytes("ray2-" +
i));
row_update.add(column, qualifier[2], Bytes.toBytes("ray3-" +
i));
table.put(row_update);
}

table.close();
}

Fleming Chiu(邱宏明)
707-6128
y_823910@...
週一無肉日吃素救地球(Meat Free Monday Taiwan)

---------------------------------------------------------------------------
TSMC PROPERTY
This email communication (and any attachments) is proprietary information
for the sole use of its
intended recipient. Any unauthorized review, use or distribution by anyone
other than the intended
recipient is strictly prohibited. If you are not the intended recipient,
please notify the sender by
replying to this email, and then delete this email and any copies of it
immediately. Thank you.
---------------------------------------------------------------------------

Re: FilterList and SingleColumnValueFilter

2009-12-15T22:42:49Z

Hey Michael,

If hbase-2037 will make it into 0.20.3, I am fine.
If not, I would greatly appreciate you breaking it out for 0.20.3.

Thanks,
Paul

On Dec 15, 2009, at 10:28 PM, stack wrote:

> Paul:
>
> I can apply the fix from hbase-2037... I can break it out of the posted
> patch thats up there. Just say the word.
>
> St.Ack
>
>
> On Tue, Dec 15, 2009 at 4:17 PM, Ram Kulbak <ram.kulbak@...> wrote:
>
>> Hi Paul,
>>
>> I've encountered the same problem. I think its fixed as part of
>> https://issues.apache.org/jira/browse/HBASE-2037
>>
>> Regards,
>> Yoram
>>
>>
>>
>> On Wed, Dec 16, 2009 at 10:45 AM, Paul Ambrose <pambrose@...> wrote:
>>
>>> I ran into some problems with FilterList and SingleColumnValueFilter.
>>>
>>> I created a FilterList with MUST_PASS_ONE and two
>> SingleColumnValueFilters
>>> (each testing equality on a different columns) and query some trivial
>> data:
>>>
>>> http://pastie.org/744890
>>>
>>> The problem that I encountered were two-fold:
>>>
>>> SingleColumnValueFilter.filterKeyValues() returns ReturnCode.INCLUDE
>>> if the column names do not match. If FilterList is employed, then when
>> the
>>> first Filter returns INCLUDE (because the column names do not match), no
>>> more filters for that KeyValue are evaluated. That is problematic
>> because
>>> when filterRow() is finally called for those filters, matchedColumn is
>>> never
>>> found to be true because they were not invoked (due to FilterList exiting
>>> from
>>> the filterList iteration when the name mismatched INCLUDE was returned).
>>> The fix (at least for this scenario) is for
>>> SingleColumnValueFilter.filterKeyValues() to
>>> return ReturnCode.NEXT_ROW (rather than INCLUDE).
>>>
>>> The second problem is at the bottom of FilterList.filterKeyValue()
>>> where ReturnCode.SKIP is returned if MUST_PASS_ONE is the operator,
>>> rather than always returning ReturnCode.INCLUDE and then leaving the
>>> final filter decision to be made by the call to filterRow(). I am sure
>>> there is a good
>>> reason for returning SKIP in other scenarios, but it is problematic in
>>> mine.
>>>
>>> Feedback would be much appreciated.
>>>
>>> Paul
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>

Re: FilterList and SingleColumnValueFilter

2009-12-15T22:28:09Z

Paul:

I can apply the fix from hbase-2037... I can break it out of the posted
patch thats up there. Just say the word.

St.Ack

On Tue, Dec 15, 2009 at 4:17 PM, Ram Kulbak <ram.kulbak@...> wrote:

> Hi Paul,
>
> I've encountered the same problem. I think its fixed as part of
> https://issues.apache.org/jira/browse/HBASE-2037
>
> Regards,
> Yoram
>
>
>
> On Wed, Dec 16, 2009 at 10:45 AM, Paul Ambrose <pambrose@...> wrote:
>
> > I ran into some problems with FilterList and SingleColumnValueFilter.
> >
> > I created a FilterList with MUST_PASS_ONE and two
> SingleColumnValueFilters
> > (each testing equality on a different columns) and query some trivial
> data:
> >
> > http://pastie.org/744890
> >
> > The problem that I encountered were two-fold:
> >
> > SingleColumnValueFilter.filterKeyValues() returns ReturnCode.INCLUDE
> > if the column names do not match. If FilterList is employed, then when
> the
> > first Filter returns INCLUDE (because the column names do not match), no
> > more filters for that KeyValue are evaluated. That is problematic
> because
> > when filterRow() is finally called for those filters, matchedColumn is
> > never
> > found to be true because they were not invoked (due to FilterList exiting
> > from
> > the filterList iteration when the name mismatched INCLUDE was returned).
> > The fix (at least for this scenario) is for
> > SingleColumnValueFilter.filterKeyValues() to
> > return ReturnCode.NEXT_ROW (rather than INCLUDE).
> >
> > The second problem is at the bottom of FilterList.filterKeyValue()
> > where ReturnCode.SKIP is returned if MUST_PASS_ONE is the operator,
> > rather than always returning ReturnCode.INCLUDE and then leaving the
> > final filter decision to be made by the call to filterRow(). I am sure
> > there is a good
> > reason for returning SKIP in other scenarios, but it is problematic in
> > mine.
> >
> > Feedback would be much appreciated.
> >
> > Paul
> >
> >
> >
> >
> >
> >
> >
> >
>

Re: HBase Utility functions (for Java 5+)

2009-12-15T16:28:31Z

Thanks for the feedback Paul.

I agree the Builder pattern is an interesting option. Please see
https://issues.apache.org/jira/browse/HBASE-2051

- Andy

________________________________
From: Paul Smith <psmith@...>
To: hbase-user@...
Sent: Tue, December 15, 2009 3:21:44 PM
Subject: Re: HBase Utility functions (for Java 5+)

On 16/12/2009, at 7:04 AM, stack wrote:

> On Tue, Dec 15, 2009 at 9:56 AM, Kevin Peterson <kevinpet@...> wrote:
>
>> These kinds of cleaner APIs would be a good way to prevent the standard
>> situation of one engineer on the team figuring out HBase, then others say
>> "why is this so complicated" so they write an internal set of wrappers and
>> utility methods.
>>
>
> This wouldn't solve the problems for people who want a full ORM, but I think
>> there's an in-between sweet spot that abstracts away byte[] but still
>> exposes column families and such.
>>
>
>
> What do fellas think of Lars' George's genercizing (sp? word?) of the client
> API? See his patch up in https://issues.apache.org/jira/browse/HBASE-1990.
> Would this be enough?
> St.Ack

That's a pretty good start, but I think a good collection of useful builders and utilities that handle the 80% case will help HBase gain much more traction. As an person starting with HBase, there are a lot of concepts to get, Bytes definitely get in the way of seeing the real underlying patterns. I'm a total believer in understanding the internals to get the best out of a product, but that often comes after experimentation, and these high-level libraries grease the wheels for faster 'grok'ing the concepts.

Thinking out loud here, but something like this may be useful (more useful?, I dunno, I'm still used to this):

PutBuilder builder = new PutBuilder(hTable);
// first Row
builder.withRowKey(1stRowKey).withColumnFamily("foo").put("columnA", valueA).put("columnB",valueB);
// secondRow
builder.withRowKey(2ndRowKey).withColumnFamily("eek").put("columnC", valueC).put("columnD",valueD);
...
builder.putAll();

I also feel a little silly, because I've only JUST discovered the Writables class, my initial example of packing 4 ints is silly, a simple Class that implements Writeable is a much more elegant solution (I wasn't sure why Bytes.add(..) only took 2 or 3 args).

Paul

Re: FilterList and SingleColumnValueFilter

2009-12-15T16:17:20Z

Hi Paul,

I've encountered the same problem. I think its fixed as part of
https://issues.apache.org/jira/browse/HBASE-2037

Regards,
Yoram

On Wed, Dec 16, 2009 at 10:45 AM, Paul Ambrose <pambrose@...> wrote:

> I ran into some problems with FilterList and SingleColumnValueFilter.
>
> I created a FilterList with MUST_PASS_ONE and two SingleColumnValueFilters
> (each testing equality on a different columns) and query some trivial data:
>
> http://pastie.org/744890
>
> The problem that I encountered were two-fold:
>
> SingleColumnValueFilter.filterKeyValues() returns ReturnCode.INCLUDE
> if the column names do not match. If FilterList is employed, then when the
> first Filter returns INCLUDE (because the column names do not match), no
> more filters for that KeyValue are evaluated. That is problematic because
> when filterRow() is finally called for those filters, matchedColumn is
> never
> found to be true because they were not invoked (due to FilterList exiting
> from
> the filterList iteration when the name mismatched INCLUDE was returned).
> The fix (at least for this scenario) is for
> SingleColumnValueFilter.filterKeyValues() to
> return ReturnCode.NEXT_ROW (rather than INCLUDE).
>
> The second problem is at the bottom of FilterList.filterKeyValue()
> where ReturnCode.SKIP is returned if MUST_PASS_ONE is the operator,
> rather than always returning ReturnCode.INCLUDE and then leaving the
> final filter decision to be made by the call to filterRow(). I am sure
> there is a good
> reason for returning SKIP in other scenarios, but it is problematic in
> mine.
>
> Feedback would be much appreciated.
>
> Paul
>
>
>
>
>
>
>
>

FilterList and SingleColumnValueFilter

2009-12-15T15:45:37Z

I ran into some problems with FilterList and SingleColumnValueFilter.

I created a FilterList with MUST_PASS_ONE and two SingleColumnValueFilters
(each testing equality on a different columns) and query some trivial data:

http://pastie.org/744890

The problem that I encountered were two-fold:

SingleColumnValueFilter.filterKeyValues() returns ReturnCode.INCLUDE
if the column names do not match. If FilterList is employed, then when the
first Filter returns INCLUDE (because the column names do not match), no
more filters for that KeyValue are evaluated. That is problematic because
when filterRow() is finally called for those filters, matchedColumn is never
found to be true because they were not invoked (due to FilterList exiting from
the filterList iteration when the name mismatched INCLUDE was returned).
The fix (at least for this scenario) is for SingleColumnValueFilter.filterKeyValues() to
return ReturnCode.NEXT_ROW (rather than INCLUDE).

The second problem is at the bottom of FilterList.filterKeyValue()
where ReturnCode.SKIP is returned if MUST_PASS_ONE is the operator,
rather than always returning ReturnCode.INCLUDE and then leaving the
final filter decision to be made by the call to filterRow(). I am sure there is a good
reason for returning SKIP in other scenarios, but it is problematic in mine.

Feedback would be much appreciated.

Paul

Re: HBase Utility functions (for Java 5+)

2009-12-15T15:21:44Z

On 16/12/2009, at 7:04 AM, stack wrote:

That's a pretty good start, but I think a good collection of useful builders and utilities that handle the 80% case will help HBase gain much more traction. As an person starting with HBase, there are a lot of concepts to get, Bytes definitely get in the way of seeing the real underlying patterns. I'm a total believer in understanding the internals to get the best out of a product, but that often comes after experimentation, and these high-level libraries grease the wheels for faster 'grok'ing the concepts.

Thinking out loud here, but something like this may be useful (more useful?, I dunno, I'm still used to this):

PutBuilder builder = new PutBuilder(hTable);
// first Row
builder.withRowKey(1stRowKey).withColumnFamily("foo").put("columnA", valueA).put("columnB",valueB);
// secondRow
builder.withRowKey(2ndRowKey).withColumnFamily("eek").put("columnC", valueC).put("columnD",valueD);
..
builder.putAll();

I also feel a little silly, because I've only JUST discovered the Writables class, my initial example of packing 4 ints is silly, a simple Class that implements Writeable is a much more elegant solution (I wasn't sure why Bytes.add(..) only took 2 or 3 args).

Paul

Re: hlogs do not get cleared

2009-12-15T15:17:07Z

On Tue, Dec 15, 2009 at 10:43 AM, Jean-Daniel Cryans <jdcryans@...>wrote:

>
> Too many hlogs means that the inserts are hitting a lot of regions,
> that those regions aren't filled enough to flush so that we have to
> force flush them to give some room. When you added region servers, it
> spread the regions load so that hlogs were getting filled at a slower
> rate.
>
> Could you tell us more about the rate of insertion, size of data, and
> number of regions per region server?
>
>

This makes some sense now. I currently have 2200 regions across 3 tables. My
largest table accounts for about 1600 of those regions and is mostly active
at one end of the keyspace -- our key is based on date, but data only
roughly arrives in order. I also write to two secondary indexes, which have
no pattern to the key at all. One of these secondary tables has 488 regions
and the other has 96 regions.

We write about 10M items per day to the main table (articles). All of these
get written to one of the secondary indexes (article-ids). About a third get
written to the other secondary index. Total volume of data is about 10GB /
day written.

I think the key is as you say that the regions aren't filled enough to
flush. The articles table gets mostly written to near one end and I see
splits happening regularly. The index tables have no pattern so the 10
millions writes get scattered across the different regions. I've looked more
closely at a log file (linked below), and if I forget about my main table
(which would tend to get flushed), and look only at the indexes, this seems
to be what's happening:

1. Up to maxLogs HLogs, it doesn't do any flushes.
2. Once it gets above maxLogs, it will start flushing one region each time
it creates a new HLog.
3. If the first HLog had edits for say 50 regions, it will need to flush the
region with oldest edits 50 times before the HLog can be removed.

If N is the number of regions getting written to, but not getting enough
writes to flush on their own, then I think this converges to maxLogs + N
logs on average. If I think of maxLogs as "number of logs to start flushing
regions at" this makes sense.

http://kdpeterson.net/paste/hbase-hadoop-regionserver-mi-prod-app35.ec2.biz360.com.log.2009-12-14

Re: Help on HBase shell alter command usage

2009-12-15T12:59:29Z

That works.

scan command gives values for columns.
Is there a shell command which lists unique row values, such as
'com.onsoft.www:http/' ?

Thanks

On Tue, Dec 15, 2009 at 12:09 PM, stack <stack@...> wrote:

> Try:
>
> hbase(main):005:0> get 'crawltable', 'com.onsoft.www:http/', { COLUMNS =>
> 'stt:'}
>
> i.e. '=>' rather than '='. Also, its COLUMNS (uppercase I believe) rather
> than column.
>
> Run 'help' in the shell for help and examples.
>
> St.Ack
>
> On Tue, Dec 15, 2009 at 11:53 AM, Ted Yu <yuzhihong@...> wrote:
>
> > Hi,
> > I saw the following from scan 'crawltable' command in hbase shell:
> > ...
> > com.onsoft.www:http/ column=stt:, timestamp=1260405530801,
> > value=\003
> > 3 row(s) in 0.2490 seconds
> >
> > How do I query the value for stt column ?
> >
> > hbase(main):005:0> get 'crawltable', 'com.onsoft.www:http/', {
> > column='stt:'
> > }
> > SyntaxError: (hbase):6: odd number list for Hash.
> > from (hbase):6
> >
> > Can someone explain this 'odd number' error ?
> >
> > Thanks
> >
> > On Mon, Dec 14, 2009 at 10:16 PM, stack <stack@...> wrote:
> >
> > > Are you using hbase 0.20? If so, there is no 'compress'. Its NONE,
> LZO,
> > > or
> > > GZIP (You'll have to build lzo yourself. See hbase wiki for how).
> > >
> > > See the shell help. It has examples of how to change parameters on
> > column
> > > families.
> > >
> > > St.Ack
> > >
> > > 2009/12/14 Xin Jing <xinjing@...>
> > >
> > > > Hi all,
> > > >
> > > > I want to change the column family property for a existing hbase
> table.
> > > > Setting one comlumn family COMPRESSION from 'none' to comress, and
> > chagne
> > > > one column family IN_MEMORY from 'false' to 'true'.
> > > >
> > > > I want to use hbase shell to achieve that, but I cannot find the
> > detailed
> > > > description on 'alter' command. Could anyone point me to a reference
> on
> > > > that?
> > > >
> > > > Thanks
> > > > - Xin
> > > >
> > >
> >
>

Re: HBase Utility functions (for Java 5+)

2009-12-15T12:14:21Z

Seems like an intuitive option to me.

Tim

On Tue, Dec 15, 2009 at 9:04 PM, stack <stack@...> wrote:

Re: Help on HBase shell alter command usage

2009-12-15T12:09:18Z

Try:

hbase(main):005:0> get 'crawltable', 'com.onsoft.www:http/', { COLUMNS =>
'stt:'}

i.e. '=>' rather than '='. Also, its COLUMNS (uppercase I believe) rather
than column.

Run 'help' in the shell for help and examples.

St.Ack

On Tue, Dec 15, 2009 at 11:53 AM, Ted Yu <yuzhihong@...> wrote:

> Hi,
> I saw the following from scan 'crawltable' command in hbase shell:
> ...
> com.onsoft.www:http/ column=stt:, timestamp=1260405530801,
> value=\003
> 3 row(s) in 0.2490 seconds
>
> How do I query the value for stt column ?
>
> hbase(main):005:0> get 'crawltable', 'com.onsoft.www:http/', {
> column='stt:'
> }
> SyntaxError: (hbase):6: odd number list for Hash.
> from (hbase):6
>
> Can someone explain this 'odd number' error ?
>
> Thanks
>
> On Mon, Dec 14, 2009 at 10:16 PM, stack <stack@...> wrote:
>
> > Are you using hbase 0.20? If so, there is no 'compress'. Its NONE, LZO,
> > or
> > GZIP (You'll have to build lzo yourself. See hbase wiki for how).
> >
> > See the shell help. It has examples of how to change parameters on
> column
> > families.
> >
> > St.Ack
> >
> > 2009/12/14 Xin Jing <xinjing@...>
> >
> > > Hi all,
> > >
> > > I want to change the column family property for a existing hbase table.
> > > Setting one comlumn family COMPRESSION from 'none' to comress, and
> chagne
> > > one column family IN_MEMORY from 'false' to 'true'.
> > >
> > > I want to use hbase shell to achieve that, but I cannot find the
> detailed
> > > description on 'alter' command. Could anyone point me to a reference on
> > > that?
> > >
> > > Thanks
> > > - Xin
> > >
> >
>

Re: HBase Utility functions (for Java 5+)

2009-12-15T12:04:23Z

On Tue, Dec 15, 2009 at 9:56 AM, Kevin Peterson <kevinpet@...> wrote:

> These kinds of cleaner APIs would be a good way to prevent the standard
> situation of one engineer on the team figuring out HBase, then others say
> "why is this so complicated" so they write an internal set of wrappers and
> utility methods.
>

This wouldn't solve the problems for people who want a full ORM, but I think
> there's an in-between sweet spot that abstracts away byte[] but still
> exposes column families and such.
>

What do fellas think of Lars' George's genercizing (sp? word?) of the client
API? See his patch up in https://issues.apache.org/jira/browse/HBASE-1990.
Would this be enough?
St.Ack

Re: Help on HBase shell alter command usage

2009-12-15T11:53:18Z

Hi,
I saw the following from scan 'crawltable' command in hbase shell:
...
com.onsoft.www:http/ column=stt:, timestamp=1260405530801,
value=\003
3 row(s) in 0.2490 seconds

How do I query the value for stt column ?

hbase(main):005:0> get 'crawltable', 'com.onsoft.www:http/', { column='stt:'
}
SyntaxError: (hbase):6: odd number list for Hash.
from (hbase):6

Can someone explain this 'odd number' error ?

Thanks

On Mon, Dec 14, 2009 at 10:16 PM, stack <stack@...> wrote:

> Are you using hbase 0.20? If so, there is no 'compress'. Its NONE, LZO,
> or
> GZIP (You'll have to build lzo yourself. See hbase wiki for how).
>
> See the shell help. It has examples of how to change parameters on column
> families.
>
> St.Ack
>
> 2009/12/14 Xin Jing <xinjing@...>
>
> > Hi all,
> >
> > I want to change the column family property for a existing hbase table.
> > Setting one comlumn family COMPRESSION from 'none' to comress, and chagne
> > one column family IN_MEMORY from 'false' to 'true'.
> >
> > I want to use hbase shell to achieve that, but I cannot find the detailed
> > description on 'alter' command. Could anyone point me to a reference on
> > that?
> >
> > Thanks
> > - Xin
> >
>

Re: hlogs do not get cleared

2009-12-15T11:49:46Z

I'd advise setting the upper limit for WALs back down to 32 rather than the
96 you have. Lets figure why old logs are not being cleared up even if only
32. When 96, it means that on crash, the log splitting process has more
logs to process (~96 rather than ~32). It'll take longer for the split
process to run and therefore longer for the regions to come back on line.

Is this the state of things across all regionservers or just one or two? As
J-D asks, your loading profile, how many regions per regionserver would be
of interest. Next up would be your putting up a regionserver log that we
could pull and look at. We'd check the edit sequence numbers to figure why
we're not letting logs go.

Thanks Kevin,
St.Ack

On Tue, Dec 15, 2009 at 10:34 AM, Kevin Peterson <kevinpet@...> wrote:

> We're running a 13 node HBase cluster. We had some problems a week ago with
> it being overloaded and errors related to not being able to find a block on
> HDFS, but adding four more nodes and increasing max heap from 3GB to 4.5GB
> on all nodes fixed any problems.
>
> Looking at the logs now, though, we see that HLogs are not getting removed:
>
> 2009-12-15 01:45:48,426 INFO org.apache.hadoop.hbase.regionserver.HLog:
> Roll
> /hbase/.logs/mi-prod-app33,60020,1260495617070/hlog.dat.1260867136036,
> entries=210524, calcsize=63757422, filesize=41073798. New hlog
> /hbase/.logs/mi-prod-app33,60020,1260495617070/hlog.dat.1260870348421
> 2009-12-15 01:45:48,427 INFO org.apache.hadoop.hbase.regionserver.HLog: Too
> many hlogs: logs=130, maxlogs=96; forcing flush of region with oldest
> edits:
> articles-article-id,f15489ea-38a4-4127-9179-1b2dc5f3b5d4,1260083783909
> 2009-12-15 01:57:14,188 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Starting compaction on region
>
> articles,\x00\x00\x01\x25\x8C\x0F\xCB\x18\xB5U\xF7\xC6\x5DoH\xB8\x98\xEBH,E\x7C\x07\x14,1260830133341
> 2009-12-15 01:57:17,519 INFO org.apache.hadoop.hbase.regionserver.HLog:
> Roll
> /hbase/.logs/mi-prod-app33,60020,1260495617070/hlog.dat.1260870348421,
> entries=92795, calcsize=63908073, filesize=54042783. New hlog
> /hbase/.logs/mi-prod-app33,60020,1260495617070/hlog.dat.1260871037510
> 2009-12-15 01:57:17,519 INFO org.apache.hadoop.hbase.regionserver.HLog: Too
> many hlogs: logs=131, maxlogs=96; forcing flush of region with oldest
> edits:
> articles-article-id,f1cd1b02-3d1b-453c-b44f-94ec5a1e3a46,1260007536878
>
> From reading the log message, I interpret this as saying that every time it
> rolls an hlog, if there are more than maxlogs logs, it will flush one
> region. I'm assuming that a log could have edits for multiple regions, so
> this seems to mean that if we have 100 regions and maxlogs set to 96, if it
> flushes one region each time it rolls a log, it will create 100 logs before
> it flushes all regions and is able to delete the log, so it will reach
> steady state at 196 hlogs. Is this correct?
>
> We're concerned because when we had problems last week, we saw lots of log
> messages related to "Too many hlogs" and had assumed they were related to
> the problems. Is this anything to worry about?
>

Re: running unit test based on HBaseClusterTestCase

2009-12-15T11:44:25Z

Order can be important. Don't forget to include conf directories. Below is
from an eclipse .classpath that seems to work for me:

<classpathentry kind="lib" path="conf"/>
<classpathentry kind="src" path="src/test"/>
<classpathentry kind="src" path="src/java"/>
<classpathentry kind="lib" path="lib/jackson-core-asl-1.0.1.jar"/>
<classpathentry kind="lib" path="build/lib/jetty-6.1.14.jar"/>
<classpathentry kind="lib" path="lib/commons-cli-2.0-SNAPSHOT.jar"/>
<classpathentry kind="lib"
path="lib/commons-el-from-jetty-5.1.4.jar"/>
<classpathentry kind="lib" path="lib/commons-httpclient-3.0.1.jar"/>
<classpathentry kind="lib" path="lib/commons-logging-1.0.4.jar"/>
<classpathentry kind="lib"
path="lib/commons-logging-api-1.0.4.jar"/>
<classpathentry kind="lib" path="lib/commons-math-1.1.jar"/>
<classpathentry kind="lib" path="lib/jasper-compiler-5.5.12.jar"/>
<classpathentry kind="lib" path="lib/jasper-runtime-5.5.12.jar"/>
<classpathentry kind="lib" path="lib/jetty-6.1.14.jar"/>
<classpathentry kind="lib" path="lib/jetty-util-6.1.14.jar"/>
<classpathentry kind="lib" path="lib/jruby-complete-1.3.1.jar"/>
<classpathentry kind="lib" path="lib/json.jar"/>
<classpathentry kind="lib" path="lib/jsp-2.1/jsp-2.1.jar"/>
<classpathentry kind="lib" path="lib/jsp-2.1/jsp-api-2.1.jar"/>
<classpathentry kind="lib" path="lib/junit-4.7.jar"/>
<classpathentry kind="lib" path="lib/libthrift-r771587.jar"/>
<classpathentry kind="lib" path="lib/log4j-1.2.15.jar"/>
<classpathentry kind="lib" path="lib/lucene-core-2.2.0.jar"/>
<classpathentry kind="lib" path="lib/zookeeper-3.2.1.jar"/>
<classpathentry kind="lib"
path="src/contrib/stargate/lib/asm-3.1.jar"/>
<classpathentry kind="lib"
path="src/contrib/stargate/lib/commons-codec-1.3.jar"/>
<classpathentry kind="lib"
path="src/contrib/stargate/lib/jaxb-impl-2.1.10.jar"/>
<classpathentry kind="lib"
path="src/contrib/stargate/lib/jersey-core-1.1.0-ea.jar"/>
<classpathentry kind="lib"
path="src/contrib/stargate/lib/jersey-json-1.1.0-ea.jar"/>
<classpathentry kind="lib"
path="src/contrib/stargate/lib/jersey-server-1.1.0-ea.jar"/>
<classpathentry kind="con"
path="org.eclipse.jdt.launching.JRE_CONTAINER"/>
<classpathentry kind="lib" path="lib/servlet-api-2.5-6.1.14.jar"/>
<classpathentry kind="lib" path="build/hbase-0.21.0-dev.jar"/>
<classpathentry kind="lib" path="lib/avro-1.2.0.jar"/>
<classpathentry kind="lib" path="lib/jackson-mapper-asl-1.0.1.jar"/>
<classpathentry kind="lib"
path="lib/hadoop-core-0.21.0-SNAPSHOT-r832250.jar"/>
<classpathentry kind="lib"
path="lib/hadoop-core-test-0.21.0-SNAPSHOT-r832250.jar"/>
<classpathentry kind="lib"
path="lib/hadoop-mapred-0.21.0-dev-r833993.jar"/>
<classpathentry kind="lib"
path="lib/hadoop-mapred-test-0.21.0-dev-r833993.jar"/>
<classpathentry kind="lib"
path="lib/hadoop-hdfs-0.21.0-dev-r836012.jar"/>
<classpathentry kind="lib"
path="lib/hadoop-hdfs-test-0.21.0-dev-r836012.jar"/>
<classpathentry kind="output" path="eclipse"/>

St.Ack

On Tue, Dec 15, 2009 at 11:21 AM, Guohua Hao <haoguohua@...> wrote:

> Yes, I included all the necessary jar files I think. I guess my problem is
> probably related to my eclipse setup.
>
> I can create a MiniDFSCluster object by running my application in command
> line (e.g., bin/hadoop myApplicationClass) , and a MiniDFSCluster object is
> created inside the main function of myApplicationClass. But I can NOT run
> this program within eclipse, probably I did not do it in the right way. I
> got the similar error message saying
>
> java.lang.NoSuchMethodError:
> org.apache.hadoop.security.
> >
> >
> >
> UserGroupInformation.setCurrentUser(Lorg/apache/hadoop/security/UserGroupInformation;)V
> > at
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:236)
> > at
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:119)
> >
>
> Could you guys please give me more hint?
>
> Thanks
> Guohua
>
>
>
> On Tue, Dec 15, 2009 at 11:59 AM, Stack <saint.ack@...> wrote:
>
> > Do you have hadoop jars in your eclipse classpath?
> > Stack
> >
> >
> >
> >
> > On Dec 14, 2009, at 10:58 PM, Guohua Hao <haoguohua@...> wrote:
> >
> > Hello All,
> >>
> >> In my own application, I have a unit test case which extends
> >> HBaseClusterTestCase in order to test some of my operation over HBase
> >> cluster. I override the setup function in my own test case, and this
> setup
> >> function begins with super.setup() function call.
> >>
> >> When I try to run my unit test from within Eclipse, I got the following
> >> error:
> >>
> >> java.lang.NoSuchMethodError:
> >>
> >>
> org.apache.hadoop.security.UserGroupInformation.setCurrentUser(Lorg/apache/hadoop/security/UserGroupInformation;)V
> >> at
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:236)
> >> at
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:119)
> >> at
> >>
> >>
> org.apache.hadoop.hbase.HBaseClusterTestCase.setUp(HBaseClusterTestCase.java:123)
> >>
> >> I included the hadoop-0.20.1-core.jar in my classpath, since this jar
> file
> >> contains the org.apache.hadoop.security.UserGroupInformation class.
> >>
> >> Could anybody give me some hint on how to solve this problem?
> >>
> >> Thank you very much,
> >> Guohua
> >>
> >
>

Re: Performance related question

2009-12-15T11:21:31Z

Btw, nothing says that ZK users (incl hbase) _must_ run a multi-node ZK
ensemble. For coordination tasks a single ZK server (standalone mode) is
often sufficient, you just need to realize you are sacrificing
reliability/availability.

Going from 1 -> 3 -> 5 -> 7 ZK servers in an ensemble should primarily
be driven by reliability requirements. See this page for details on
performance studies I've made for standalone and 3 server ZK ensembles:
http://bit.ly/4ekN8G

Patrick

Jean-Daniel Cryans wrote:

> Given that m1.small has 1 CPU, 1.7GB of RAM and 1/8 (or less) the IO
> of the host machine and counting in the fact that those machines are
> networked as a whole I expect it to much much slower that your local
> machine. Those machines are so under-powered that the overhead of
> hadoop/hbase probably overwhelms any gain from the total number of
> nodes. Instead do this:
>
> - Replace all your m1.small with m1.large in a factor of 4:1.
> - Don't give ZK their own machine, in such a small environment it
> doesn't make much sense. (give them their own EBS maybe)
> - Use an ensemble of only 3 peers.
> - Give HBase plenty of RAM like 4GB.
>
> WRT your mappers, make sure you use scanner pre-fetching. In your job
> setup set hbase.client.scanner.caching to something like 30.
>
> J-D
>
> On Tue, Dec 15, 2009 at 9:14 AM, Something Something
> <mailinglists19@...> wrote:
>> Thanks J-D & Mtohiko for the tips. Significant improvement in performance,
>> but there's still room for improvement. In my local pseudo distributed mode
>> the 2 map reduce jobs now run in less than 4 minutes (from 32 mins) and in
>> cluster of 10 nodes + 5 zk nodes they run in 11 minutes (down from 1 hour &
>> 30 mins). But still I would like to come to a point where they run faster
>> on a cluster than on my local machine.
>>
>> Here's what I did:
>>
>> 1) Fixed a bug in my code that was causing unnecessary writes to HBase.
>> 2) Added these two lines after creating 'new HTable':
>> table.setAutoFlush(false);
>> table.setWriteBufferSize(1024*1024*12);
>> 3) Added this line after Put:
>> put.setWriteToWAL(false);
>> 4) Added this line (only when running on cluster):
>> job.setNumReduceTasks(20);
>>
>> There are other 64-bit related improvements which I cannot try; mainly
>> because Amazon charges (way) too much for 64-bit machines. It costs me over
>> $25 for 15 machines for less than 3 hours, so I switched to 'm1.small'
>> 32-bit machines. Of course, one of the promises of the distributed
>> computing is that we will be able to use "cheap commodity hardware", right
>> :) So I would like to stick with 'm1.small' for now. (But I am willing to
>> use about 30 machines if that's going to help.)
>>
>> Anyway, I have noticed that one of my Mappers is taking too long. If anyone
>> would share ideas of how to improve Mapper speed, that would be greatly
>> appreciated. Basically, in this Mapper I read about 50,000 rows from a
>> HBase table using TableMapReduceUtil.initTableMapperJob() and do some
>> complex processing for "values" of each row. I don't write anything back in
>> HBase, but I do write quite a few lines (context.write()) to HDFS. Any
>> suggestions?
>>
>> Thanks once again for the help.
>>
>>
>>
>> 2009/12/13 <motohiko.mouri@...>
>>
>>> Hello,
>>>
>>> Something Something <mailinglists19@...> wrote：
>>>> PS: One thing I have noticed is that it goes to 66% very fast and then
>>>> slows down from there..
>>> It seems that only one reducer works. You should increase reduce tasks.
>>> The default reduce task's number is written on
>>> hadoop/docs/mapred-default.html.
>>> The default parameter of mapred.reduce.tasks is 1. So only one reduce task
>>> runs.
>>>
>>> There are two ways to increase reduce tasks:
>>> 1. Use Job.setNumReduceTasks(int tasks) on your MapReduce job file.
>>> 2. Denote more mapred.reduce.tasks on hadoop/conf/mapred-site.xml.
>>>
>>> You can get the best perfomance if you run 20 reduce tasks. The detail of
>>> the number
>>> of reduce tasks is written on
>>> http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html#Reducer
>>> at "How many Reduces?" as J-D wrote. Notice that
>>> JobConf.setNumReduceTasks(int) is
>>> already deprecated, so you should use Job.setNumReduceTasks(int tasks)
>>> rather than
>>> JobConf.setNumReduceTasks(int).
>>> --
>>> Motohiko Mouri
>>>

Re: running unit test based on HBaseClusterTestCase

2009-12-15T11:21:16Z

Yes, I included all the necessary jar files I think. I guess my problem is
probably related to my eclipse setup.

I can create a MiniDFSCluster object by running my application in command
line (e.g., bin/hadoop myApplicationClass) , and a MiniDFSCluster object is
created inside the main function of myApplicationClass. But I can NOT run
this program within eclipse, probably I did not do it in the right way. I
got the similar error message saying

java.lang.NoSuchMethodError:
org.apache.hadoop.security.
>
>
> UserGroupInformation.setCurrentUser(Lorg/apache/hadoop/security/UserGroupInformation;)V
> at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:236)
> at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:119)
>

Could you guys please give me more hint?

Thanks
Guohua

On Tue, Dec 15, 2009 at 11:59 AM, Stack <saint.ack@...> wrote:

> Do you have hadoop jars in your eclipse classpath?
> Stack
>
>
>
>
> On Dec 14, 2009, at 10:58 PM, Guohua Hao <haoguohua@...> wrote:
>
> Hello All,
>>
>> In my own application, I have a unit test case which extends
>> HBaseClusterTestCase in order to test some of my operation over HBase
>> cluster. I override the setup function in my own test case, and this setup
>> function begins with super.setup() function call.
>>
>> When I try to run my unit test from within Eclipse, I got the following
>> error:
>>
>> java.lang.NoSuchMethodError:
>>
>> org.apache.hadoop.security.UserGroupInformation.setCurrentUser(Lorg/apache/hadoop/security/UserGroupInformation;)V
>> at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:236)
>> at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:119)
>> at
>>
>> org.apache.hadoop.hbase.HBaseClusterTestCase.setUp(HBaseClusterTestCase.java:123)
>>
>> I included the hadoop-0.20.1-core.jar in my classpath, since this jar file
>> contains the org.apache.hadoop.security.UserGroupInformation class.
>>
>> Could anybody give me some hint on how to solve this problem?
>>
>> Thank you very much,
>> Guohua
>>
>

Re: Performance related question

2009-12-15T11:13:54Z

Given that m1.small has 1 CPU, 1.7GB of RAM and 1/8 (or less) the IO
of the host machine and counting in the fact that those machines are
networked as a whole I expect it to much much slower that your local
machine. Those machines are so under-powered that the overhead of
hadoop/hbase probably overwhelms any gain from the total number of
nodes. Instead do this:

- Replace all your m1.small with m1.large in a factor of 4:1.
- Don't give ZK their own machine, in such a small environment it
doesn't make much sense. (give them their own EBS maybe)
- Use an ensemble of only 3 peers.
- Give HBase plenty of RAM like 4GB.

WRT your mappers, make sure you use scanner pre-fetching. In your job
setup set hbase.client.scanner.caching to something like 30.

J-D

On Tue, Dec 15, 2009 at 9:14 AM, Something Something
<mailinglists19@...> wrote:

> Thanks J-D & Mtohiko for the tips. Significant improvement in performance,
> but there's still room for improvement. In my local pseudo distributed mode
> the 2 map reduce jobs now run in less than 4 minutes (from 32 mins) and in
> cluster of 10 nodes + 5 zk nodes they run in 11 minutes (down from 1 hour &
> 30 mins). But still I would like to come to a point where they run faster
> on a cluster than on my local machine.
>
> Here's what I did:
>
> 1) Fixed a bug in my code that was causing unnecessary writes to HBase.
> 2) Added these two lines after creating 'new HTable':
> table.setAutoFlush(false);
> table.setWriteBufferSize(1024*1024*12);
> 3) Added this line after Put:
> put.setWriteToWAL(false);
> 4) Added this line (only when running on cluster):
> job.setNumReduceTasks(20);
>
> There are other 64-bit related improvements which I cannot try; mainly
> because Amazon charges (way) too much for 64-bit machines. It costs me over
> $25 for 15 machines for less than 3 hours, so I switched to 'm1.small'
> 32-bit machines. Of course, one of the promises of the distributed
> computing is that we will be able to use "cheap commodity hardware", right
> :) So I would like to stick with 'm1.small' for now. (But I am willing to
> use about 30 machines if that's going to help.)
>
> Anyway, I have noticed that one of my Mappers is taking too long. If anyone
> would share ideas of how to improve Mapper speed, that would be greatly
> appreciated. Basically, in this Mapper I read about 50,000 rows from a
> HBase table using TableMapReduceUtil.initTableMapperJob() and do some
> complex processing for "values" of each row. I don't write anything back in
> HBase, but I do write quite a few lines (context.write()) to HDFS. Any
> suggestions?
>
> Thanks once again for the help.
>
>
>
> 2009/12/13 <motohiko.mouri@...>
>
>> Hello,
>>
>> Something Something <mailinglists19@...> wrote：
>> > PS: One thing I have noticed is that it goes to 66% very fast and then
>> > slows down from there..
>>
>> It seems that only one reducer works. You should increase reduce tasks.
>> The default reduce task's number is written on
>> hadoop/docs/mapred-default.html.
>> The default parameter of mapred.reduce.tasks is 1. So only one reduce task
>> runs.
>>
>> There are two ways to increase reduce tasks:
>> 1. Use Job.setNumReduceTasks(int tasks) on your MapReduce job file.
>> 2. Denote more mapred.reduce.tasks on hadoop/conf/mapred-site.xml.
>>
>> You can get the best perfomance if you run 20 reduce tasks. The detail of
>> the number
>> of reduce tasks is written on
>> http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html#Reducer
>> at "How many Reduces?" as J-D wrote. Notice that
>> JobConf.setNumReduceTasks(int) is
>> already deprecated, so you should use Job.setNumReduceTasks(int tasks)
>> rather than
>> JobConf.setNumReduceTasks(int).
>> --
>> Motohiko Mouri
>>
>

Re: hlogs do not get cleared

2009-12-15T10:43:29Z

Kevin,

Too many hlogs means that the inserts are hitting a lot of regions,
that those regions aren't filled enough to flush so that we have to
force flush them to give some room. When you added region servers, it
spread the regions load so that hlogs were getting filled at a slower
rate.

Could you tell us more about the rate of insertion, size of data, and
number of regions per region server?

Thx,

J-D

On Tue, Dec 15, 2009 at 10:34 AM, Kevin Peterson <kevinpet@...> wrote:

> We're running a 13 node HBase cluster. We had some problems a week ago with
> it being overloaded and errors related to not being able to find a block on
> HDFS, but adding four more nodes and increasing max heap from 3GB to 4.5GB
> on all nodes fixed any problems.
>
> Looking at the logs now, though, we see that HLogs are not getting removed:
>
> 2009-12-15 01:45:48,426 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
> /hbase/.logs/mi-prod-app33,60020,1260495617070/hlog.dat.1260867136036,
> entries=210524, calcsize=63757422, filesize=41073798. New hlog
> /hbase/.logs/mi-prod-app33,60020,1260495617070/hlog.dat.1260870348421
> 2009-12-15 01:45:48,427 INFO org.apache.hadoop.hbase.regionserver.HLog: Too
> many hlogs: logs=130, maxlogs=96; forcing flush of region with oldest edits:
> articles-article-id,f15489ea-38a4-4127-9179-1b2dc5f3b5d4,1260083783909
> 2009-12-15 01:57:14,188 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Starting compaction on region
> articles,\x00\x00\x01\x25\x8C\x0F\xCB\x18\xB5U\xF7\xC6\x5DoH\xB8\x98\xEBH,E\x7C\x07\x14,1260830133341
> 2009-12-15 01:57:17,519 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
> /hbase/.logs/mi-prod-app33,60020,1260495617070/hlog.dat.1260870348421,
> entries=92795, calcsize=63908073, filesize=54042783. New hlog
> /hbase/.logs/mi-prod-app33,60020,1260495617070/hlog.dat.1260871037510
> 2009-12-15 01:57:17,519 INFO org.apache.hadoop.hbase.regionserver.HLog: Too
> many hlogs: logs=131, maxlogs=96; forcing flush of region with oldest edits:
> articles-article-id,f1cd1b02-3d1b-453c-b44f-94ec5a1e3a46,1260007536878
>
> From reading the log message, I interpret this as saying that every time it
> rolls an hlog, if there are more than maxlogs logs, it will flush one
> region. I'm assuming that a log could have edits for multiple regions, so
> this seems to mean that if we have 100 regions and maxlogs set to 96, if it
> flushes one region each time it rolls a log, it will create 100 logs before
> it flushes all regions and is able to delete the log, so it will reach
> steady state at 196 hlogs. Is this correct?
>
> We're concerned because when we had problems last week, we saw lots of log
> messages related to "Too many hlogs" and had assumed they were related to
> the problems. Is this anything to worry about?
>

hlogs do not get cleared

2009-12-15T10:34:51Z

We're running a 13 node HBase cluster. We had some problems a week ago with
it being overloaded and errors related to not being able to find a block on
HDFS, but adding four more nodes and increasing max heap from 3GB to 4.5GB
on all nodes fixed any problems.

Looking at the logs now, though, we see that HLogs are not getting removed:

2009-12-15 01:45:48,426 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/hbase/.logs/mi-prod-app33,60020,1260495617070/hlog.dat.1260867136036,
entries=210524, calcsize=63757422, filesize=41073798. New hlog
/hbase/.logs/mi-prod-app33,60020,1260495617070/hlog.dat.1260870348421
2009-12-15 01:45:48,427 INFO org.apache.hadoop.hbase.regionserver.HLog: Too
many hlogs: logs=130, maxlogs=96; forcing flush of region with oldest edits:
articles-article-id,f15489ea-38a4-4127-9179-1b2dc5f3b5d4,1260083783909
2009-12-15 01:57:14,188 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region
articles,\x00\x00\x01\x25\x8C\x0F\xCB\x18\xB5U\xF7\xC6\x5DoH\xB8\x98\xEBH,E\x7C\x07\x14,1260830133341
2009-12-15 01:57:17,519 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/hbase/.logs/mi-prod-app33,60020,1260495617070/hlog.dat.1260870348421,
entries=92795, calcsize=63908073, filesize=54042783. New hlog
/hbase/.logs/mi-prod-app33,60020,1260495617070/hlog.dat.1260871037510
2009-12-15 01:57:17,519 INFO org.apache.hadoop.hbase.regionserver.HLog: Too
many hlogs: logs=131, maxlogs=96; forcing flush of region with oldest edits:
articles-article-id,f1cd1b02-3d1b-453c-b44f-94ec5a1e3a46,1260007536878

From reading the log message, I interpret this as saying that every time it
rolls an hlog, if there are more than maxlogs logs, it will flush one
region. I'm assuming that a log could have edits for multiple regions, so
this seems to mean that if we have 100 regions and maxlogs set to 96, if it
flushes one region each time it rolls a log, it will create 100 logs before
it flushes all regions and is able to delete the log, so it will reach
steady state at 196 hlogs. Is this correct?

We're concerned because when we had problems last week, we saw lots of log
messages related to "Too many hlogs" and had assumed they were related to
the problems. Is this anything to worry about?

Re: Fwd: What should we expect from Hama Examples Rand?

2009-12-15T10:04:17Z

You are missing some supporting jar.

> java.io.IOException: java.io.IOException: java.lang.NullPointerException
> at java.lang.Class.searchMethods(Unknown Source)

Note that the exception is in a JVM method (java.lang.Class.searchMethods).
This is not really a HBase problem per se, but instead very likely a
classpath issue. It looks to me like the JVM cannot (transitively) load a
class to handle the RPC.

Are all of the supporting jars for Hadoop and HBase on the classpath? It
seems at least one is not.

- Andy

________________________________
From: Edward J. Yoon <edwardyoon@...>
To: hbase-user@...
Cc: Alan.Ratner@...
Sent: Tue, December 15, 2009 1:07:02 AM
Subject: Fwd: What should we expect from Hama Examples Rand?

Hi,

'RAND' example of hama-examples.jar is basically a simple M/R job that
creates a table filled with random numbers. So, before the run Hama,
Pls check whether you able to create tables via hbase shell.

Can someone of Hbase the developers help this problem?

Thanks

---------- Forwarded message ----------
From: Ratner, Alan S (IS) <Alan.Ratner@...>
Date: Sat, Dec 12, 2009 at 1:32 AM
Subject: What should we expect from Hama Examples Rand?
To: hama-user@...

Having fixed the groomserver problem I did the following:

1) clean out /tmp files

2) format Hadoop namenode

3) start Hadoop

4) start HBase/Zookeeper

5) start Hama

6) launch Hama examples rand -m 10 -r 10 2000 2000 30.5% matrixA

The outcome is puzzling. A 3-second long diary shows up in the Hama
log file reporting nothing unusual. But the terminal reported
problems with HBase. When I Googled this problem all I saw was
someone who had multiple versions of HBase on their system. I am
using a fresh VM with Ubuntu 8.04, Hadoop 0.20.1, Zookeeper 3.2.1,
HBase 0.20.2, Hama 0.2.0 and JDK 1.6.0_17. No older versions of
anything was ever installed.

BTW: I thought the problem might be related to my running HBase in
standalone mode so I just switched to HBase pseudo-distributed mode
but I see the same problems.

Any help appreciated. -- Alan

Hama log

==================================================================================================================================

Fri Dec 11 09:23:03 EST 2009 Starting master on ngc

ulimit -n 1024

2009-12-11 09:23:05,512 INFO org.apache.hama.HamaMaster: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting HamaMaster

STARTUP_MSG: host = ngc/127.0.1.1

STARTUP_MSG: args = [start]

STARTUP_MSG: version = 0.20.1

STARTUP_MSG: build =
http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1
-r 810220; compiled by 'oom' on Tue Sep 1 20:55:56 UTC 2009

************************************************************/

2009-12-11 09:23:05,939 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=HamaMaster, port=40000

2009-12-11 09:23:06,634 INFO org.apache.hama.HamaMaster: Cleaning up
the system directory

2009-12-11 09:23:06,710 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting

2009-12-11 09:23:06,716 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 40000: starting

2009-12-11 09:23:06,721 INFO org.apache.hama.HamaMaster: Starting RUNNING

2009-12-11 09:23:06,721 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 40000: starting

Hama terminal

/hadoop-0.20.1-test.jar:/home/ngc/Desktop/hama/bin/../lib/hbase-0.20.0.jar:/home/ngc/Desktop/hama/bin/../lib/hbase-0.20.0-test.jar:/home/ngc/Desktop/hama/bin/../lib/jasper-

compiler-5.5.12.jar:/home/ngc/Desktop/hama/bin/../lib/jasper-runtime-5.5.12.jar:/home/ngc/Desktop/hama/bin/../lib/javacc.jar:/home/ngc/Desktop/hama/bin/../lib/jetty-6.1.14.jar:/home/ngc/Desktop/hama/bin/../lib/jetty-util-6.1.14.jar:/home/ngc/Desktop/hama/bin/../lib/jruby-complete-1.2.0.jar:/home/ngc/Desktop/hama/bin/../lib/json.jar:/home/ngc/Desktop/hama/bin/../lib/junit-3.8.1.jar:/home/ngc/Desktop/hama/bin/../lib/libthrift-r771587.jar:/home/ngc/Desktop/hama/bin/../lib/log4j-1.2.13.jar:/home/ngc/Desktop/hama/bin/../lib/log4j-1.2.15.jar:/home/ngc/Desktop/hama/bin/../lib/servlet-api-2.5-6.1.14.jar:/home/ngc/Desktop/hama/bin/../lib/xmlenc-0.52.jar:/home/ngc/Desktop/hama/bin/../lib/zookeeper-3.2.1.jar:/home/ngc/Desktop/hama/bin/../lib/jetty-ext/*.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/annotations.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/ant.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/asm-3.0.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/as
m-analysis-3.0.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/asm-commons-3.0.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/asm-tree-3.0.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/asm-util-3.0.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/asm-xml-3.0.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/bcel.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/dom4j-full.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/findbugs-ant.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/findbugsGUI.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/findbugs.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/jsr305.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/plugin/coreplugin.jar:/home/ngc/Desktop/hadoop-0.20.1/conf:/home/ngc/Desktop/hbase-0.20.2/conf

09/12/11 09:25:12 INFO zookeeper.ZooKeeper: Client
environment:java.library.path=/home/ngc/Desktop/jre1.6.0_17/lib/i386/client:/home/ngc/Desktop/jre1.6.0_17/lib/i386:/home/ngc/Desktop/jre1.6.0_17/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib

09/12/11 09:25:12 INFO zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp

09/12/11 09:25:12 INFO zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>

09/12/11 09:25:12 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux

09/12/11 09:25:12 INFO zookeeper.ZooKeeper: Client environment:os.arch=i386

09/12/11 09:25:12 INFO zookeeper.ZooKeeper: Client
environment:os.version=2.6.24-23-generic

09/12/11 09:25:12 INFO zookeeper.ZooKeeper: Client environment:user.name=ngc

09/12/11 09:25:12 INFO zookeeper.ZooKeeper: Client
environment:user.home=/home/ngc

09/12/11 09:25:12 INFO zookeeper.ZooKeeper: Client
environment:user.dir=/home/ngc/Desktop/hama

09/12/11 09:25:12 INFO zookeeper.ZooKeeper: Initiating client
connection, connectString=localhost:2181 sessionTimeout=60000
watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher@1d8957f

09/12/11 09:25:12 INFO zookeeper.ClientCnxn:
zookeeper.disableAutoWatchReset is false

09/12/11 09:25:12 INFO zookeeper.ClientCnxn: Attempting connection to
server localhost/127.0.0.1:2181

09/12/11 09:25:12 INFO zookeeper.ClientCnxn: Priming connection to
java.nio.channels.SocketChannel[connected local=/127.0.0.1:37044
remote=localhost/127.0.0.1:2181]

09/12/11 09:25:12 INFO zookeeper.ClientCnxn: Server connection successful

09/12/11 09:26:30 WARN client.HConnectionManager$TableServers: Testing
for table existence threw exception

org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
contact region server null for region , row '', but failed after 10
attempts.

Exceptions:

java.io.IOException: java.io.IOException: java.lang.NullPointerException

at java.lang.Class.searchMethods(Unknown Source)

at java.lang.Class.getMethod0(Unknown Source)

at java.lang.Class.getMethod(Unknown Source)

at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:643)

at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)

(repeated many times)

HBase log

2009-12-11 09:22:56,110 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes:
Total=0.9374695MB (983008), Free=198.75003MB (208404512),
Max=199.6875MB (209387520), Counts: Blocks=1, Access=10, Hit=9,
Miss=1, Evictions=0, Evicted=0, Ratios: Hit Ratio=89.99999761581421%,
Miss Ratio=10.000000149011612%, Evicted/Run=NaN

2009-12-11 09:23:35,186 INFO
org.apache.hadoop.hbase.master.ServerManager: 1 region servers, 0
dead, average load 2.0

2009-12-11 09:23:35,839 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scanning meta region {server: 127.0.1.1:37976, regionname: -ROOT-,,0,
startKey: <>}

2009-12-11 09:23:35,876 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scan of 1 row(s) of meta region {server: 127.0.1.1:37976, regionname:
-ROOT-,,0, startKey: <>} complete

2009-12-11 09:23:35,886 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scanning meta region {server: 127.0.1.1:37976, regionname: .META.,,1,
startKey: <>}

2009-12-11 09:23:35,893 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scan of 0 row(s) of meta region {server: 127.0.1.1:37976, regionname:
..META.,,1, startKey: <>} complete

2009-12-11 09:23:35,893 INFO
org.apache.hadoop.hbase.master.BaseScanner: All 1 .META. region(s)
scanned

2009-12-11 09:23:56,084 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes:
Total=0.9374695MB (983008), Free=198.75003MB (208404512),
Max=199.6875MB (209387520), Counts: Blocks=1, Access=12, Hit=11,
Miss=1, Evictions=0, Evicted=0, Ratios: Hit Ratio=91.66666865348816%,
Miss Ratio=8.33333358168602%, Evicted/Run=NaN

2009-12-11 09:24:35,188 INFO
org.apache.hadoop.hbase.master.ServerManager: 1 region servers, 0
dead, average load 2.0

2009-12-11 09:24:35,840 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scanning meta region {server: 127.0.1.1:37976, regionname: -ROOT-,,0,
startKey: <>}

2009-12-11 09:24:35,864 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scan of 1 row(s) of meta region {server: 127.0.1.1:37976, regionname:
-ROOT-,,0, startKey: <>} complete

2009-12-11 09:24:35,889 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scanning meta region {server: 127.0.1.1:37976, regionname: .META.,,1,
startKey: <>}

2009-12-11 09:24:35,904 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scan of 0 row(s) of meta region {server: 127.0.1.1:37976, regionname:
..META.,,1, startKey: <>} complete

2009-12-11 09:24:35,904 INFO
org.apache.hadoop.hbase.master.BaseScanner: All 1 .META. region(s)
scanned

2009-12-11 09:24:56,066 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes:
Total=0.9374695MB (983008), Free=198.75003MB (208404512),
Max=199.6875MB (209387520), Counts: Blocks=1, Access=14, Hit=13,
Miss=1, Evictions=0, Evicted=0, Ratios: Hit Ratio=92.85714030265808%,
Miss Ratio=7.14285746216774%, Evicted/Run=NaN

2009-12-11 09:25:02,258 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: setting compaction
limit to 4

2009-12-11 09:25:12,715 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 6 on 37976, call null([B@1c01ba9) from 127.0.1.1:55801:
error: java.io.IOException: java.lang.NullPointerException

java.io.IOException: java.lang.NullPointerException

at java.lang.Class.searchMethods(Unknown Source)

at java.lang.Class.getMethod0(Unknown Source)

at java.lang.Class.getMethod(Unknown Source)

at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:643)

at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)

(repeated many times)

--
Best Regards, Edward J. Yoon @ NHN, corp.
edwardyoon@...
http://blog.udanax.org

Re: running unit test based on HBaseClusterTestCase

2009-12-15T09:59:23Z

Do you have hadoop jars in your eclipse classpath?
Stack

On Dec 14, 2009, at 10:58 PM, Guohua Hao <haoguohua@...> wrote:

> Hello All,
>
> In my own application, I have a unit test case which extends
> HBaseClusterTestCase in order to test some of my operation over HBase
> cluster. I override the setup function in my own test case, and this
> setup
> function begins with super.setup() function call.
>
> When I try to run my unit test from within Eclipse, I got the
> following
> error:
>
> java.lang.NoSuchMethodError:
> org.apache.hadoop.security.UserGroupInformation.setCurrentUser(Lorg/
> apache/hadoop/security/UserGroupInformation;)V
> at org.apache.hadoop.hdfs.MiniDFSCluster.<init>
> (MiniDFSCluster.java:236)
> at org.apache.hadoop.hdfs.MiniDFSCluster.<init>
> (MiniDFSCluster.java:119)
> at
> org.apache.hadoop.hbase.HBaseClusterTestCase.setUp
> (HBaseClusterTestCase.java:123)
>
> I included the hadoop-0.20.1-core.jar in my classpath, since this
> jar file
> contains the org.apache.hadoop.security.UserGroupInformation class.
>
> Could anybody give me some hint on how to solve this problem?
>
> Thank you very much,
> Guohua

Re: HBase Utility functions (for Java 5+)

2009-12-15T09:56:12Z

On Tue, Dec 15, 2009 at 9:21 AM, Gary Helmling <ghelmling@...> wrote:

> I completely agree with the need to understand both the fundamental HBase
> API, and how HBase stores data at a low level. Both are very important in
> knowing how to structure your data for best performance. Which you should
> figure out before moving on to other niceties. Code (ASL it). I think
> Hbase is
>
> On the other hand, forcing the user to understand the details of how data
is stored instead of presenting them with a well abstracted api makes the
learning curve steeper.

These kinds of cleaner APIs would be a good way to prevent the standard
situation of one engineer on the team figuring out HBase, then others say
"why is this so complicated" so they write an internal set of wrappers and
utility methods.

This wouldn't solve the problems for people who want a full ORM, but I think
there's an in-between sweet spot that abstracts away byte[] but still
exposes column families and such.