None of my tables are showing up

[Ananth Sarathy]

I just restarted Hbase and when I go into the shell and type list, none of
my tables are listed, but I see all the data/blocks in s3.

here is the master log when it's restarted

http://pastebin.com/m1ebb7217

this happened once before, but we just started over since it was early in
our process. This time we have a lot of data, and need to keep it.


Ananth T Sarathy

[Jonathan Gray-2]

Do you see the files/blocks in HDFS?

Ananth T. Sarathy wrote:

> I just restarted Hbase and when I go into the shell and type list, none of
> my tables are listed, but I see all the data/blocks in s3.
>
> here is the master log when it's restarted
>
> http://pastebin.com/m1ebb7217
>
> this happened once before, but we just started over since it was early in
> our process. This time we have a lot of data, and need to keep it.
>
>
> Ananth T Sarathy
>

[Ananth Sarathy]

i see all my blocks in my s3 bucket.
Ananth T Sarathy


On Mon, Oct 26, 2009 at 12:17 PM, Jonathan Gray <jlist@...> wrote:

> Do you see the files/blocks in HDFS?
>
>
> Ananth T. Sarathy wrote:
>
>> I just restarted Hbase and when I go into the shell and type list, none of
>> my tables are listed, but I see all the data/blocks in s3.
>>
>> here is the master log when it's restarted
>>
>> http://pastebin.com/m1ebb7217
>>
>> this happened once before, but we just started over since it was early in
>> our process. This time we have a lot of data, and need to keep it.
>>
>>
>> Ananth T Sarathy
>>
>>

[Jonathan Gray-2]

Not S3, HDFS.  Can you checkout the web ui or using the command-line
interface?

$HADOOP_HOME/bin/hadoop dfs -lsr /hbase

...would be a good start

Ananth T. Sarathy wrote:

> i see all my blocks in my s3 bucket.
> Ananth T Sarathy
>
>
> On Mon, Oct 26, 2009 at 12:17 PM, Jonathan Gray <jlist@...> wrote:
>
>> Do you see the files/blocks in HDFS?
>>
>>
>> Ananth T. Sarathy wrote:
>>
>>> I just restarted Hbase and when I go into the shell and type list, none of
>>> my tables are listed, but I see all the data/blocks in s3.
>>>
>>> here is the master log when it's restarted
>>>
>>> http://pastebin.com/m1ebb7217
>>>
>>> this happened once before, but we just started over since it was early in
>>> our process. This time we have a lot of data, and need to keep it.
>>>
>>>
>>> Ananth T Sarathy
>>>
>>>
>

[Ananth Sarathy]

I am confused , why would I need a hadoop home if I am using s3 and the
jets3t package to write to s3?
Ananth T Sarathy


On Mon, Oct 26, 2009 at 12:25 PM, Jonathan Gray <jlist@...> wrote:

> Not S3, HDFS.  Can you checkout the web ui or using the command-line
> interface?
>
> $HADOOP_HOME/bin/hadoop dfs -lsr /hbase
>
> ...would be a good start
>
>
> Ananth T. Sarathy wrote:
>
>> i see all my blocks in my s3 bucket.
>> Ananth T Sarathy
>>
>>
>> On Mon, Oct 26, 2009 at 12:17 PM, Jonathan Gray <jlist@...>
>> wrote:
>>
>>  Do you see the files/blocks in HDFS?
>>>
>>>
>>> Ananth T. Sarathy wrote:
>>>
>>>  I just restarted Hbase and when I go into the shell and type list, none
>>>> of
>>>> my tables are listed, but I see all the data/blocks in s3.
>>>>
>>>> here is the master log when it's restarted
>>>>
>>>> http://pastebin.com/m1ebb7217
>>>>
>>>> this happened once before, but we just started over since it was early
>>>> in
>>>> our process. This time we have a lot of data, and need to keep it.
>>>>
>>>>
>>>> Ananth T Sarathy
>>>>
>>>>
>>>>
>>

[stack-3]

Your log would seem to say that there are no tables in hbase:

2009-10-26 11:40:13,984 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of 0 row(s) of meta region {server:
10.245.82.160:60020, regionname: .META.,,1, startKey: <>} complete

Do as Jon suggests.  Do you see listing of regions?  If so, it would seem
that edits to .META. table are not persisting on your s3 hdfs.   You might
be able to add in all tables using the bin/loadtable.rb script; it reads the
.regioninfo files in all regions and per region adds to .META. an entry.

St.Ack

On Mon, Oct 26, 2009 at 9:31 AM, Ananth T. Sarathy <
ananth.t.sarathy@...> wrote:

> I am confused , why would I need a hadoop home if I am using s3 and the
> jets3t package to write to s3?
> Ananth T Sarathy
>
>
> On Mon, Oct 26, 2009 at 12:25 PM, Jonathan Gray <jlist@...> wrote:
>
> > Not S3, HDFS.  Can you checkout the web ui or using the command-line
> > interface?
> >
> > $HADOOP_HOME/bin/hadoop dfs -lsr /hbase
> >
> > ...would be a good start
> >
> >
> > Ananth T. Sarathy wrote:
> >
> >> i see all my blocks in my s3 bucket.
> >> Ananth T Sarathy
> >>
> >>
> >> On Mon, Oct 26, 2009 at 12:17 PM, Jonathan Gray <jlist@...>
> >> wrote:
> >>
> >>  Do you see the files/blocks in HDFS?
> >>>
> >>>
> >>> Ananth T. Sarathy wrote:
> >>>
> >>>  I just restarted Hbase and when I go into the shell and type list,
> none
> >>>> of
> >>>> my tables are listed, but I see all the data/blocks in s3.
> >>>>
> >>>> here is the master log when it's restarted
> >>>>
> >>>> http://pastebin.com/m1ebb7217
> >>>>
> >>>> this happened once before, but we just started over since it was early
> >>>> in
> >>>> our process. This time we have a lot of data, and need to keep it.
> >>>>
> >>>>
> >>>> Ananth T Sarathy
> >>>>
> >>>>
> >>>>
> >>
>

[Jonathan Gray-2]

So you are not using Hadoop?  HBase is connecting directly to S3?  As
far as I know, this is not possible.  If it is possible, I don't
recommend it and we cannot provide much help because in general we don't
really recommend using S3 at all (even with HDFS).

stack wrote:

> Your log would seem to say that there are no tables in hbase:
>
> 2009-10-26 11:40:13,984 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scan of 0 row(s) of meta region {server:
> 10.245.82.160:60020, regionname: .META.,,1, startKey: <>} complete
>
> Do as Jon suggests.  Do you see listing of regions?  If so, it would seem
> that edits to .META. table are not persisting on your s3 hdfs.   You might
> be able to add in all tables using the bin/loadtable.rb script; it reads the
> .regioninfo files in all regions and per region adds to .META. an entry.
>
> St.Ack
>
> On Mon, Oct 26, 2009 at 9:31 AM, Ananth T. Sarathy <
> ananth.t.sarathy@...> wrote:
>
>> I am confused , why would I need a hadoop home if I am using s3 and the
>> jets3t package to write to s3?
>> Ananth T Sarathy
>>
>>
>> On Mon, Oct 26, 2009 at 12:25 PM, Jonathan Gray <jlist@...> wrote:
>>
>>> Not S3, HDFS.  Can you checkout the web ui or using the command-line
>>> interface?
>>>
>>> $HADOOP_HOME/bin/hadoop dfs -lsr /hbase
>>>
>>> ...would be a good start
>>>
>>>
>>> Ananth T. Sarathy wrote:
>>>
>>>> i see all my blocks in my s3 bucket.
>>>> Ananth T Sarathy
>>>>
>>>>
>>>> On Mon, Oct 26, 2009 at 12:17 PM, Jonathan Gray <jlist@...>
>>>> wrote:
>>>>
>>>>  Do you see the files/blocks in HDFS?
>>>>>
>>>>> Ananth T. Sarathy wrote:
>>>>>
>>>>>  I just restarted Hbase and when I go into the shell and type list,
>> none
>>>>>> of
>>>>>> my tables are listed, but I see all the data/blocks in s3.
>>>>>>
>>>>>> here is the master log when it's restarted
>>>>>>
>>>>>> http://pastebin.com/m1ebb7217
>>>>>>
>>>>>> this happened once before, but we just started over since it was early
>>>>>> in
>>>>>> our process. This time we have a lot of data, and need to keep it.
>>>>>>
>>>>>>
>>>>>> Ananth T Sarathy
>>>>>>
>>>>>>
>>>>>>
>

[Ananth Sarathy]

Definetly possible. It's reading the core-site.xml from my hbase/conf dir.
It's using http://jets3t.s3.amazonaws.com/downloads.html to connect.   When
I was investigating how to connect, that's what i cam across.  If this
wrong, I suppose I can redo it.

Ananth T Sarathy


On Mon, Oct 26, 2009 at 12:48 PM, Jonathan Gray <jlist@...> wrote:

> So you are not using Hadoop?  HBase is connecting directly to S3?  As far
> as I know, this is not possible.  If it is possible, I don't recommend it
> and we cannot provide much help because in general we don't really recommend
> using S3 at all (even with HDFS).
>
>
> stack wrote:
>
>> Your log would seem to say that there are no tables in hbase:
>>
>> 2009-10-26 11:40:13,984 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scan of 0 row(s) of meta region {server:
>> 10.245.82.160:60020, regionname: .META.,,1, startKey: <>} complete
>>
>> Do as Jon suggests.  Do you see listing of regions?  If so, it would seem
>> that edits to .META. table are not persisting on your s3 hdfs.   You might
>> be able to add in all tables using the bin/loadtable.rb script; it reads
>> the
>> .regioninfo files in all regions and per region adds to .META. an entry.
>>
>> St.Ack
>>
>> On Mon, Oct 26, 2009 at 9:31 AM, Ananth T. Sarathy <
>> ananth.t.sarathy@...> wrote:
>>
>>  I am confused , why would I need a hadoop home if I am using s3 and the
>>> jets3t package to write to s3?
>>> Ananth T Sarathy
>>>
>>>
>>> On Mon, Oct 26, 2009 at 12:25 PM, Jonathan Gray <jlist@...>
>>> wrote:
>>>
>>>  Not S3, HDFS.  Can you checkout the web ui or using the command-line
>>>> interface?
>>>>
>>>> $HADOOP_HOME/bin/hadoop dfs -lsr /hbase
>>>>
>>>> ...would be a good start
>>>>
>>>>
>>>> Ananth T. Sarathy wrote:
>>>>
>>>>  i see all my blocks in my s3 bucket.
>>>>> Ananth T Sarathy
>>>>>
>>>>>
>>>>> On Mon, Oct 26, 2009 at 12:17 PM, Jonathan Gray <jlist@...>
>>>>> wrote:
>>>>>
>>>>>  Do you see the files/blocks in HDFS?
>>>>>
>>>>>>
>>>>>> Ananth T. Sarathy wrote:
>>>>>>
>>>>>>  I just restarted Hbase and when I go into the shell and type list,
>>>>>>
>>>>> none
>>>
>>>> of
>>>>>>> my tables are listed, but I see all the data/blocks in s3.
>>>>>>>
>>>>>>> here is the master log when it's restarted
>>>>>>>
>>>>>>> http://pastebin.com/m1ebb7217
>>>>>>>
>>>>>>> this happened once before, but we just started over since it was
>>>>>>> early
>>>>>>> in
>>>>>>> our process. This time we have a lot of data, and need to keep it.
>>>>>>>
>>>>>>>
>>>>>>> Ananth T Sarathy
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>

[Ananth Sarathy]

I don't have a loadtable.rb

is that in 0.20.0?
This is what i have in bin

Formatter.rb   hbase-config.sh   regionservers.sh  zookeepers.sh
HBase.rb       hbase-daemon.sh   rename_table.rb
copy_table.rb  hbase-daemons.sh  start-hbase.sh
hbase          hirb.rb           stop-hbase.sh

Ananth T Sarathy


On Mon, Oct 26, 2009 at 12:45 PM, stack <stack@...> wrote:

> Your log would seem to say that there are no tables in hbase:
>
> 2009-10-26 11:40:13,984 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scan of 0 row(s) of meta region {server:
> 10.245.82.160:60020, regionname: .META.,,1, startKey: <>} complete
>
> Do as Jon suggests.  Do you see listing of regions?  If so, it would seem
> that edits to .META. table are not persisting on your s3 hdfs.   You might
> be able to add in all tables using the bin/loadtable.rb script; it reads
> the
> .regioninfo files in all regions and per region adds to .META. an entry.
>
> St.Ack
>
> On Mon, Oct 26, 2009 at 9:31 AM, Ananth T. Sarathy <
> ananth.t.sarathy@...> wrote:
>
> > I am confused , why would I need a hadoop home if I am using s3 and the
> > jets3t package to write to s3?
> > Ananth T Sarathy
> >
> >
> > On Mon, Oct 26, 2009 at 12:25 PM, Jonathan Gray <jlist@...>
> wrote:
> >
> > > Not S3, HDFS.  Can you checkout the web ui or using the command-line
> > > interface?
> > >
> > > $HADOOP_HOME/bin/hadoop dfs -lsr /hbase
> > >
> > > ...would be a good start
> > >
> > >
> > > Ananth T. Sarathy wrote:
> > >
> > >> i see all my blocks in my s3 bucket.
> > >> Ananth T Sarathy
> > >>
> > >>
> > >> On Mon, Oct 26, 2009 at 12:17 PM, Jonathan Gray <jlist@...>
> > >> wrote:
> > >>
> > >>  Do you see the files/blocks in HDFS?
> > >>>
> > >>>
> > >>> Ananth T. Sarathy wrote:
> > >>>
> > >>>  I just restarted Hbase and when I go into the shell and type list,
> > none
> > >>>> of
> > >>>> my tables are listed, but I see all the data/blocks in s3.
> > >>>>
> > >>>> here is the master log when it's restarted
> > >>>>
> > >>>> http://pastebin.com/m1ebb7217
> > >>>>
> > >>>> this happened once before, but we just started over since it was
> early
> > >>>> in
> > >>>> our process. This time we have a lot of data, and need to keep it.
> > >>>>
> > >>>>
> > >>>> Ananth T Sarathy
> > >>>>
> > >>>>
> > >>>>
> > >>
> >
>

[Jonathan Gray-2]

I think it's only in 0.20.1 and above (might even only be in 0.20 branch).

I'm a bit confused about what you're doing.  Reading the conf file from S3?

What is the configuration value for "hbase.rootdir"

Ananth T. Sarathy wrote:

> I don't have a loadtable.rb
>
> is that in 0.20.0?
> This is what i have in bin
>
> Formatter.rb   hbase-config.sh   regionservers.sh  zookeepers.sh
> HBase.rb       hbase-daemon.sh   rename_table.rb
> copy_table.rb  hbase-daemons.sh  start-hbase.sh
> hbase          hirb.rb           stop-hbase.sh
>
> Ananth T Sarathy
>
>
> On Mon, Oct 26, 2009 at 12:45 PM, stack <stack@...> wrote:
>
>> Your log would seem to say that there are no tables in hbase:
>>
>> 2009-10-26 11:40:13,984 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scan of 0 row(s) of meta region {server:
>> 10.245.82.160:60020, regionname: .META.,,1, startKey: <>} complete
>>
>> Do as Jon suggests.  Do you see listing of regions?  If so, it would seem
>> that edits to .META. table are not persisting on your s3 hdfs.   You might
>> be able to add in all tables using the bin/loadtable.rb script; it reads
>> the
>> .regioninfo files in all regions and per region adds to .META. an entry.
>>
>> St.Ack
>>
>> On Mon, Oct 26, 2009 at 9:31 AM, Ananth T. Sarathy <
>> ananth.t.sarathy@...> wrote:
>>
>>> I am confused , why would I need a hadoop home if I am using s3 and the
>>> jets3t package to write to s3?
>>> Ananth T Sarathy
>>>
>>>
>>> On Mon, Oct 26, 2009 at 12:25 PM, Jonathan Gray <jlist@...>
>> wrote:
>>>> Not S3, HDFS.  Can you checkout the web ui or using the command-line
>>>> interface?
>>>>
>>>> $HADOOP_HOME/bin/hadoop dfs -lsr /hbase
>>>>
>>>> ...would be a good start
>>>>
>>>>
>>>> Ananth T. Sarathy wrote:
>>>>
>>>>> i see all my blocks in my s3 bucket.
>>>>> Ananth T Sarathy
>>>>>
>>>>>
>>>>> On Mon, Oct 26, 2009 at 12:17 PM, Jonathan Gray <jlist@...>
>>>>> wrote:
>>>>>
>>>>>  Do you see the files/blocks in HDFS?
>>>>>>
>>>>>> Ananth T. Sarathy wrote:
>>>>>>
>>>>>>  I just restarted Hbase and when I go into the shell and type list,
>>> none
>>>>>>> of
>>>>>>> my tables are listed, but I see all the data/blocks in s3.
>>>>>>>
>>>>>>> here is the master log when it's restarted
>>>>>>>
>>>>>>> http://pastebin.com/m1ebb7217
>>>>>>>
>>>>>>> this happened once before, but we just started over since it was
>> early
>>>>>>> in
>>>>>>> our process. This time we have a lot of data, and need to keep it.
>>>>>>>
>>>>>>>
>>>>>>> Ananth T Sarathy
>>>>>>>
>>>>>>>
>>>>>>>
>

[stack-3]

Its in head of the 0.20 branch (We should roll a 0.20.2 soon).

What happens if you do a lsr /hbase?

St.Ack

On Mon, Oct 26, 2009 at 9:52 AM, Ananth T. Sarathy <
ananth.t.sarathy@...> wrote:

> I don't have a loadtable.rb
>
> is that in 0.20.0?
> This is what i have in bin
>
> Formatter.rb   hbase-config.sh   regionservers.sh  zookeepers.sh
> HBase.rb       hbase-daemon.sh   rename_table.rb
> copy_table.rb  hbase-daemons.sh  start-hbase.sh
> hbase          hirb.rb           stop-hbase.sh
>
> Ananth T Sarathy
>
>
> On Mon, Oct 26, 2009 at 12:45 PM, stack <stack@...> wrote:
>
> > Your log would seem to say that there are no tables in hbase:
> >
> > 2009-10-26 11:40:13,984 INFO org.apache.hadoop.hbase.master.BaseScanner:
> > RegionManager.metaScanner scan of 0 row(s) of meta region {server:
> > 10.245.82.160:60020, regionname: .META.,,1, startKey: <>} complete
> >
> > Do as Jon suggests.  Do you see listing of regions?  If so, it would seem
> > that edits to .META. table are not persisting on your s3 hdfs.   You
> might
> > be able to add in all tables using the bin/loadtable.rb script; it reads
> > the
> > .regioninfo files in all regions and per region adds to .META. an entry.
> >
> > St.Ack
> >
> > On Mon, Oct 26, 2009 at 9:31 AM, Ananth T. Sarathy <
> > ananth.t.sarathy@...> wrote:
> >
> > > I am confused , why would I need a hadoop home if I am using s3 and the
> > > jets3t package to write to s3?
> > > Ananth T Sarathy
> > >
> > >
> > > On Mon, Oct 26, 2009 at 12:25 PM, Jonathan Gray <jlist@...>
> > wrote:
> > >
> > > > Not S3, HDFS.  Can you checkout the web ui or using the command-line
> > > > interface?
> > > >
> > > > $HADOOP_HOME/bin/hadoop dfs -lsr /hbase
> > > >
> > > > ...would be a good start
> > > >
> > > >
> > > > Ananth T. Sarathy wrote:
> > > >
> > > >> i see all my blocks in my s3 bucket.
> > > >> Ananth T Sarathy
> > > >>
> > > >>
> > > >> On Mon, Oct 26, 2009 at 12:17 PM, Jonathan Gray <jlist@...>
> > > >> wrote:
> > > >>
> > > >>  Do you see the files/blocks in HDFS?
> > > >>>
> > > >>>
> > > >>> Ananth T. Sarathy wrote:
> > > >>>
> > > >>>  I just restarted Hbase and when I go into the shell and type list,
> > > none
> > > >>>> of
> > > >>>> my tables are listed, but I see all the data/blocks in s3.
> > > >>>>
> > > >>>> here is the master log when it's restarted
> > > >>>>
> > > >>>> http://pastebin.com/m1ebb7217
> > > >>>>
> > > >>>> this happened once before, but we just started over since it was
> > early
> > > >>>> in
> > > >>>> our process. This time we have a lot of data, and need to keep it.
> > > >>>>
> > > >>>>
> > > >>>> Ananth T Sarathy
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>
> > >
> >
>

[Ananth Sarathy]

<property>
    <name>hbase.rootdir</name>
    <value>s3://hbase20:80/hbasedata</value>
    <description>The directory shared by region servers.
    Should be fully-qualified to include the filesystem to use.
    E.g: hdfs://NAMENODE_SERVER:PORT/HBASE_ROOTDIR
    </description>
  </property>

Ananth T Sarathy


On Mon, Oct 26, 2009 at 12:57 PM, Jonathan Gray <jlist@...> wrote:

> I think it's only in 0.20.1 and above (might even only be in 0.20 branch).
>
> I'm a bit confused about what you're doing.  Reading the conf file from S3?
>
> What is the configuration value for "hbase.rootdir"
>
>
> Ananth T. Sarathy wrote:
>
>> I don't have a loadtable.rb
>>
>> is that in 0.20.0?
>> This is what i have in bin
>>
>> Formatter.rb   hbase-config.sh   regionservers.sh  zookeepers.sh
>> HBase.rb       hbase-daemon.sh   rename_table.rb
>> copy_table.rb  hbase-daemons.sh  start-hbase.sh
>> hbase          hirb.rb           stop-hbase.sh
>>
>> Ananth T Sarathy
>>
>>
>> On Mon, Oct 26, 2009 at 12:45 PM, stack <stack@...> wrote:
>>
>>  Your log would seem to say that there are no tables in hbase:
>>>
>>> 2009-10-26 11:40:13,984 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> RegionManager.metaScanner scan of 0 row(s) of meta region {server:
>>> 10.245.82.160:60020, regionname: .META.,,1, startKey: <>} complete
>>>
>>> Do as Jon suggests.  Do you see listing of regions?  If so, it would seem
>>> that edits to .META. table are not persisting on your s3 hdfs.   You
>>> might
>>> be able to add in all tables using the bin/loadtable.rb script; it reads
>>> the
>>> .regioninfo files in all regions and per region adds to .META. an entry.
>>>
>>> St.Ack
>>>
>>> On Mon, Oct 26, 2009 at 9:31 AM, Ananth T. Sarathy <
>>> ananth.t.sarathy@...> wrote:
>>>
>>>  I am confused , why would I need a hadoop home if I am using s3 and the
>>>> jets3t package to write to s3?
>>>> Ananth T Sarathy
>>>>
>>>>
>>>> On Mon, Oct 26, 2009 at 12:25 PM, Jonathan Gray <jlist@...>
>>>>
>>> wrote:
>>>
>>>> Not S3, HDFS.  Can you checkout the web ui or using the command-line
>>>>> interface?
>>>>>
>>>>> $HADOOP_HOME/bin/hadoop dfs -lsr /hbase
>>>>>
>>>>> ...would be a good start
>>>>>
>>>>>
>>>>> Ananth T. Sarathy wrote:
>>>>>
>>>>>  i see all my blocks in my s3 bucket.
>>>>>> Ananth T Sarathy
>>>>>>
>>>>>>
>>>>>> On Mon, Oct 26, 2009 at 12:17 PM, Jonathan Gray <jlist@...>
>>>>>> wrote:
>>>>>>
>>>>>>  Do you see the files/blocks in HDFS?
>>>>>>
>>>>>>>
>>>>>>> Ananth T. Sarathy wrote:
>>>>>>>
>>>>>>>  I just restarted Hbase and when I go into the shell and type list,
>>>>>>>
>>>>>> none
>>>>
>>>>> of
>>>>>>>> my tables are listed, but I see all the data/blocks in s3.
>>>>>>>>
>>>>>>>> here is the master log when it's restarted
>>>>>>>>
>>>>>>>> http://pastebin.com/m1ebb7217
>>>>>>>>
>>>>>>>> this happened once before, but we just started over since it was
>>>>>>>>
>>>>>>> early
>>>
>>>> in
>>>>>>>> our process. This time we have a lot of data, and need to keep it.
>>>>>>>>
>>>>>>>>
>>>>>>>> Ananth T Sarathy
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>

[Ananth Sarathy]

@ the command line? I get a command not found.
Ananth T Sarathy


On Mon, Oct 26, 2009 at 1:02 PM, stack <stack@...> wrote:

> Its in head of the 0.20 branch (We should roll a 0.20.2 soon).
>
> What happens if you do a lsr /hbase?
>
> St.Ack
>
> On Mon, Oct 26, 2009 at 9:52 AM, Ananth T. Sarathy <
> ananth.t.sarathy@...> wrote:
>
> > I don't have a loadtable.rb
> >
> > is that in 0.20.0?
> > This is what i have in bin
> >
> > Formatter.rb   hbase-config.sh   regionservers.sh  zookeepers.sh
> > HBase.rb       hbase-daemon.sh   rename_table.rb
> > copy_table.rb  hbase-daemons.sh  start-hbase.sh
> > hbase          hirb.rb           stop-hbase.sh
> >
> > Ananth T Sarathy
> >
> >
> > On Mon, Oct 26, 2009 at 12:45 PM, stack <stack@...> wrote:
> >
> > > Your log would seem to say that there are no tables in hbase:
> > >
> > > 2009-10-26 11:40:13,984 INFO
> org.apache.hadoop.hbase.master.BaseScanner:
> > > RegionManager.metaScanner scan of 0 row(s) of meta region {server:
> > > 10.245.82.160:60020, regionname: .META.,,1, startKey: <>} complete
> > >
> > > Do as Jon suggests.  Do you see listing of regions?  If so, it would
> seem
> > > that edits to .META. table are not persisting on your s3 hdfs.   You
> > might
> > > be able to add in all tables using the bin/loadtable.rb script; it
> reads
> > > the
> > > .regioninfo files in all regions and per region adds to .META. an
> entry.
> > >
> > > St.Ack
> > >
> > > On Mon, Oct 26, 2009 at 9:31 AM, Ananth T. Sarathy <
> > > ananth.t.sarathy@...> wrote:
> > >
> > > > I am confused , why would I need a hadoop home if I am using s3 and
> the
> > > > jets3t package to write to s3?
> > > > Ananth T Sarathy
> > > >
> > > >
> > > > On Mon, Oct 26, 2009 at 12:25 PM, Jonathan Gray <jlist@...>
> > > wrote:
> > > >
> > > > > Not S3, HDFS.  Can you checkout the web ui or using the
> command-line
> > > > > interface?
> > > > >
> > > > > $HADOOP_HOME/bin/hadoop dfs -lsr /hbase
> > > > >
> > > > > ...would be a good start
> > > > >
> > > > >
> > > > > Ananth T. Sarathy wrote:
> > > > >
> > > > >> i see all my blocks in my s3 bucket.
> > > > >> Ananth T Sarathy
> > > > >>
> > > > >>
> > > > >> On Mon, Oct 26, 2009 at 12:17 PM, Jonathan Gray <
> jlist@...>
> > > > >> wrote:
> > > > >>
> > > > >>  Do you see the files/blocks in HDFS?
> > > > >>>
> > > > >>>
> > > > >>> Ananth T. Sarathy wrote:
> > > > >>>
> > > > >>>  I just restarted Hbase and when I go into the shell and type
> list,
> > > > none
> > > > >>>> of
> > > > >>>> my tables are listed, but I see all the data/blocks in s3.
> > > > >>>>
> > > > >>>> here is the master log when it's restarted
> > > > >>>>
> > > > >>>> http://pastebin.com/m1ebb7217
> > > > >>>>
> > > > >>>> this happened once before, but we just started over since it was
> > > early
> > > > >>>> in
> > > > >>>> our process. This time we have a lot of data, and need to keep
> it.
> > > > >>>>
> > > > >>>>
> > > > >>>> Ananth T Sarathy
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>
> > > >
> > >
> >
>

[Jonathan Gray-2]

We really can't help much if you're using S3 directly w/o HDFS.

That's not a supported configuration afaik.

Ananth T. Sarathy wrote:

> @ the command line? I get a command not found.
> Ananth T Sarathy
>
>
> On Mon, Oct 26, 2009 at 1:02 PM, stack <stack@...> wrote:
>
>> Its in head of the 0.20 branch (We should roll a 0.20.2 soon).
>>
>> What happens if you do a lsr /hbase?
>>
>> St.Ack
>>
>> On Mon, Oct 26, 2009 at 9:52 AM, Ananth T. Sarathy <
>> ananth.t.sarathy@...> wrote:
>>
>>> I don't have a loadtable.rb
>>>
>>> is that in 0.20.0?
>>> This is what i have in bin
>>>
>>> Formatter.rb   hbase-config.sh   regionservers.sh  zookeepers.sh
>>> HBase.rb       hbase-daemon.sh   rename_table.rb
>>> copy_table.rb  hbase-daemons.sh  start-hbase.sh
>>> hbase          hirb.rb           stop-hbase.sh
>>>
>>> Ananth T Sarathy
>>>
>>>
>>> On Mon, Oct 26, 2009 at 12:45 PM, stack <stack@...> wrote:
>>>
>>>> Your log would seem to say that there are no tables in hbase:
>>>>
>>>> 2009-10-26 11:40:13,984 INFO
>> org.apache.hadoop.hbase.master.BaseScanner:
>>>> RegionManager.metaScanner scan of 0 row(s) of meta region {server:
>>>> 10.245.82.160:60020, regionname: .META.,,1, startKey: <>} complete
>>>>
>>>> Do as Jon suggests.  Do you see listing of regions?  If so, it would
>> seem
>>>> that edits to .META. table are not persisting on your s3 hdfs.   You
>>> might
>>>> be able to add in all tables using the bin/loadtable.rb script; it
>> reads
>>>> the
>>>> .regioninfo files in all regions and per region adds to .META. an
>> entry.
>>>> St.Ack
>>>>
>>>> On Mon, Oct 26, 2009 at 9:31 AM, Ananth T. Sarathy <
>>>> ananth.t.sarathy@...> wrote:
>>>>
>>>>> I am confused , why would I need a hadoop home if I am using s3 and
>> the
>>>>> jets3t package to write to s3?
>>>>> Ananth T Sarathy
>>>>>
>>>>>
>>>>> On Mon, Oct 26, 2009 at 12:25 PM, Jonathan Gray <jlist@...>
>>>> wrote:
>>>>>> Not S3, HDFS.  Can you checkout the web ui or using the
>> command-line
>>>>>> interface?
>>>>>>
>>>>>> $HADOOP_HOME/bin/hadoop dfs -lsr /hbase
>>>>>>
>>>>>> ...would be a good start
>>>>>>
>>>>>>
>>>>>> Ananth T. Sarathy wrote:
>>>>>>
>>>>>>> i see all my blocks in my s3 bucket.
>>>>>>> Ananth T Sarathy
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Oct 26, 2009 at 12:17 PM, Jonathan Gray <
>> jlist@...>
>>>>>>> wrote:
>>>>>>>
>>>>>>>  Do you see the files/blocks in HDFS?
>>>>>>>>
>>>>>>>> Ananth T. Sarathy wrote:
>>>>>>>>
>>>>>>>>  I just restarted Hbase and when I go into the shell and type
>> list,
>>>>> none
>>>>>>>>> of
>>>>>>>>> my tables are listed, but I see all the data/blocks in s3.
>>>>>>>>>
>>>>>>>>> here is the master log when it's restarted
>>>>>>>>>
>>>>>>>>> http://pastebin.com/m1ebb7217
>>>>>>>>>
>>>>>>>>> this happened once before, but we just started over since it was
>>>> early
>>>>>>>>> in
>>>>>>>>> our process. This time we have a lot of data, and need to keep
>> it.
>>>>>>>>>
>>>>>>>>> Ananth T Sarathy
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>

[stack-3]

When you do bin/hadoop fs -lsr /hbase.rootdir what happens?
St.Ack

On Mon, Oct 26, 2009 at 10:05 AM, Ananth T. Sarathy <
ananth.t.sarathy@...> wrote:

> @ the command line? I get a command not found.
> Ananth T Sarathy
>
>
> On Mon, Oct 26, 2009 at 1:02 PM, stack <stack@...> wrote:
>
> > Its in head of the 0.20 branch (We should roll a 0.20.2 soon).
> >
> > What happens if you do a lsr /hbase?
> >
> > St.Ack
> >
> > On Mon, Oct 26, 2009 at 9:52 AM, Ananth T. Sarathy <
> > ananth.t.sarathy@...> wrote:
> >
> > > I don't have a loadtable.rb
> > >
> > > is that in 0.20.0?
> > > This is what i have in bin
> > >
> > > Formatter.rb   hbase-config.sh   regionservers.sh  zookeepers.sh
> > > HBase.rb       hbase-daemon.sh   rename_table.rb
> > > copy_table.rb  hbase-daemons.sh  start-hbase.sh
> > > hbase          hirb.rb           stop-hbase.sh
> > >
> > > Ananth T Sarathy
> > >
> > >
> > > On Mon, Oct 26, 2009 at 12:45 PM, stack <stack@...> wrote:
> > >
> > > > Your log would seem to say that there are no tables in hbase:
> > > >
> > > > 2009-10-26 11:40:13,984 INFO
> > org.apache.hadoop.hbase.master.BaseScanner:
> > > > RegionManager.metaScanner scan of 0 row(s) of meta region {server:
> > > > 10.245.82.160:60020, regionname: .META.,,1, startKey: <>} complete
> > > >
> > > > Do as Jon suggests.  Do you see listing of regions?  If so, it would
> > seem
> > > > that edits to .META. table are not persisting on your s3 hdfs.   You
> > > might
> > > > be able to add in all tables using the bin/loadtable.rb script; it
> > reads
> > > > the
> > > > .regioninfo files in all regions and per region adds to .META. an
> > entry.
> > > >
> > > > St.Ack
> > > >
> > > > On Mon, Oct 26, 2009 at 9:31 AM, Ananth T. Sarathy <
> > > > ananth.t.sarathy@...> wrote:
> > > >
> > > > > I am confused , why would I need a hadoop home if I am using s3 and
> > the
> > > > > jets3t package to write to s3?
> > > > > Ananth T Sarathy
> > > > >
> > > > >
> > > > > On Mon, Oct 26, 2009 at 12:25 PM, Jonathan Gray <jlist@...
> >
> > > > wrote:
> > > > >
> > > > > > Not S3, HDFS.  Can you checkout the web ui or using the
> > command-line
> > > > > > interface?
> > > > > >
> > > > > > $HADOOP_HOME/bin/hadoop dfs -lsr /hbase
> > > > > >
> > > > > > ...would be a good start
> > > > > >
> > > > > >
> > > > > > Ananth T. Sarathy wrote:
> > > > > >
> > > > > >> i see all my blocks in my s3 bucket.
> > > > > >> Ananth T Sarathy
> > > > > >>
> > > > > >>
> > > > > >> On Mon, Oct 26, 2009 at 12:17 PM, Jonathan Gray <
> > jlist@...>
> > > > > >> wrote:
> > > > > >>
> > > > > >>  Do you see the files/blocks in HDFS?
> > > > > >>>
> > > > > >>>
> > > > > >>> Ananth T. Sarathy wrote:
> > > > > >>>
> > > > > >>>  I just restarted Hbase and when I go into the shell and type
> > list,
> > > > > none
> > > > > >>>> of
> > > > > >>>> my tables are listed, but I see all the data/blocks in s3.
> > > > > >>>>
> > > > > >>>> here is the master log when it's restarted
> > > > > >>>>
> > > > > >>>> http://pastebin.com/m1ebb7217
> > > > > >>>>
> > > > > >>>> this happened once before, but we just started over since it
> was
> > > > early
> > > > > >>>> in
> > > > > >>>> our process. This time we have a lot of data, and need to keep
> > it.
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> Ananth T Sarathy
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
>

[Ananth Sarathy]

well where i have hadoop installed (not running) since not using it for hdfs

 bin/hadoop fs -lsr /hbase.rootdir
lsr: Cannot access /hbase.rootdir: No such file or directory.



Ananth T Sarathy


On Mon, Oct 26, 2009 at 1:24 PM, stack <stack@...> wrote:

> When you do bin/hadoop fs -lsr /hbase.rootdir what happens?
> St.Ack
>
> On Mon, Oct 26, 2009 at 10:05 AM, Ananth T. Sarathy <
> ananth.t.sarathy@...> wrote:
>
> > @ the command line? I get a command not found.
> > Ananth T Sarathy
> >
> >
> > On Mon, Oct 26, 2009 at 1:02 PM, stack <stack@...> wrote:
> >
> > > Its in head of the 0.20 branch (We should roll a 0.20.2 soon).
> > >
> > > What happens if you do a lsr /hbase?
> > >
> > > St.Ack
> > >
> > > On Mon, Oct 26, 2009 at 9:52 AM, Ananth T. Sarathy <
> > > ananth.t.sarathy@...> wrote:
> > >
> > > > I don't have a loadtable.rb
> > > >
> > > > is that in 0.20.0?
> > > > This is what i have in bin
> > > >
> > > > Formatter.rb   hbase-config.sh   regionservers.sh  zookeepers.sh
> > > > HBase.rb       hbase-daemon.sh   rename_table.rb
> > > > copy_table.rb  hbase-daemons.sh  start-hbase.sh
> > > > hbase          hirb.rb           stop-hbase.sh
> > > >
> > > > Ananth T Sarathy
> > > >
> > > >
> > > > On Mon, Oct 26, 2009 at 12:45 PM, stack <stack@...> wrote:
> > > >
> > > > > Your log would seem to say that there are no tables in hbase:
> > > > >
> > > > > 2009-10-26 11:40:13,984 INFO
> > > org.apache.hadoop.hbase.master.BaseScanner:
> > > > > RegionManager.metaScanner scan of 0 row(s) of meta region {server:
> > > > > 10.245.82.160:60020, regionname: .META.,,1, startKey: <>} complete
> > > > >
> > > > > Do as Jon suggests.  Do you see listing of regions?  If so, it
> would
> > > seem
> > > > > that edits to .META. table are not persisting on your s3 hdfs.
> You
> > > > might
> > > > > be able to add in all tables using the bin/loadtable.rb script; it
> > > reads
> > > > > the
> > > > > .regioninfo files in all regions and per region adds to .META. an
> > > entry.
> > > > >
> > > > > St.Ack
> > > > >
> > > > > On Mon, Oct 26, 2009 at 9:31 AM, Ananth T. Sarathy <
> > > > > ananth.t.sarathy@...> wrote:
> > > > >
> > > > > > I am confused , why would I need a hadoop home if I am using s3
> and
> > > the
> > > > > > jets3t package to write to s3?
> > > > > > Ananth T Sarathy
> > > > > >
> > > > > >
> > > > > > On Mon, Oct 26, 2009 at 12:25 PM, Jonathan Gray <
> jlist@...
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Not S3, HDFS.  Can you checkout the web ui or using the
> > > command-line
> > > > > > > interface?
> > > > > > >
> > > > > > > $HADOOP_HOME/bin/hadoop dfs -lsr /hbase
> > > > > > >
> > > > > > > ...would be a good start
> > > > > > >
> > > > > > >
> > > > > > > Ananth T. Sarathy wrote:
> > > > > > >
> > > > > > >> i see all my blocks in my s3 bucket.
> > > > > > >> Ananth T Sarathy
> > > > > > >>
> > > > > > >>
> > > > > > >> On Mon, Oct 26, 2009 at 12:17 PM, Jonathan Gray <
> > > jlist@...>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>  Do you see the files/blocks in HDFS?
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> Ananth T. Sarathy wrote:
> > > > > > >>>
> > > > > > >>>  I just restarted Hbase and when I go into the shell and type
> > > list,
> > > > > > none
> > > > > > >>>> of
> > > > > > >>>> my tables are listed, but I see all the data/blocks in s3.
> > > > > > >>>>
> > > > > > >>>> here is the master log when it's restarted
> > > > > > >>>>
> > > > > > >>>> http://pastebin.com/m1ebb7217
> > > > > > >>>>
> > > > > > >>>> this happened once before, but we just started over since it
> > was
> > > > > early
> > > > > > >>>> in
> > > > > > >>>> our process. This time we have a lot of data, and need to
> keep
> > > it.
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> Ananth T Sarathy
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

[stack-3]

If you list the hbase homedir in s3, do you see tables and then under that
regions?

The loadtable.rb script won't work against a fs that is other than hdfs
unfortunately.
St.Ack

On Mon, Oct 26, 2009 at 10:36 AM, Ananth T. Sarathy <
ananth.t.sarathy@...> wrote:

> well where i have hadoop installed (not running) since not using it for
> hdfs
>
>  bin/hadoop fs -lsr /hbase.rootdir
> lsr: Cannot access /hbase.rootdir: No such file or directory.
>
>
>
> Ananth T Sarathy
>
>
> On Mon, Oct 26, 2009 at 1:24 PM, stack <stack@...> wrote:
>
> > When you do bin/hadoop fs -lsr /hbase.rootdir what happens?
> > St.Ack
> >
> > On Mon, Oct 26, 2009 at 10:05 AM, Ananth T. Sarathy <
> > ananth.t.sarathy@...> wrote:
> >
> > > @ the command line? I get a command not found.
> > > Ananth T Sarathy
> > >
> > >
> > > On Mon, Oct 26, 2009 at 1:02 PM, stack <stack@...> wrote:
> > >
> > > > Its in head of the 0.20 branch (We should roll a 0.20.2 soon).
> > > >
> > > > What happens if you do a lsr /hbase?
> > > >
> > > > St.Ack
> > > >
> > > > On Mon, Oct 26, 2009 at 9:52 AM, Ananth T. Sarathy <
> > > > ananth.t.sarathy@...> wrote:
> > > >
> > > > > I don't have a loadtable.rb
> > > > >
> > > > > is that in 0.20.0?
> > > > > This is what i have in bin
> > > > >
> > > > > Formatter.rb   hbase-config.sh   regionservers.sh  zookeepers.sh
> > > > > HBase.rb       hbase-daemon.sh   rename_table.rb
> > > > > copy_table.rb  hbase-daemons.sh  start-hbase.sh
> > > > > hbase          hirb.rb           stop-hbase.sh
> > > > >
> > > > > Ananth T Sarathy
> > > > >
> > > > >
> > > > > On Mon, Oct 26, 2009 at 12:45 PM, stack <stack@...> wrote:
> > > > >
> > > > > > Your log would seem to say that there are no tables in hbase:
> > > > > >
> > > > > > 2009-10-26 11:40:13,984 INFO
> > > > org.apache.hadoop.hbase.master.BaseScanner:
> > > > > > RegionManager.metaScanner scan of 0 row(s) of meta region
> {server:
> > > > > > 10.245.82.160:60020, regionname: .META.,,1, startKey: <>}
> complete
> > > > > >
> > > > > > Do as Jon suggests.  Do you see listing of regions?  If so, it
> > would
> > > > seem
> > > > > > that edits to .META. table are not persisting on your s3 hdfs.
> > You
> > > > > might
> > > > > > be able to add in all tables using the bin/loadtable.rb script;
> it
> > > > reads
> > > > > > the
> > > > > > .regioninfo files in all regions and per region adds to .META. an
> > > > entry.
> > > > > >
> > > > > > St.Ack
> > > > > >
> > > > > > On Mon, Oct 26, 2009 at 9:31 AM, Ananth T. Sarathy <
> > > > > > ananth.t.sarathy@...> wrote:
> > > > > >
> > > > > > > I am confused , why would I need a hadoop home if I am using s3
> > and
> > > > the
> > > > > > > jets3t package to write to s3?
> > > > > > > Ananth T Sarathy
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Oct 26, 2009 at 12:25 PM, Jonathan Gray <
> > jlist@...
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Not S3, HDFS.  Can you checkout the web ui or using the
> > > > command-line
> > > > > > > > interface?
> > > > > > > >
> > > > > > > > $HADOOP_HOME/bin/hadoop dfs -lsr /hbase
> > > > > > > >
> > > > > > > > ...would be a good start
> > > > > > > >
> > > > > > > >
> > > > > > > > Ananth T. Sarathy wrote:
> > > > > > > >
> > > > > > > >> i see all my blocks in my s3 bucket.
> > > > > > > >> Ananth T Sarathy
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Mon, Oct 26, 2009 at 12:17 PM, Jonathan Gray <
> > > > jlist@...>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >>  Do you see the files/blocks in HDFS?
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> Ananth T. Sarathy wrote:
> > > > > > > >>>
> > > > > > > >>>  I just restarted Hbase and when I go into the shell and
> type
> > > > list,
> > > > > > > none
> > > > > > > >>>> of
> > > > > > > >>>> my tables are listed, but I see all the data/blocks in s3.
> > > > > > > >>>>
> > > > > > > >>>> here is the master log when it's restarted
> > > > > > > >>>>
> > > > > > > >>>> http://pastebin.com/m1ebb7217
> > > > > > > >>>>
> > > > > > > >>>> this happened once before, but we just started over since
> it
> > > was
> > > > > > early
> > > > > > > >>>> in
> > > > > > > >>>> our process. This time we have a lot of data, and need to
> > keep
> > > > it.
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>> Ananth T Sarathy
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Ananth Sarathy]

yeah I  see a folder with my table name. and inside that I see 3 other
folders named with a bunch of numbers (which i assume are the regions).

Ananth T Sarathy


On Mon, Oct 26, 2009 at 1:38 PM, stack <stack@...> wrote:

> If you list the hbase homedir in s3, do you see tables and then under that
> regions?
>
> The loadtable.rb script won't work against a fs that is other than hdfs
> unfortunately.
> St.Ack
>
> On Mon, Oct 26, 2009 at 10:36 AM, Ananth T. Sarathy <
> ananth.t.sarathy@...> wrote:
>
> > well where i have hadoop installed (not running) since not using it for
> > hdfs
> >
> >  bin/hadoop fs -lsr /hbase.rootdir
> > lsr: Cannot access /hbase.rootdir: No such file or directory.
> >
> >
> >
> > Ananth T Sarathy
> >
> >
> > On Mon, Oct 26, 2009 at 1:24 PM, stack <stack@...> wrote:
> >
> > > When you do bin/hadoop fs -lsr /hbase.rootdir what happens?
> > > St.Ack
> > >
> > > On Mon, Oct 26, 2009 at 10:05 AM, Ananth T. Sarathy <
> > > ananth.t.sarathy@...> wrote:
> > >
> > > > @ the command line? I get a command not found.
> > > > Ananth T Sarathy
> > > >
> > > >
> > > > On Mon, Oct 26, 2009 at 1:02 PM, stack <stack@...> wrote:
> > > >
> > > > > Its in head of the 0.20 branch (We should roll a 0.20.2 soon).
> > > > >
> > > > > What happens if you do a lsr /hbase?
> > > > >
> > > > > St.Ack
> > > > >
> > > > > On Mon, Oct 26, 2009 at 9:52 AM, Ananth T. Sarathy <
> > > > > ananth.t.sarathy@...> wrote:
> > > > >
> > > > > > I don't have a loadtable.rb
> > > > > >
> > > > > > is that in 0.20.0?
> > > > > > This is what i have in bin
> > > > > >
> > > > > > Formatter.rb   hbase-config.sh   regionservers.sh  zookeepers.sh
> > > > > > HBase.rb       hbase-daemon.sh   rename_table.rb
> > > > > > copy_table.rb  hbase-daemons.sh  start-hbase.sh
> > > > > > hbase          hirb.rb           stop-hbase.sh
> > > > > >
> > > > > > Ananth T Sarathy
> > > > > >
> > > > > >
> > > > > > On Mon, Oct 26, 2009 at 12:45 PM, stack <stack@...>
> wrote:
> > > > > >
> > > > > > > Your log would seem to say that there are no tables in hbase:
> > > > > > >
> > > > > > > 2009-10-26 11:40:13,984 INFO
> > > > > org.apache.hadoop.hbase.master.BaseScanner:
> > > > > > > RegionManager.metaScanner scan of 0 row(s) of meta region
> > {server:
> > > > > > > 10.245.82.160:60020, regionname: .META.,,1, startKey: <>}
> > complete
> > > > > > >
> > > > > > > Do as Jon suggests.  Do you see listing of regions?  If so, it
> > > would
> > > > > seem
> > > > > > > that edits to .META. table are not persisting on your s3 hdfs.
> > > You
> > > > > > might
> > > > > > > be able to add in all tables using the bin/loadtable.rb script;
> > it
> > > > > reads
> > > > > > > the
> > > > > > > .regioninfo files in all regions and per region adds to .META.
> an
> > > > > entry.
> > > > > > >
> > > > > > > St.Ack
> > > > > > >
> > > > > > > On Mon, Oct 26, 2009 at 9:31 AM, Ananth T. Sarathy <
> > > > > > > ananth.t.sarathy@...> wrote:
> > > > > > >
> > > > > > > > I am confused , why would I need a hadoop home if I am using
> s3
> > > and
> > > > > the
> > > > > > > > jets3t package to write to s3?
> > > > > > > > Ananth T Sarathy
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Oct 26, 2009 at 12:25 PM, Jonathan Gray <
> > > jlist@...
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Not S3, HDFS.  Can you checkout the web ui or using the
> > > > > command-line
> > > > > > > > > interface?
> > > > > > > > >
> > > > > > > > > $HADOOP_HOME/bin/hadoop dfs -lsr /hbase
> > > > > > > > >
> > > > > > > > > ...would be a good start
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Ananth T. Sarathy wrote:
> > > > > > > > >
> > > > > > > > >> i see all my blocks in my s3 bucket.
> > > > > > > > >> Ananth T Sarathy
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Mon, Oct 26, 2009 at 12:17 PM, Jonathan Gray <
> > > > > jlist@...>
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >>  Do you see the files/blocks in HDFS?
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> Ananth T. Sarathy wrote:
> > > > > > > > >>>
> > > > > > > > >>>  I just restarted Hbase and when I go into the shell and
> > type
> > > > > list,
> > > > > > > > none
> > > > > > > > >>>> of
> > > > > > > > >>>> my tables are listed, but I see all the data/blocks in
> s3.
> > > > > > > > >>>>
> > > > > > > > >>>> here is the master log when it's restarted
> > > > > > > > >>>>
> > > > > > > > >>>> http://pastebin.com/m1ebb7217
> > > > > > > > >>>>
> > > > > > > > >>>> this happened once before, but we just started over
> since
> > it
> > > > was
> > > > > > > early
> > > > > > > > >>>> in
> > > > > > > > >>>> our process. This time we have a lot of data, and need
> to
> > > keep
> > > > > it.
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> Ananth T Sarathy
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Tatsuya Kawano]

I'm very confused, but since you're using "s3://..." URL in your
hbase-site.xml, you're still using HDFS **client** library along with
jets3t to access S3, aren't you?

So please try:
bin/hadoop fs -lsr s3://hbase20:80/hbasedata


--
Tatsuya Kawano (Mr.)
Tokyo, Japan


On Tue, Oct 27, 2009 at 2:36 AM, Ananth T. Sarathy
<ananth.t.sarathy@...> wrote:

> well where i have hadoop installed (not running) since not using it for hdfs
>
>  bin/hadoop fs -lsr /hbase.rootdir
> lsr: Cannot access /hbase.rootdir: No such file or directory.
>
>
>
> Ananth T Sarathy
>
>
> On Mon, Oct 26, 2009 at 1:24 PM, stack <stack@...> wrote:
>
>> When you do bin/hadoop fs -lsr /hbase.rootdir what happens?
>> St.Ack

[Ananth Sarathy]

So should I just set up hadoop to use the same bucket and then point my
hbase to hadoop?  Would it then be able to read the data in s3?


Ananth T Sarathy


On Mon, Oct 26, 2009 at 1:42 PM, Ananth T. Sarathy <
ananth.t.sarathy@...> wrote:

> yeah I  see a folder with my table name. and inside that I see 3 other
> folders named with a bunch of numbers (which i assume are the regions).
>
> Ananth T Sarathy
>
>
>
> On Mon, Oct 26, 2009 at 1:38 PM, stack <stack@...> wrote:
>
>> If you list the hbase homedir in s3, do you see tables and then under that
>> regions?
>>
>> The loadtable.rb script won't work against a fs that is other than hdfs
>> unfortunately.
>> St.Ack
>>
>> On Mon, Oct 26, 2009 at 10:36 AM, Ananth T. Sarathy <
>> ananth.t.sarathy@...> wrote:
>>
>> > well where i have hadoop installed (not running) since not using it for
>> > hdfs
>> >
>> >  bin/hadoop fs -lsr /hbase.rootdir
>> > lsr: Cannot access /hbase.rootdir: No such file or directory.
>> >
>> >
>> >
>> > Ananth T Sarathy
>> >
>> >
>> > On Mon, Oct 26, 2009 at 1:24 PM, stack <stack@...> wrote:
>> >
>> > > When you do bin/hadoop fs -lsr /hbase.rootdir what happens?
>> > > St.Ack
>> > >
>> > > On Mon, Oct 26, 2009 at 10:05 AM, Ananth T. Sarathy <
>> > > ananth.t.sarathy@...> wrote:
>> > >
>> > > > @ the command line? I get a command not found.
>> > > > Ananth T Sarathy
>> > > >
>> > > >
>> > > > On Mon, Oct 26, 2009 at 1:02 PM, stack <stack@...> wrote:
>> > > >
>> > > > > Its in head of the 0.20 branch (We should roll a 0.20.2 soon).
>> > > > >
>> > > > > What happens if you do a lsr /hbase?
>> > > > >
>> > > > > St.Ack
>> > > > >
>> > > > > On Mon, Oct 26, 2009 at 9:52 AM, Ananth T. Sarathy <
>> > > > > ananth.t.sarathy@...> wrote:
>> > > > >
>> > > > > > I don't have a loadtable.rb
>> > > > > >
>> > > > > > is that in 0.20.0?
>> > > > > > This is what i have in bin
>> > > > > >
>> > > > > > Formatter.rb   hbase-config.sh   regionservers.sh  zookeepers.sh
>> > > > > > HBase.rb       hbase-daemon.sh   rename_table.rb
>> > > > > > copy_table.rb  hbase-daemons.sh  start-hbase.sh
>> > > > > > hbase          hirb.rb           stop-hbase.sh
>> > > > > >
>> > > > > > Ananth T Sarathy
>> > > > > >
>> > > > > >
>> > > > > > On Mon, Oct 26, 2009 at 12:45 PM, stack <stack@...>
>> wrote:
>> > > > > >
>> > > > > > > Your log would seem to say that there are no tables in hbase:
>> > > > > > >
>> > > > > > > 2009-10-26 11:40:13,984 INFO
>> > > > > org.apache.hadoop.hbase.master.BaseScanner:
>> > > > > > > RegionManager.metaScanner scan of 0 row(s) of meta region
>> > {server:
>> > > > > > > 10.245.82.160:60020, regionname: .META.,,1, startKey: <>}
>> > complete
>> > > > > > >
>> > > > > > > Do as Jon suggests.  Do you see listing of regions?  If so, it
>> > > would
>> > > > > seem
>> > > > > > > that edits to .META. table are not persisting on your s3 hdfs.
>> > > You
>> > > > > > might
>> > > > > > > be able to add in all tables using the bin/loadtable.rb
>> script;
>> > it
>> > > > > reads
>> > > > > > > the
>> > > > > > > .regioninfo files in all regions and per region adds to .META.
>> an
>> > > > > entry.
>> > > > > > >
>> > > > > > > St.Ack
>> > > > > > >
>> > > > > > > On Mon, Oct 26, 2009 at 9:31 AM, Ananth T. Sarathy <
>> > > > > > > ananth.t.sarathy@...> wrote:
>> > > > > > >
>> > > > > > > > I am confused , why would I need a hadoop home if I am using
>> s3
>> > > and
>> > > > > the
>> > > > > > > > jets3t package to write to s3?
>> > > > > > > > Ananth T Sarathy
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Mon, Oct 26, 2009 at 12:25 PM, Jonathan Gray <
>> > > jlist@...
>> > > > >
>> > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Not S3, HDFS.  Can you checkout the web ui or using the
>> > > > > command-line
>> > > > > > > > > interface?
>> > > > > > > > >
>> > > > > > > > > $HADOOP_HOME/bin/hadoop dfs -lsr /hbase
>> > > > > > > > >
>> > > > > > > > > ...would be a good start
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > Ananth T. Sarathy wrote:
>> > > > > > > > >
>> > > > > > > > >> i see all my blocks in my s3 bucket.
>> > > > > > > > >> Ananth T Sarathy
>> > > > > > > > >>
>> > > > > > > > >>
>> > > > > > > > >> On Mon, Oct 26, 2009 at 12:17 PM, Jonathan Gray <
>> > > > > jlist@...>
>> > > > > > > > >> wrote:
>> > > > > > > > >>
>> > > > > > > > >>  Do you see the files/blocks in HDFS?
>> > > > > > > > >>>
>> > > > > > > > >>>
>> > > > > > > > >>> Ananth T. Sarathy wrote:
>> > > > > > > > >>>
>> > > > > > > > >>>  I just restarted Hbase and when I go into the shell and
>> > type
>> > > > > list,
>> > > > > > > > none
>> > > > > > > > >>>> of
>> > > > > > > > >>>> my tables are listed, but I see all the data/blocks in
>> s3.
>> > > > > > > > >>>>
>> > > > > > > > >>>> here is the master log when it's restarted
>> > > > > > > > >>>>
>> > > > > > > > >>>> http://pastebin.com/m1ebb7217
>> > > > > > > > >>>>
>> > > > > > > > >>>> this happened once before, but we just started over
>> since
>> > it
>> > > > was
>> > > > > > > early
>> > > > > > > > >>>> in
>> > > > > > > > >>>> our process. This time we have a lot of data, and need
>> to
>> > > keep
>> > > > > it.
>> > > > > > > > >>>>
>> > > > > > > > >>>>
>> > > > > > > > >>>> Ananth T Sarathy
>> > > > > > > > >>>>
>> > > > > > > > >>>>
>> > > > > > > > >>>>
>> > > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>