[jira] Created: (HADOOP-2668) [hbase] After 2643, cluster won't start if FS was created by an older hbase version

[hbase] After 2643, cluster won't start if FS was created by an older hbase version
-----------------------------------------------------------------------------------

                 Key: HADOOP-2668
                 URL: https://issues.apache.org/jira/browse/HADOOP-2668
             Project: Hadoop
          Issue Type: Bug
          Components: contrib/hbase
            Reporter: stack
            Priority: Blocker
             Fix For: 0.16.0


Hbase now checks for a version file.  If none, it reports a version mismatch.  There will be no version file if the hbase was made by a version older than r613469

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HADOOP-2668:
--------------------------

    Attachment: migrate.patch

Changed the checkVersion to throw an FNFE if version file is not present.  Up in the master, catch any FNFE and create the version file (logging fact at WARN level).  Also changed version from 0.1 to 1.  1 will sit better as prefix for migration scripts and integers are how RPC and classes are versioned.

> [hbase] After 2643, cluster won't start if FS was created by an older hbase version
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-2668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2668
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>            Reporter: stack
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: migrate.patch
>
>
> Hbase now checks for a version file.  If none, it reports a version mismatch.  There will be no version file if the hbase was made by a version older than r613469

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560748#action_12560748 ]

Jim Kellerman commented on HADOOP-2668:
---------------------------------------

If you run the migrate tool as the exception suggested, it will write the version file and then the system will start.

> [hbase] After 2643, cluster won't start if FS was created by an older hbase version
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-2668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2668
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>            Reporter: stack
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: migrate.patch
>
>
> Hbase now checks for a version file.  If none, it reports a version mismatch.  There will be no version file if the hbase was made by a version older than r613469

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560752#action_12560752 ]

stack commented on HADOOP-2668:
-------------------------------

It didn't occur to me that migration was the way to fix the missing version file.  I also figured we should just auto-migrate this one case of a missing version file (If in future, version file goes missing, I'd think it the job of hbsfck recreating it, rather than migration?).

But I'm fine w/ forcing users to run the migration.  It needs to be better documented and added to the bin/hbase script with verb 'migrate' I'd say.

I tried to run the migration but it wants to connect to a HMaster.  That ain't going to work (Cluster won't start because no version file... can't migrate because cluster ain't up...).

> [hbase] After 2643, cluster won't start if FS was created by an older hbase version
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-2668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2668
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>            Reporter: stack
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: migrate.patch
>
>
> Hbase now checks for a version file.  If none, it reports a version mismatch.  There will be no version file if the hbase was made by a version older than r613469

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack reassigned HADOOP-2668:
-----------------------------

    Assignee: stack

> [hbase] After 2643, cluster won't start if FS was created by an older hbase version
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-2668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2668
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>            Reporter: stack
>            Assignee: stack
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: migrate.patch
>
>
> Hbase now checks for a version file.  If none, it reports a version mismatch.  There will be no version file if the hbase was made by a version older than r613469

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560753#action_12560753 ]

Jim Kellerman commented on HADOOP-2668:
---------------------------------------

> It didn't occur to me that migration was the way to fix the missing version file.

From HMaster.java(894, 5):

{code}
        throw new IOException(
            "file system not correct version. Run hbase.util.Migrate");
{code}

> I also figured we should just auto-migrate this one case of a missing version file (If in future,
> version file goes missing, I'd think it the job of hbsfck recreating it, rather than migration?).

Suppose you have a file system that has not been migrated? (i.e. regions are stored in
/hbase/hregion_nnnnnnn) The master would start up write the version file and then
proceed to recreate the root and meta regions because they aren't under
/hbase/-ROOT- and /hbase/.META. respectively.

Additionally the first thing the migrate tool does is look for the version file. If it finds it and
the version number matches, it figures that the file system has been upgraded already
and does nothing.

> But I'm fine w/ forcing users to run the migration. It needs to be better documented and added
> to the bin/hbase script with verb 'migrate' I'd say.

Agreed. How about this changing this patch to update bin/hbase and add documentation
(where ?)?

> I tried to run the migration but it wants to connect to a HMaster. That ain't going to work (Cluster
>  won't start because no version file... can't migrate because cluster ain't up...).

It tries to connect to the master to ensure it isn't running (uses HBaseAdmin.isMasterRunning())
We wouldn't want to do a upgrade with the cluster running.

> [hbase] After 2643, cluster won't start if FS was created by an older hbase version
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-2668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2668
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>            Reporter: stack
>            Assignee: stack
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: migrate.patch
>
>
> Hbase now checks for a version file.  If none, it reports a version mismatch.  There will be no version file if the hbase was made by a version older than r613469

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560753#action_12560753 ]

jimk edited comment on HADOOP-2668 at 1/19/08 5:05 PM:
----------------------------------------------------------------

> It didn't occur to me that migration was the way to fix the missing version file.

From HMaster.java(894, 5):

{code}
        throw new IOException(
            "file system not correct version. Run hbase.util.Migrate");
{code}

> I also figured we should just auto-migrate this one case of a missing version file (If in future,
> version file goes missing, I'd think it the job of hbsfck recreating it, rather than migration?).

Suppose you have a file system that has not been migrated? (i.e. regions are stored in
=/hbase/hregion_nnnnnnn=) The master would start up write the version file and then
proceed to recreate the root and meta regions because they aren't under
=/hbase/-ROOT-= and =/hbase/.META.= respectively.

Additionally the first thing the migrate tool does is look for the version file. If it finds it and
the version number matches, it figures that the file system has been upgraded already
and does nothing.

> But I'm fine w/ forcing users to run the migration. It needs to be better documented and added
> to the bin/hbase script with verb 'migrate' I'd say.

Agreed. How about this changing this patch to update bin/hbase and add documentation
(where ?)?

> I tried to run the migration but it wants to connect to a HMaster. That ain't going to work (Cluster
>  won't start because no version file... can't migrate because cluster ain't up...).

It tries to connect to the master to ensure it isn't running (uses HBaseAdmin.isMasterRunning())
We wouldn't want to do a upgrade with the cluster running.


      was (Author: jimk):
    > It didn't occur to me that migration was the way to fix the missing version file.

From HMaster.java(894, 5):

{code}
        throw new IOException(
            "file system not correct version. Run hbase.util.Migrate");
{code}

> I also figured we should just auto-migrate this one case of a missing version file (If in future,
> version file goes missing, I'd think it the job of hbsfck recreating it, rather than migration?).

Suppose you have a file system that has not been migrated? (i.e. regions are stored in
/hbase/hregion_nnnnnnn) The master would start up write the version file and then
proceed to recreate the root and meta regions because they aren't under
/hbase/-ROOT- and /hbase/.META. respectively.

Additionally the first thing the migrate tool does is look for the version file. If it finds it and
the version number matches, it figures that the file system has been upgraded already
and does nothing.

> But I'm fine w/ forcing users to run the migration. It needs to be better documented and added
> to the bin/hbase script with verb 'migrate' I'd say.

Agreed. How about this changing this patch to update bin/hbase and add documentation
(where ?)?

> I tried to run the migration but it wants to connect to a HMaster. That ain't going to work (Cluster
>  won't start because no version file... can't migrate because cluster ain't up...).

It tries to connect to the master to ensure it isn't running (uses HBaseAdmin.isMasterRunning())
We wouldn't want to do a upgrade with the cluster running.

 

> [hbase] After 2643, cluster won't start if FS was created by an older hbase version
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-2668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2668
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>            Reporter: stack
>            Assignee: stack
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: migrate.patch
>
>
> Hbase now checks for a version file.  If none, it reports a version mismatch.  There will be no version file if the hbase was made by a version older than r613469

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560753#action_12560753 ]

jimk edited comment on HADOOP-2668 at 1/19/08 5:06 PM:
----------------------------------------------------------------

> It didn't occur to me that migration was the way to fix the missing version file.

From HMaster.java(894, 5):

{code}
        throw new IOException(
            "file system not correct version. Run hbase.util.Migrate");
{code}

> I also figured we should just auto-migrate this one case of a missing version file (If in future,
> version file goes missing, I'd think it the job of hbsfck recreating it, rather than migration?).

Suppose you have a file system that has not been migrated? (i.e. regions are stored in
{code}/hbase/hregion_nnnnnnn{code}) The master would start up write the version file and then
proceed to recreate the root and meta regions because they aren't under
{code}/hbase/-ROOT-{code} and {code}/hbase/.META.{code} respectively.

Additionally the first thing the migrate tool does is look for the version file. If it finds it and
the version number matches, it figures that the file system has been upgraded already
and does nothing.

> But I'm fine w/ forcing users to run the migration. It needs to be better documented and added
> to the bin/hbase script with verb 'migrate' I'd say.

Agreed. How about this changing this patch to update bin/hbase and add documentation
(where ?)?

> I tried to run the migration but it wants to connect to a HMaster. That ain't going to work (Cluster
>  won't start because no version file... can't migrate because cluster ain't up...).

It tries to connect to the master to ensure it isn't running (uses HBaseAdmin.isMasterRunning())
We wouldn't want to do a upgrade with the cluster running.


      was (Author: jimk):
    > It didn't occur to me that migration was the way to fix the missing version file.

From HMaster.java(894, 5):

{code}
        throw new IOException(
            "file system not correct version. Run hbase.util.Migrate");
{code}

> I also figured we should just auto-migrate this one case of a missing version file (If in future,
> version file goes missing, I'd think it the job of hbsfck recreating it, rather than migration?).

Suppose you have a file system that has not been migrated? (i.e. regions are stored in
=/hbase/hregion_nnnnnnn=) The master would start up write the version file and then
proceed to recreate the root and meta regions because they aren't under
=/hbase/-ROOT-= and =/hbase/.META.= respectively.

Additionally the first thing the migrate tool does is look for the version file. If it finds it and
the version number matches, it figures that the file system has been upgraded already
and does nothing.

> But I'm fine w/ forcing users to run the migration. It needs to be better documented and added
> to the bin/hbase script with verb 'migrate' I'd say.

Agreed. How about this changing this patch to update bin/hbase and add documentation
(where ?)?

> I tried to run the migration but it wants to connect to a HMaster. That ain't going to work (Cluster
>  won't start because no version file... can't migrate because cluster ain't up...).

It tries to connect to the master to ensure it isn't running (uses HBaseAdmin.isMasterRunning())
We wouldn't want to do a upgrade with the cluster running.

 

> [hbase] After 2643, cluster won't start if FS was created by an older hbase version
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-2668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2668
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>            Reporter: stack
>            Assignee: stack
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: migrate.patch
>
>
> Hbase now checks for a version file.  If none, it reports a version mismatch.  There will be no version file if the hbase was made by a version older than r613469

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HADOOP-2668:
--------------------------

    Summary: [hbase] Documentation and improved logging so fact that hbase now requires migration comes as less of a surprise  (was: [hbase] After 2643, cluster won't start if FS was created by an older hbase version)

I should have said, in spite of the exception, I did not get that I was supposed to run the migration.  The thing just looks broke.  My guess is that users will think the same.

Agreed, to move past r613469, running migration is needed and yes, lets make this issue be about making the migration easier to find and run -- add to bin/hbase -- and about documentation ('Getting Started' doc., incompatible changes in the release notes, IRC, etc.).

Would help if there was a prefatory message that script is verifying the cluster is down.

> [hbase] Documentation and improved logging so fact that hbase now requires migration comes as less of a surprise
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-2668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2668
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>            Reporter: stack
>            Assignee: stack
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: migrate.patch
>
>
> Hbase now checks for a version file.  If none, it reports a version mismatch.  There will be no version file if the hbase was made by a version older than r613469

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560779#action_12560779 ]

stack commented on HADOOP-2668:
-------------------------------

I tried to do a migrate and it looks like the below:

{code}
durruti:~/Documents/checkouts/hadoop-commit/src/contrib/hbase stack$ ./bin/hbase org.apache.hadoop.hbase.util.Migrate
08/01/19 23:05:21 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 1 time(s).
08/01/19 23:05:22 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 2 time(s).
08/01/19 23:05:23 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 3 time(s).
08/01/19 23:05:24 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 4 time(s).
08/01/19 23:05:25 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 5 time(s).
08/01/19 23:05:26 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 6 time(s).
08/01/19 23:05:27 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 7 time(s).
08/01/19 23:05:28 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 8 time(s).
08/01/19 23:05:29 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 9 time(s).
08/01/19 23:05:30 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 10 time(s).
08/01/19 23:05:31 INFO util.Migrate: unrecognized file -ROOT- ignoring
08/01/19 23:05:31 INFO util.Migrate: unrecognized file .hbase.version.crc ignoring
08/01/19 23:05:31 INFO util.Migrate: unrecognized file .META. ignoring
08/01/19 23:05:31 ERROR util.Migrate: exiting due to error
java.io.IOException: unrecovered region server log file log_192.168.0.100_1200373881646_51251 aborting
        at org.apache.hadoop.hbase.util.Migrate.extraFile(Migrate.java:206)
        at org.apache.hadoop.hbase.util.Migrate.extraFiles(Migrate.java:184)
        at org.apache.hadoop.hbase.util.Migrate.run(Migrate.java:148)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
        at org.apache.hadoop.hbase.util.Migrate.main(Migrate.java:450)
{code}

The 'unrecognized' -ROOT- and .META. and *.crc do not inspire confidence.  Can we make the message better?

Looking at the FS, I see the version file and figure that rerunning the start, it'd probably work for me (running the logs).. but it just fails again telling me to migrate.

> [hbase] Documentation and improved logging so fact that hbase now requires migration comes as less of a surprise
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-2668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2668
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>            Reporter: stack
>            Assignee: stack
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: migrate.patch
>
>
> Hbase now checks for a version file.  If none, it reports a version mismatch.  There will be no version file if the hbase was made by a version older than r613469

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560780#action_12560780 ]

stack commented on HADOOP-2668:
-------------------------------

I removed the hbase dir completely.  Restarted.  Came up nice.  Shut it down.  Remove the hbase.version and .hbase.version.crc.  Tried to start it.  It complained no version file.  Run migration.  I did and it made following emissions:

{code}

08/01/19 23:17:56 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 1 time(s).08/01/19 23:17:57 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 2 time(s).08/01/19 23:17:58 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 3 time(s).08/01/19 23:17:59 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 4 time(s).08/01/19 23:18:00 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 5 time(s).08/01/19 23:18:01 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 6 time(s).08/01/19 23:18:02 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 7 time(s).08/01/19 23:18:03 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 8 time(s).08/01/19 23:18:04 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 9 time(s).08/01/19 23:18:05 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 10 time(s).08/01/19 23:18:06 INFO util.Migrate: unrecognized file -ROOT- ignoring08/01/19 23:18:06 INFO util.Migrate: unrecognized file .META. ignoring08/01/19 23:18:06 ERROR util.Migrate: exiting due to errorjava.io.IOException: cannot find root region file:/tmp/hadoop-stack/hbase/hregion_70236052        at org.apache.hadoop.hbase.util.Migrate.run(Migrate.java:156)        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)        at org.apache.hadoop.hbase.util.Migrate.main(Migrate.java:450)

{code}

Odd is that there is not hregion_70236052 file:

{code}
$ ls -la /tmp/hadoop-stack/hbase/
total 0
drwxr-xr-x   3 stack  wheel  102 Jan 19 23:16 -ROOT-
drwxr-xr-x   4 stack  wheel  136 Jan 19 23:17 .
drwxr-xr-x   3 stack  wheel  102 Jan 19 23:16 ..
drwxr-xr-x   3 stack  wheel  102 Jan 19 23:16 .META.
{code}

> [hbase] Documentation and improved logging so fact that hbase now requires migration comes as less of a surprise
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-2668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2668
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>            Reporter: stack
>            Assignee: stack
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: migrate.patch
>
>
> Hbase now checks for a version file.  If none, it reports a version mismatch.  There will be no version file if the hbase was made by a version older than r613469

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HADOOP-2668:
--------------------------

    Attachment: migration.patch

This patch is imcomplete.  Adds migrate to ./bin/hbase.  Adds doc. about necessary migration to 'getting started' under javadoc.  I started in adding to the migration script extra info-level logging talking users through the process so they have a better sense of whats happening but didn't finish the job because need to study more what this script is up to (How about adding description of migration steps to the class comment -- we can fill it out as we add new steps).

The script doesn't have usage/help (-h/--help do nought).  If I supply them, it goes ahead and starts up the migration regardless.  Do you think it should do this?  Should it not first emit flashing neon warnings about how this script rewrites hbase.rootdir and of how it might make sense making a backup first, etc. (Perhaps point at the MR CopyFiles or ./bin/hadoop distcp).

One idea would be that this script first checks to see if any migratory actions are required; if so, user needs to rerun the migration script with an option that says actually go ahead and rearrange hbase.rootdir.

> [hbase] Documentation and improved logging so fact that hbase now requires migration comes as less of a surprise
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-2668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2668
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>            Reporter: stack
>            Assignee: stack
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: migrate.patch, migration.patch
>
>
> Hbase now checks for a version file.  If none, it reports a version mismatch.  There will be no version file if the hbase was made by a version older than r613469

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560893#action_12560893 ]

Jim Kellerman commented on HADOOP-2668:
---------------------------------------

Ok, there is definitely some work to do here. I'll work on fixing Migrate.

> [hbase] Documentation and improved logging so fact that hbase now requires migration comes as less of a surprise
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-2668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2668
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>            Reporter: stack
>            Assignee: Jim Kellerman
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: migrate.patch, migration.patch
>
>
> Hbase now checks for a version file.  If none, it reports a version mismatch.  There will be no version file if the hbase was made by a version older than r613469

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman reassigned HADOOP-2668:
-------------------------------------

    Assignee: Jim Kellerman  (was: stack)

> [hbase] Documentation and improved logging so fact that hbase now requires migration comes as less of a surprise
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-2668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2668
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>            Reporter: stack
>            Assignee: Jim Kellerman
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: migrate.patch, migration.patch
>
>
> Hbase now checks for a version file.  If none, it reports a version mismatch.  There will be no version file if the hbase was made by a version older than r613469

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-2668:
----------------------------------

    Affects Version/s: 0.16.0
               Status: Patch Available  (was: Open)

Works locally, try hudson. - Stack, please review patch.

> [hbase] Documentation and improved logging so fact that hbase now requires migration comes as less of a surprise
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-2668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2668
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.16.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: migrate.patch, migration.patch, patch.txt
>
>
> Hbase now checks for a version file.  If none, it reports a version mismatch.  There will be no version file if the hbase was made by a version older than r613469

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-2668:
----------------------------------

    Attachment: patch.txt

Lots more checking, clean up several bugs, new read-only mode, usage, etc.

> [hbase] Documentation and improved logging so fact that hbase now requires migration comes as less of a surprise
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-2668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2668
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.16.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: migrate.patch, migration.patch, patch.txt
>
>
> Hbase now checks for a version file.  If none, it reports a version mismatch.  There will be no version file if the hbase was made by a version older than r613469

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-2668:
----------------------------------

    Status: Patch Available  (was: Open)

> [hbase] Documentation and improved logging so fact that hbase now requires migration comes as less of a surprise
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-2668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2668
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.16.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: migrate.patch, migration.patch, patch.txt
>
>
> Hbase now checks for a version file.  If none, it reports a version mismatch.  There will be no version file if the hbase was made by a version older than r613469

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-2668:
----------------------------------

    Status: Open  (was: Patch Available)

It appears that hudson lost this patch when it went down. Resubmitting.

> [hbase] Documentation and improved logging so fact that hbase now requires migration comes as less of a surprise
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-2668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2668
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.16.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: migrate.patch, migration.patch, patch.txt
>
>
> Hbase now checks for a version file.  If none, it reports a version mismatch.  There will be no version file if the hbase was made by a version older than r613469

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/HADOOP-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561200#action_12561200 ]

stack commented on HADOOP-2668:
-------------------------------

This patch looks great.

Shouldn't the 'check' emit a result? That is, shouldn't check say "hbase.rootdir is up to date" or "hbase.rootdir needs to be migrated"?  Same for actual migration.  Should say "migration successful" or "migration failed" or "up to date"?

Regards the check for a running cluster, what we've seen internally is that often a newer version of hbase will just timeout trying to connect to a running instance without an exception being thrown or complaint about mismatched RPC version numbers.  There ain't much that the migration script can do in this case.  Add to the info log a more forceful note that hbase cluster must be down when this script runs?

If the HDFS to upgrade is not running, it doesn't recognize it.  See below:

{code}
[stack@aa0-000-12 hbase]$ ./bin/hbase migrate check
08/01/22 02:11:35 INFO util.Migrate: Verifying that HBase is not running...
08/01/22 02:11:35 INFO ipc.Client: Retrying connect to server: XX.XX.XX.139:60000. Already tried 1 time(s).
08/01/22 02:11:36 INFO ipc.Client: Retrying connect to server: XX.XX.XX.139:60000. Already tried 2 time(s).
...
08/01/22 02:11:45 INFO util.Migrate: Starting migration check
08/01/22 02:11:45 INFO ipc.Client: Retrying connect to server: XX.XX.XX.139:9123. Already tried 1 time(s).
....
{code}

I don't think this too important.

> [hbase] Documentation and improved logging so fact that hbase now requires migration comes as less of a surprise
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-2668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2668
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.16.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: migrate.patch, migration.patch, patch.txt
>
>
> Hbase now checks for a version file.  If none, it reports a version mismatch.  There will be no version file if the hbase was made by a version older than r613469

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/HADOOP-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-2668:
----------------------------------

    Status: Open  (was: Patch Available)

Hudson lost this patch. Resubmitting.

> [hbase] Documentation and improved logging so fact that hbase now requires migration comes as less of a surprise
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-2668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2668
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.16.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>            Priority: Blocker
>             Fix For: 0.16.0
>
>         Attachments: migrate.patch, migration.patch, patch.txt
>
>
> Hbase now checks for a version file.  If none, it reports a version mismatch.  There will be no version file if the hbase was made by a version older than r613469

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.