Nabble - Hadoop lucene-dev

HADOOP-6318: upgrade to Avro 1.2.0

2009-10-22T13:44:56Z

Can another committer please review this issue?

https://issues.apache.org/jira/browse/HADOOP-6318

Thanks,

Doug

Thread safety issue in Path.getFileSystem()

2009-06-22T14:04:57Z

I have two hadoop jobs which makes a Path and tries to get the FileSystem using.

Path destFile = new Path("AAA");
FileSystem fs = destFile.getFileSystem(new Configuration());

I am seeing this exception .. I am using hadoop-0.20.0.

java.util.ConcurrentModificationException
at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
at java.util.AbstractList$Itr.next(AbstractList.java:343)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1028)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:979)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:435)
at org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:103)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:180)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:175)

Have folks seen these exceptions before ??

[jira] Commented: (HADOOP-2693) [hbase] NPE in getClosestRowBefore

2008-01-23T15:14:35Z

[ https://issues.apache.org/jira/browse/HADOOP-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561862#action_12561862 ]

stack commented on HADOOP-2693:
-------------------------------

This is what it looks like client-side:
{code}
08/01/23 22:44:13 ERROR hbase.PerformanceEvaluation: Failed
java.io.IOException: java.io.IOException: java.lang.NullPointerException

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
at org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:469)
at org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:343)
at org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:306)
at org.apache.hadoop.hbase.HTable.<init>(HTable.java:90)
at org.apache.hadoop.hbase.PerformanceEvaluation$Test.testSetup(PerformanceEvaluation.java:308)
at org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:323)
at org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:525)
at org.apache.hadoop.hbase.PerformanceEvaluation.runNIsOne(PerformanceEvaluation.java:546)
at org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:568)
at org.apache.hadoop.hbase.PerformanceEvaluation.doCommandLine(PerformanceEvaluation.java:663)
at org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:682)
{code}

> [hbase] NPE in getClosestRowBefore
> ----------------------------------
>
> Key: HADOOP-2693
> URL: https://issues.apache.org/jira/browse/HADOOP-2693
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hbase
> Reporter: stack
>
> Just saw this in a load test:
> {code}
> 2008-01-23 22:43:42,831 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call getClosestRowBefore(.META.,,1, TestTable,,999999999999999, 9223372036854775807) from XX.XX.XX.141:49068: error: java.io.IOException: java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.io.Text.compareTo(Text.java:278)
> at org.apache.hadoop.hbase.HStore.getRowKeyAtOrBefore(HStore.java:1846)
> at org.apache.hadoop.hbase.HRegion.getClosestRowBefore(HRegion.java:1109)
> at org.apache.hadoop.hbase.HRegionServer.getClosestRowBefore(HRegionServer.java:1434)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:908)
> {code}

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (HADOOP-2518) [hbase] Refactor HStore and internal (and related) classes into hbase.store subpackage

2008-01-23T15:14:34Z

[ https://issues.apache.org/jira/browse/HADOOP-2518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman resolved HADOOP-2518.
-----------------------------------

Resolution: Duplicate

> [hbase] Refactor HStore and internal (and related) classes into hbase.store subpackage
> --------------------------------------------------------------------------------------
>
> Key: HADOOP-2518
> URL: https://issues.apache.org/jira/browse/HADOOP-2518
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Attachments: 2518.patch
>
>
> HStore, the class responsible for managing data storage for column families within a region, is a very large class with more than a few internal classes. I propose moving HStore and HStoreFile into the hbase.store subpackage, and then moving internal classes out of HStore and into their own package-internal classes within the subpackage. This will make the code much more readable and manageable. In addition, it will help to logically delineate which classes are involved in managing a store.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HADOOP-2566) need FileSystem#globStatus method

2008-01-23T15:08:39Z

[ https://issues.apache.org/jira/browse/HADOOP-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-2566:
----------------------------------

Attachment: globStatus7.patch

> need FileSystem#globStatus method
> ---------------------------------
>
> Key: HADOOP-2566
> URL: https://issues.apache.org/jira/browse/HADOOP-2566
> Project: Hadoop Core
> Issue Type: Improvement
> Components: fs
> Reporter: Doug Cutting
> Assignee: Hairong Kuang
> Fix For: 0.16.0
>
> Attachments: globStatus.patch, globStatus1.patch, globStatus2.patch, globStatus3.patch, globStatus4.patch, globStatus5.patch, globStatus6.patch, globStatus7.patch
>
>
> To remove the cache of FileStatus in DFSPath (HADOOP-2565) without hurting performance, we must use file enumeration APIs that return FileStatus[] rather than Path[]. Currently we have FileSystem#globPaths(), but that method should be deprecated and replaced with a FileSystem#globStatus().
> We need to deprecate FileSystem#globPaths() in 0.16 in order to remove the cache in 0.17.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HADOOP-2566) need FileSystem#globStatus method

2008-01-23T15:08:37Z

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-2692) [hbase] Client exits with IllegalStateException

2008-01-23T15:04:37Z

[ https://issues.apache.org/jira/browse/HADOOP-2692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561860#action_12561860 ]

stack commented on HADOOP-2692:
-------------------------------

This could be problem w/ server returning ISE when perhaps should return NSRE

> [hbase] Client exits with IllegalStateException
> -----------------------------------------------
>
> Key: HADOOP-2692
> URL: https://issues.apache.org/jira/browse/HADOOP-2692
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hbase
> Reporter: stack
>
> Running a load test, I see clients exiting with the below exception:
> {code}
> 08/01/23 22:12:12 ERROR hbase.PerformanceEvaluation: Failed
> java.lang.IllegalStateException: region offline: TestTable,0000099498,1201125579414
> at org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:432)
> at org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:343)
> at org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:306)
> at org.apache.hadoop.hbase.HTable.getRegionLocation(HTable.java:102)
> at org.apache.hadoop.hbase.HTable.commit(HTable.java:959)
> at org.apache.hadoop.hbase.HTable.commit(HTable.java:939)
> at org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest.testRow(PerformanceEvaluation.java:461)
> at org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:329)
> at org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:525)
> at org.apache.hadoop.hbase.PerformanceEvaluation.runNIsOne(PerformanceEvaluation.java:546)
> at org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:568)
> at org.apache.hadoop.hbase.PerformanceEvaluation.doCommandLine(PerformanceEvaluation.java:663)
> at org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:682)
> {code}
> Looking in code, above is thrown because region is offline. A split is probably going on. Client should retry in this case after a little pause?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-2660) Regions getting messages from master to MSG_REGION_CLOSE_WITHOUT_REPORT

2008-01-23T15:02:39Z

[ https://issues.apache.org/jira/browse/HADOOP-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561859#action_12561859 ]

Jim Kellerman commented on HADOOP-2660:
---------------------------------------

> I thank that of two options that would help solve this problem and might need to use both

> option 1
> build in a backlog limit on how many pending opens we can have in any one region server before stop
> accepting new opens.
> example finding the maximum sequence id for a region takes a lot less time then doing a recovery to a
> region. So its que would fill up faster making the master send some open request to different servers
> while this one catches up or loop until one of the region servers has open slots in it pending open que. I
> thank 60 secs is the default loop time so they should be able to hand 10 pending opens or something
> like that many be make it an option limit in the conf.

> option 2
>
> 1.Confirm we received the masters open request once we received it
>
> Once confirmed master should not reassign the region to any other region server unless the region
> server goes off line and loses it lease

In fact this exactly what happens today. When a region server receives an open region request, it replies
in its next heartbeat to the master with MSG_REPORT_PROCESS_OPEN which means, I got your request
and am working on it. When the master receives this message, it adds
hbase.hbasemaster.maxregionopen (currently 30 seconds) to the amount of time before it will try to
assign the region again. If it is taking longer than 30 seconds for a region server to open a region,
I would suggest increasing the value of this parameter to 60000 (60 seconds).

> 2 Confirm the open of the region success or failed

When the region server has opened the region, it sends a MSG_REPORT_OPEN to the master
meaning that it is now serving the region.

> The master can make sure the region server is still alive by keeping up with heartbeat

It is the region server that sends the heartbeat to the master, but this is exactly what happens.

> Regions getting messages from master to MSG_REGION_CLOSE_WITHOUT_REPORT
> -----------------------------------------------------------------------
>
> Key: HADOOP-2660
> URL: https://issues.apache.org/jira/browse/HADOOP-2660
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hbase
> Reporter: Billy Pearson
>
> I thank we addressed this here
> HADOOP-2295
> but I have found it showing up again
> my hlog size is set to 250,000
> so on a recovery from a failed region server the recovery of scanning the logs takes longer then the
> hbase.hbasemaster.maxregionopen default of 30 secs
> and the master is thinks the region is open but the region server closes the region when done recovering becuase the master sent a
> MSG_REGION_CLOSE_WITHOUT_REPORT to the region server.
> I was able to get my table back online completely by adding
> hbase.hbasemaster.maxregionopen with a value of 300000 mili secs to my hbase-site.xml file
> and restart.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (HADOOP-2693) [hbase] NPE in getClosestRowBefore

2008-01-23T14:48:34Z

[hbase] NPE in getClosestRowBefore
----------------------------------

Key: HADOOP-2693
URL: https://issues.apache.org/jira/browse/HADOOP-2693
Project: Hadoop Core
Issue Type: Bug
Components: contrib/hbase
Reporter: stack

Just saw this in a load test:

{code}
2008-01-23 22:43:42,831 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call getClosestRowBefore(.META.,,1, TestTable,,999999999999999, 9223372036854775807) from XX.XX.XX.141:49068: error: java.io.IOException: java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
at org.apache.hadoop.io.Text.compareTo(Text.java:278)
at org.apache.hadoop.hbase.HStore.getRowKeyAtOrBefore(HStore.java:1846)
at org.apache.hadoop.hbase.HRegion.getClosestRowBefore(HRegion.java:1109)
at org.apache.hadoop.hbase.HRegionServer.getClosestRowBefore(HRegionServer.java:1434)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:908)
{code}

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-2596) add SequenceFile.createWriter() method that takes block size as parameter

2008-01-23T14:48:34Z

[ https://issues.apache.org/jira/browse/HADOOP-2596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561853#action_12561853 ]

Doug Cutting commented on HADOOP-2596:
--------------------------------------

I don't think we ought to hold this up for HADOOP-938. We do need to address the plethora of options, but that's a separate issue.

> add SequenceFile.createWriter() method that takes block size as parameter
> -------------------------------------------------------------------------
>
> Key: HADOOP-2596
> URL: https://issues.apache.org/jira/browse/HADOOP-2596
> Project: Hadoop Core
> Issue Type: Improvement
> Components: io
> Environment: all
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Minor
> Fix For: 0.16.0
>
> Attachments: patch2596.txt
>
>
> Currently it is not possible to create a SequenceFile.Writer using a block size other than the default.
> The createWriter() method should be overloaded with a signature receiving block size as parameter should be added to the the SequenceFile class.
> With all the current signatures for this method there is a significant code duplication, if possible the createWriter() methods should be refactored to avoid such duplication.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-938) too many SequenceFile.createWriter() methods

2008-01-23T14:48:34Z

[ https://issues.apache.org/jira/browse/HADOOP-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561852#action_12561852 ]

Doug Cutting commented on HADOOP-938:
-------------------------------------

Another approach would be to add an open() method, and use SequenceFile.Writer setter methods, e.g.:

SequenceFile.Writer writer = SequenceFile.createWriter(path);
writer.setCompressionType(Compress.BLOCK);
writer.setBlockSize(128*1024*1024);
writer.open();

> too many SequenceFile.createWriter() methods
> --------------------------------------------
>
> Key: HADOOP-938
> URL: https://issues.apache.org/jira/browse/HADOOP-938
> Project: Hadoop Core
> Issue Type: Improvement
> Components: io
> Reporter: Doug Cutting
>
> There are too many SequenceFile.createWriter() method signatures. This method has two required paramters: a Configuration and a Path. It has one obsolete parameter: a FileSystem. And it has five optional parameters: CompressionType, CompressionCodec, Progress, replication, and metadata.
> We should remove the obsolete parameter and make all optional parameters into setters.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Assigned: (HADOOP-2673) Flusher flushing back to back when it should not be

2008-01-23T14:46:34Z

[ https://issues.apache.org/jira/browse/HADOOP-2673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman reassigned HADOOP-2673:
-------------------------------------

Assignee: Jim Kellerman

> Flusher flushing back to back when it should not be
> ---------------------------------------------------
>
> Key: HADOOP-2673
> URL: https://issues.apache.org/jira/browse/HADOOP-2673
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hbase
> Environment: hadoop svn 613782
> Reporter: Billy Pearson
> Assignee: Jim Kellerman
> Attachments: log.log
>
>
> settings:
> hbase.regionserver.optionalcacheflushinterval = 300000 (5 mins)
> hbase.hregion.memcache.flush.size = 67108864 (64MB)
> I am attaching a log for this everything works fine until the first flush of the memcache. but once it flushes the first time correctly it flushes back to back many times even when it does not exceed one of the two limits above.
> I have not found but one spot where we update the variable LastFlushTime except when the server starts so this could be the problem but I may be missing it in the code somewhere. If it is getting updated then we need to look if something could be blocking it from updating after a memcache flush.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-2653) [HQL] Unnecessary HQLClient Object creation in a shell loop.

2008-01-23T14:36:35Z

[ https://issues.apache.org/jira/browse/HADOOP-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561848#action_12561848 ]

Nigel Daley commented on HADOOP-2653:
-------------------------------------

sorry, I belive the -1 from HadoopQA was wrong. Resubmitting it...

> [HQL] Unnecessary HQLClient Object creation in a shell loop.
> ------------------------------------------------------------
>
> Key: HADOOP-2653
> URL: https://issues.apache.org/jira/browse/HADOOP-2653
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hbase
> Affects Versions: 0.16.0
> Reporter: Edward Yoon
> Assignee: Edward Yoon
> Priority: Trivial
> Fix For: 0.17.0
>
> Attachments: 2653.patch, 2653_v01.patch
>
>
> {code}
> Shell.java
> while(....) {
> HQLClient hql = new HQLClient(...);
> }
> {code}

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (HADOOP-2692) [hbase] Client exits with IllegalStateException

2008-01-23T14:36:34Z

[hbase] Client exits with IllegalStateException
-----------------------------------------------

Key: HADOOP-2692
URL: https://issues.apache.org/jira/browse/HADOOP-2692
Project: Hadoop Core
Issue Type: Bug
Components: contrib/hbase
Reporter: stack

Running a load test, I see clients exiting with the below exception:

{code}
08/01/23 22:12:12 ERROR hbase.PerformanceEvaluation: Failed
java.lang.IllegalStateException: region offline: TestTable,0000099498,1201125579414
at org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:432)
at org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:343)
at org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:306)
at org.apache.hadoop.hbase.HTable.getRegionLocation(HTable.java:102)
at org.apache.hadoop.hbase.HTable.commit(HTable.java:959)
at org.apache.hadoop.hbase.HTable.commit(HTable.java:939)
at org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest.testRow(PerformanceEvaluation.java:461)
at org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:329)
at org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:525)
at org.apache.hadoop.hbase.PerformanceEvaluation.runNIsOne(PerformanceEvaluation.java:546)
at org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:568)
at org.apache.hadoop.hbase.PerformanceEvaluation.doCommandLine(PerformanceEvaluation.java:663)
at org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:682)
{code}

Looking in code, above is thrown because region is offline. A split is probably going on. Client should retry in this case after a little pause?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HADOOP-2691) Some junit tests fail with the exception: All datanodes are bad. Aborting...

2008-01-23T14:32:35Z

[ https://issues.apache.org/jira/browse/HADOOP-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-2691:
-------------------------------------

Status: Patch Available (was: Open)

> Some junit tests fail with the exception: All datanodes are bad. Aborting...
> ----------------------------------------------------------------------------
>
> Key: HADOOP-2691
> URL: https://issues.apache.org/jira/browse/HADOOP-2691
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.15.2
> Reporter: Hairong Kuang
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: datanodesBad.patch
>
>
> Some junit tests fail with the following exception:
> java.io.IOException: All datanodes are bad. Aborting...
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:1831)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1100(DFSClient.java:1479)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1571)
> The log contains the following message:
> 2008-01-19 23:00:25,557 INFO dfs.StateChange (FSNamesystem.java:allocateBlock(1274)) - BLOCK* NameSystem.allocateBlock: /srcdat/three/3189919341591612220. blk_6989304691537873255
> 2008-01-19 23:00:25,559 INFO fs.DFSClient (DFSClient.java:createBlockOutputStream(1982)) - pipeline = 127.0.0.1:40678
> 2008-01-19 23:00:25,559 INFO fs.DFSClient (DFSClient.java:createBlockOutputStream(1982)) - pipeline = 127.0.0.1:40680
> 2008-01-19 23:00:25,559 INFO fs.DFSClient (DFSClient.java:createBlockOutputStream(1985)) - Connecting to 127.0.0.1:40678
> 2008-01-19 23:00:25,570 INFO dfs.DataNode (DataNode.java:writeBlock(1084)) - Receiving block blk_6989304691537873255 from /127.0.0.1
> 2008-01-19 23:00:25,572 INFO dfs.DataNode (DataNode.java:writeBlock(1084)) - Receiving block blk_6989304691537873255 from /127.0.0.1
> 2008-01-19 23:00:25,573 INFO dfs.DataNode (DataNode.java:writeBlock(1169)) - Datanode 0 forwarding connect ack to upstream firstbadlink is
> 2008-01-19 23:00:25,573 INFO dfs.DataNode (DataNode.java:writeBlock(1150)) - Datanode 1 got response for connect ack from downstream datanode with firstbadlink as
> 2008-01-19 23:00:25,573 INFO dfs.DataNode (DataNode.java:writeBlock(1169)) - Datanode 1 forwarding connect ack to upstream firstbadlink is
> 2008-01-19 23:00:25,574 INFO dfs.DataNode (DataNode.java:lastDataNodeRun(1802)) - Received block blk_6989304691537873255 of size 34 from /127.0.0.1
> 2008-01-19 23:00:25,575 INFO dfs.DataNode (DataNode.java:lastDataNodeRun(1819)) - PacketResponder 0 for block blk_6989304691537873255 terminating
> 2008-01-19 23:00:25,575 INFO dfs.StateChange (FSNamesystem.java:addStoredBlock(2467)) - BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:40680 is added to blk_6989304691537873255 size 34
> 2008-01-19 23:00:25,575 INFO dfs.DataNode (DataNode.java:close(2013)) - BlockReceiver for block blk_6989304691537873255 waiting for last write to drain.
> 2008-01-19 23:01:31,577 WARN fs.DFSClient (DFSClient.java:run(1764)) - DFSOutputStream ResponseProcessor exception for block blk_6989304691537873255java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at java.io.DataInputStream.readFully(DataInputStream.java:176)
> at java.io.DataInputStream.readLong(DataInputStream.java:380)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:1726)
> 2008-01-19 23:01:31,578 INFO fs.DFSClient (DFSClient.java:run(1653)) - Closing old block blk_6989304691537873255
> 2008-01-19 23:01:31,579 WARN fs.DFSClient (DFSClient.java:processDatanodeError(1803)) - Error Recovery for block blk_6989304691537873255 bad datanode[0] 127.0.0.1:40678
> 2008-01-19 23:01:31,580 WARN fs.DFSClient (DFSClient.java:processDatanodeError(1836)) - Error Recovery for block blk_6989304691537873255 bad datanode 127.0.0.1:40678
> 2008-01-19 23:01:31,580 INFO fs.DFSClient (DFSClient.java:createBlockOutputStream(1982)) - pipeline = 127.0.0.1:40680
> 2008-01-19 23:01:31,580 INFO fs.DFSClient (DFSClient.java:createBlockOutputStream(1985)) - Connecting to 127.0.0.1:40680
> 2008-01-19 23:01:31,582 INFO dfs.DataNode (DataNode.java:writeBlock(1084)) - Receiving block blk_6989304691537873255 from /127.0.0.1
> 2008-01-19 23:01:31,584 INFO dfs.DataNode (DataNode.java:writeBlock(1196)) - writeBlock blk_6989304691537873255 received exception java.io.IOException: Reopen Block blk_6989304691537873255 is valid, and cannot be written to.
> 2008-01-19 23:01:31,584 ERROR dfs.DataNode (DataNode.java:run(997)) - 127.0.0.1:40680:DataXceiver: java.io.IOException: Reopen Block blk_6989304691537873255 is valid, and cannot be written to.
> at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:613)
> at org.apache.hadoop.dfs.DataNode$BlockReceiver.<init>(DataNode.java:1996)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:1109)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:982)
> at java.lang.Thread.run(Thread.java:595)
> 2008-01-19 23:01:31,585 INFO fs.DFSClient (DFSClient.java:createBlockOutputStream(2024)) - Exception in createBlockOutputStream java.io.EOFException
> The log shows that blk_6989304691537873255 was successfully written to two datanodes. But dfsclient timed out waiting for a response from the first datanode. It tried to recover from the failure by resending the data to the second datanode. However, the recovery failed because the second datanode threw an IOException when it detected that it already had the block. It would be nice that the second datanode does not throw an exception for a finalized block during a recovery.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Assigned: (HADOOP-2691) Some junit tests fail with the exception: All datanodes are bad. Aborting...

2008-01-23T14:32:34Z

[ https://issues.apache.org/jira/browse/HADOOP-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur reassigned HADOOP-2691:
----------------------------------------

Assignee: dhruba borthakur

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HADOOP-2691) Some junit tests fail with the exception: All datanodes are bad. Aborting...

2008-01-23T14:32:34Z

> Some junit tests fail with the exception: All datanodes are bad. Aborting...
> ----------------------------------------------------------------------------
>
> Key: HADOOP-2691
> URL: https://issues.apache.org/jira/browse/HADOOP-2691
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.15.2
> Reporter: Hairong Kuang
> Fix For: 0.16.0
>
> Attachments: datanodesBad.patch
>
>
> Some junit tests fail with the following exception:
> java.io.IOException: All datanodes are bad. Aborting...
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:1831)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1100(DFSClient.java:1479)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1571)
> The log contains the following message:
> 2008-01-19 23:00:25,557 INFO dfs.StateChange (FSNamesystem.java:allocateBlock(1274)) - BLOCK* NameSystem.allocateBlock: /srcdat/three/3189919341591612220. blk_6989304691537873255
> 2008-01-19 23:00:25,559 INFO fs.DFSClient (DFSClient.java:createBlockOutputStream(1982)) - pipeline = 127.0.0.1:40678
> 2008-01-19 23:00:25,559 INFO fs.DFSClient (DFSClient.java:createBlockOutputStream(1982)) - pipeline = 127.0.0.1:40680
> 2008-01-19 23:00:25,559 INFO fs.DFSClient (DFSClient.java:createBlockOutputStream(1985)) - Connecting to 127.0.0.1:40678
> 2008-01-19 23:00:25,570 INFO dfs.DataNode (DataNode.java:writeBlock(1084)) - Receiving block blk_6989304691537873255 from /127.0.0.1
> 2008-01-19 23:00:25,572 INFO dfs.DataNode (DataNode.java:writeBlock(1084)) - Receiving block blk_6989304691537873255 from /127.0.0.1
> 2008-01-19 23:00:25,573 INFO dfs.DataNode (DataNode.java:writeBlock(1169)) - Datanode 0 forwarding connect ack to upstream firstbadlink is
> 2008-01-19 23:00:25,573 INFO dfs.DataNode (DataNode.java:writeBlock(1150)) - Datanode 1 got response for connect ack from downstream datanode with firstbadlink as
> 2008-01-19 23:00:25,573 INFO dfs.DataNode (DataNode.java:writeBlock(1169)) - Datanode 1 forwarding connect ack to upstream firstbadlink is
> 2008-01-19 23:00:25,574 INFO dfs.DataNode (DataNode.java:lastDataNodeRun(1802)) - Received block blk_6989304691537873255 of size 34 from /127.0.0.1
> 2008-01-19 23:00:25,575 INFO dfs.DataNode (DataNode.java:lastDataNodeRun(1819)) - PacketResponder 0 for block blk_6989304691537873255 terminating
> 2008-01-19 23:00:25,575 INFO dfs.StateChange (FSNamesystem.java:addStoredBlock(2467)) - BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:40680 is added to blk_6989304691537873255 size 34
> 2008-01-19 23:00:25,575 INFO dfs.DataNode (DataNode.java:close(2013)) - BlockReceiver for block blk_6989304691537873255 waiting for last write to drain.
> 2008-01-19 23:01:31,577 WARN fs.DFSClient (DFSClient.java:run(1764)) - DFSOutputStream ResponseProcessor exception for block blk_6989304691537873255java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at java.io.DataInputStream.readFully(DataInputStream.java:176)
> at java.io.DataInputStream.readLong(DataInputStream.java:380)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:1726)
> 2008-01-19 23:01:31,578 INFO fs.DFSClient (DFSClient.java:run(1653)) - Closing old block blk_6989304691537873255
> 2008-01-19 23:01:31,579 WARN fs.DFSClient (DFSClient.java:processDatanodeError(1803)) - Error Recovery for block blk_6989304691537873255 bad datanode[0] 127.0.0.1:40678
> 2008-01-19 23:01:31,580 WARN fs.DFSClient (DFSClient.java:processDatanodeError(1836)) - Error Recovery for block blk_6989304691537873255 bad datanode 127.0.0.1:40678
> 2008-01-19 23:01:31,580 INFO fs.DFSClient (DFSClient.java:createBlockOutputStream(1982)) - pipeline = 127.0.0.1:40680
> 2008-01-19 23:01:31,580 INFO fs.DFSClient (DFSClient.java:createBlockOutputStream(1985)) - Connecting to 127.0.0.1:40680
> 2008-01-19 23:01:31,582 INFO dfs.DataNode (DataNode.java:writeBlock(1084)) - Receiving block blk_6989304691537873255 from /127.0.0.1
> 2008-01-19 23:01:31,584 INFO dfs.DataNode (DataNode.java:writeBlock(1196)) - writeBlock blk_6989304691537873255 received exception java.io.IOException: Reopen Block blk_6989304691537873255 is valid, and cannot be written to.
> 2008-01-19 23:01:31,584 ERROR dfs.DataNode (DataNode.java:run(997)) - 127.0.0.1:40680:DataXceiver: java.io.IOException: Reopen Block blk_6989304691537873255 is valid, and cannot be written to.
> at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:613)
> at org.apache.hadoop.dfs.DataNode$BlockReceiver.<init>(DataNode.java:1996)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:1109)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:982)
> at java.lang.Thread.run(Thread.java:595)
> 2008-01-19 23:01:31,585 INFO fs.DFSClient (DFSClient.java:createBlockOutputStream(2024)) - Exception in createBlockOutputStream java.io.EOFException
> The log shows that blk_6989304691537873255 was successfully written to two datanodes. But dfsclient timed out waiting for a response from the first datanode. It tried to recover from the failure by resending the data to the second datanode. However, the recovery failed because the second datanode threw an IOException when it detected that it already had the block. It would be nice that the second datanode does not throw an exception for a finalized block during a recovery.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HADOOP-2688) [hbase] IllegalArgumentException processing a shutdown stops server going down and results in millions of lines of output

2008-01-23T14:28:37Z

> [hbase] IllegalArgumentException processing a shutdown stops server going down and results in millions of lines of output
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-2688
> URL: https://issues.apache.org/jira/browse/HADOOP-2688
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hbase
> Reporter: stack
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: emptywritable-v2.patch, emptywritable.patch
>
>
> {code}
> 2008-01-23 02:38:54,537 WARN org.apache.hadoop.hbase.HMaster: Processing pending operations: ProcessServerShutdown of XX.XX.XX.128:60020
> java.lang.IllegalArgumentException: Can't build a writable with empty bytes array
> at org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:81)
> at org.apache.hadoop.hbase.HMaster$ProcessServerShutdown.scanMetaRegion(HMaster.java:2045)
> at org.apache.hadoop.hbase.HMaster$ProcessServerShutdown.process(HMaster.java:2230)
> at org.apache.hadoop.hbase.HMaster.run(HMaster.java:1059)
> {code}

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HADOOP-2688) [hbase] IllegalArgumentException processing a shutdown stops server going down and results in millions of lines of output

2008-01-23T14:28:35Z

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HADOOP-2645) Additional metrics & jmx beans and cleanup to use the recent metrics libraries

2008-01-23T14:26:34Z

[ https://issues.apache.org/jira/browse/HADOOP-2645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-2645:
---------------------------------

Attachment: metrics7.patch

> Additional metrics & jmx beans and cleanup to use the recent metrics libraries
> -------------------------------------------------------------------------------
>
> Key: HADOOP-2645
> URL: https://issues.apache.org/jira/browse/HADOOP-2645
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metrics2.patch, metrics3.patch, metrics4.patch, metrics5.patch, metrics6.patch, metrics7.patch
>
>
> This patch does following:
> Data node :
> Additional metrics
> Cleunup old metrics to use recent metrics lib
> MBeans for data node metrics
> Namenode
> Cleunup old updates of metrics to use recent metrics lib
> Split Name none mbean to 2 MBeans - statistics and FSNamesystem status
> rename of name node mbean for consistency.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-2653) [HQL] Unnecessary HQLClient Object creation in a shell loop.

2008-01-23T14:24:34Z

[ https://issues.apache.org/jira/browse/HADOOP-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561839#action_12561839 ]

Hadoop QA commented on HADOOP-2653:
-----------------------------------

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12373664/2653_v01.patch
against trunk revision 614699.

@author +1. The patch does not contain any @author tags.

patch -1. The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1652/console

This message is automatically generated.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (HADOOP-2691) Some junit tests fail with the exception: All datanodes are bad. Aborting...

2008-01-23T14:17:34Z

Some junit tests fail with the exception: All datanodes are bad. Aborting...
----------------------------------------------------------------------------

Key: HADOOP-2691
URL: https://issues.apache.org/jira/browse/HADOOP-2691
Project: Hadoop Core
Issue Type: Bug
Components: dfs
Affects Versions: 0.15.2
Reporter: Hairong Kuang
Fix For: 0.16.0

Some junit tests fail with the following exception:
java.io.IOException: All datanodes are bad. Aborting...
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:1831)
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1100(DFSClient.java:1479)
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1571)
The log contains the following message:
2008-01-19 23:00:25,557 INFO dfs.StateChange (FSNamesystem.java:allocateBlock(1274)) - BLOCK* NameSystem.allocateBlock: /srcdat/three/3189919341591612220. blk_6989304691537873255
2008-01-19 23:00:25,559 INFO fs.DFSClient (DFSClient.java:createBlockOutputStream(1982)) - pipeline = 127.0.0.1:40678
2008-01-19 23:00:25,559 INFO fs.DFSClient (DFSClient.java:createBlockOutputStream(1982)) - pipeline = 127.0.0.1:40680
2008-01-19 23:00:25,559 INFO fs.DFSClient (DFSClient.java:createBlockOutputStream(1985)) - Connecting to 127.0.0.1:40678
2008-01-19 23:00:25,570 INFO dfs.DataNode (DataNode.java:writeBlock(1084)) - Receiving block blk_6989304691537873255 from /127.0.0.1
2008-01-19 23:00:25,572 INFO dfs.DataNode (DataNode.java:writeBlock(1084)) - Receiving block blk_6989304691537873255 from /127.0.0.1
2008-01-19 23:00:25,573 INFO dfs.DataNode (DataNode.java:writeBlock(1169)) - Datanode 0 forwarding connect ack to upstream firstbadlink is
2008-01-19 23:00:25,573 INFO dfs.DataNode (DataNode.java:writeBlock(1150)) - Datanode 1 got response for connect ack from downstream datanode with firstbadlink as
2008-01-19 23:00:25,573 INFO dfs.DataNode (DataNode.java:writeBlock(1169)) - Datanode 1 forwarding connect ack to upstream firstbadlink is
2008-01-19 23:00:25,574 INFO dfs.DataNode (DataNode.java:lastDataNodeRun(1802)) - Received block blk_6989304691537873255 of size 34 from /127.0.0.1
2008-01-19 23:00:25,575 INFO dfs.DataNode (DataNode.java:lastDataNodeRun(1819)) - PacketResponder 0 for block blk_6989304691537873255 terminating
2008-01-19 23:00:25,575 INFO dfs.StateChange (FSNamesystem.java:addStoredBlock(2467)) - BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:40680 is added to blk_6989304691537873255 size 34
2008-01-19 23:00:25,575 INFO dfs.DataNode (DataNode.java:close(2013)) - BlockReceiver for block blk_6989304691537873255 waiting for last write to drain.
2008-01-19 23:01:31,577 WARN fs.DFSClient (DFSClient.java:run(1764)) - DFSOutputStream ResponseProcessor exception for block blk_6989304691537873255java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.DataInputStream.readFully(DataInputStream.java:176)
at java.io.DataInputStream.readLong(DataInputStream.java:380)
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:1726)

2008-01-19 23:01:31,578 INFO fs.DFSClient (DFSClient.java:run(1653)) - Closing old block blk_6989304691537873255
2008-01-19 23:01:31,579 WARN fs.DFSClient (DFSClient.java:processDatanodeError(1803)) - Error Recovery for block blk_6989304691537873255 bad datanode[0] 127.0.0.1:40678
2008-01-19 23:01:31,580 WARN fs.DFSClient (DFSClient.java:processDatanodeError(1836)) - Error Recovery for block blk_6989304691537873255 bad datanode 127.0.0.1:40678
2008-01-19 23:01:31,580 INFO fs.DFSClient (DFSClient.java:createBlockOutputStream(1982)) - pipeline = 127.0.0.1:40680
2008-01-19 23:01:31,580 INFO fs.DFSClient (DFSClient.java:createBlockOutputStream(1985)) - Connecting to 127.0.0.1:40680
2008-01-19 23:01:31,582 INFO dfs.DataNode (DataNode.java:writeBlock(1084)) - Receiving block blk_6989304691537873255 from /127.0.0.1
2008-01-19 23:01:31,584 INFO dfs.DataNode (DataNode.java:writeBlock(1196)) - writeBlock blk_6989304691537873255 received exception java.io.IOException: Reopen Block blk_6989304691537873255 is valid, and cannot be written to.
2008-01-19 23:01:31,584 ERROR dfs.DataNode (DataNode.java:run(997)) - 127.0.0.1:40680:DataXceiver: java.io.IOException: Reopen Block blk_6989304691537873255 is valid, and cannot be written to.
at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:613)
at org.apache.hadoop.dfs.DataNode$BlockReceiver.<init>(DataNode.java:1996)
at org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:1109)
at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:982)
at java.lang.Thread.run(Thread.java:595)

2008-01-19 23:01:31,585 INFO fs.DFSClient (DFSClient.java:createBlockOutputStream(2024)) - Exception in createBlockOutputStream java.io.EOFException

The log shows that blk_6989304691537873255 was successfully written to two datanodes. But dfsclient timed out waiting for a response from the first datanode. It tried to recover from the failure by resending the data to the second datanode. However, the recovery failed because the second datanode threw an IOException when it detected that it already had the block. It would be nice that the second datanode does not throw an exception for a finalized block during a recovery.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-2576) Namenode performance degradation over time

2008-01-23T14:03:39Z

[ https://issues.apache.org/jira/browse/HADOOP-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561829#action_12561829 ]

Christian Kunz commented on HADOOP-2576:
----------------------------------------

I applied Raghu's patch, restarted the nameserver, and will monitor its performance.

> Namenode performance degradation over time
> ------------------------------------------
>
> Key: HADOOP-2576
> URL: https://issues.apache.org/jira/browse/HADOOP-2576
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Reporter: Christian Kunz
> Assignee: Raghu Angadi
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: HADOOP-2576.patch
>
>
> We have a cluster running the same applications again and again with a high turnover of files.
> The performance of these applications seem to be correlated to the lifetime of the namenode:
> After starting the namenode, the applications need increasingly more time to complete, with about 50% more time after 1 week.
> During that time the namenode average cpu usage increases from typically 10% to 30%, memory usage nearly doubles (although the average amount of data on dfs stays the same), and the average load factor increases by a factor of 2-3 (although not significantly high, <2).
> When looking at the namenode and datanode logs, I see a lot of asks to delete blocks coming from the namenode for blocks not in the blockmap of the datanodes, repeatedly for the same blocks.
> When I counted the number of blocks asked by the namenode to be deleted, I noticed a noticeable increase with the lifetime of the namenode (a factor of 2-3 after 1 week).
> This makes me wonder whether the namenode does not purge the list of invalid blocks from non-existing blocks.
> But independently, the namenode has a degradation issue.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HADOOP-2688) [hbase] IllegalArgumentException processing a shutdown stops server going down and results in millions of lines of output

2008-01-23T13:24:34Z

[ https://issues.apache.org/jira/browse/HADOOP-2688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HADOOP-2688:
--------------------------

Attachment: emptywritable.patch

Jim, please review this patch. In the name of minimal changes this close to release, I did not change HMaster.run making it stay up whatever the exception.

M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStoreFile.java
(getReader): Remove synchronize
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMaster.java
Refactor all meta scans to use two new utility methods that
add checks for empty cells - the cause of this issue.
(getHRegionInfo, getRowMap): Added utility methods for converting
HbaseMapWritable to map keyed by column and a get of COL_REGIONINFO
that does check for empty cell.

> [hbase] IllegalArgumentException processing a shutdown stops server going down and results in millions of lines of output
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-2688
> URL: https://issues.apache.org/jira/browse/HADOOP-2688
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hbase
> Reporter: stack
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: emptywritable.patch
>
>
> {code}
> 2008-01-23 02:38:54,537 WARN org.apache.hadoop.hbase.HMaster: Processing pending operations: ProcessServerShutdown of XX.XX.XX.128:60020
> java.lang.IllegalArgumentException: Can't build a writable with empty bytes array
> at org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:81)
> at org.apache.hadoop.hbase.HMaster$ProcessServerShutdown.scanMetaRegion(HMaster.java:2045)
> at org.apache.hadoop.hbase.HMaster$ProcessServerShutdown.process(HMaster.java:2230)
> at org.apache.hadoop.hbase.HMaster.run(HMaster.java:1059)
> {code}

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HADOOP-2339) [Hbase Shell] Delete command with no WHERE clause

2008-01-23T13:17:34Z

[ https://issues.apache.org/jira/browse/HADOOP-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HADOOP-2339:
----------------------------------

Fix Version/s: (was: 0.16.0)

> [Hbase Shell] Delete command with no WHERE clause
> -------------------------------------------------
>
> Key: HADOOP-2339
> URL: https://issues.apache.org/jira/browse/HADOOP-2339
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hbase
> Affects Versions: 0.16.0
> Reporter: Edward Yoon
> Assignee: stack
> Priority: Minor
> Attachments: 2339.patch, 2339_v02.patch, 2339_v03.patch, 2339_v04.patch
>
>
> using HbaseAdmin.deleteColumn() method.
> {code}
> DELETE column_name FROM table_name;
> {code}

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HADOOP-2584) Web UI displays an IOException instead of the Tables

2008-01-23T13:10:33Z

[ https://issues.apache.org/jira/browse/HADOOP-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HADOOP-2584:
--------------------------

Resolution: Fixed
Status: Resolved (was: Patch Available)

Committed (a while back). Resolving.

> Web UI displays an IOException instead of the Tables
> ----------------------------------------------------
>
> Key: HADOOP-2584
> URL: https://issues.apache.org/jira/browse/HADOOP-2584
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hbase
> Affects Versions: 0.15.2
> Reporter: Lars George
> Priority: Trivial
> Fix For: 0.16.0
>
> Attachments: ui.patch
>
>
> For me after every second restart I get an error when loading the Hbase UI. Here the page:
> Master: 192.168.105.11:60000
> HQL, Local logs, Thread Dump, Log Level
> ______________________________________________________________________________________________________________________
>
> Master Attributes
> Attribute Name Value Description
> Filesystem lv1-xen-pdc-2.worldlingo.com:9000 Filesystem hbase is running on
> Hbase Root Directory /hbase Location of hbase home directory
>
> Online META Regions
>
> Name Server
> -ROOT- 192.168.105.31:60020
> .META.,,1 192.168.105.39:60020
>
> Tables
> error msg : java.io.IOException: java.io.IOException: HStoreScanner failed construction at
> org.apache.hadoop.hbase.HStore$StoreFileScanner.(HStore.java:1879) at
> org.apache.hadoop.hbase.HStore$HStoreScanner.(HStore.java:2000) at
> org.apache.hadoop.hbase.HStore.getScanner(HStore.java:1822) at
> org.apache.hadoop.hbase.HRegion$HScanner.(HRegion.java:1543) at
> org.apache.hadoop.hbase.HRegion.getScanner(HRegion.java:1118) at
> org.apache.hadoop.hbase.HRegionServer.openScanner(HRegionServer.java:1465) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at
> java.lang.reflect.Method.invoke(Method.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:401) at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:892) Caused by: org.apache.hadoop.ipc.RemoteException:
> java.io.IOException: File does not exist: /hbase/hregion_1028785192/info/mapfiles/6628785818889695133/data at
> org.apache.hadoop.dfs.FSDirectory.getFileInfo(FSDirectory.java:489) at
> org.apache.hadoop.dfs.FSNamesystem.getFileInfo(FSNamesystem.java:1380) at
> org.apache.hadoop.dfs.NameNode.getFileInfo(NameNode.java:425) at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown
> Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at
> java.lang.reflect.Method.invoke(Method.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:401) at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:892) at org.apache.hadoop.ipc.Client.call(Client.java:509) at
> org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:198) at org.apache.hadoop.dfs.$Proxy1.getFileInfo(Unknown Source) at
> sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at
> java.lang.reflect.Method.invoke(Method.java:585) at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) at
> org.apache.hadoop.dfs.$Proxy1.getFileInfo(Unknown Source) at
> org.apache.hadoop.dfs.DFSClient.getFileInfo(DFSClient.java:416) at
> org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:331) at
> org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:492) at
> org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1349) at
> org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1344) at org.apache.hadoop.io.MapFile$Reader.(MapFile.java:254) at org.apache.hadoop.io.MapFile$Reader.(MapFile.java:242) at
> org.apache.hadoop.hbase.HStoreFile$BloomFilterMapFile$Reader.(HStoreFile.java:816) at org.apache.hadoop.hbase.HStoreFile.getReader(HStoreFile.java:936) at
> org.apache.hadoop.hbase.HStore$StoreFileScanner.(HStore.java:1854) ... 11 more
> Region Servers
> Address Start Code Load
> 192.168.105.41:60020 1200095856575 requests: 0 regions: 48
> 192.168.105.31:60020 1200095853516 requests: 0 regions: 38
> 192.168.105.51:60020 1200095856709 requests: 0 regions: 20
> 192.168.105.21:60020 1200095852330 requests: 0 regions: 22
> 192.168.105.61:60020 1200095856758 requests: 0 regions: 39
> 192.168.105.71:60020 1200095856975 requests: 0 regions: 21
> 192.168.105.53:60020 1200095856655 requests: 0 regions: 19
> 192.168.105.43:60020 1200095854323 requests: 0 regions: 25
> 192.168.105.33:60020 1200095853508 requests: 0 regions: 218
> 192.168.105.23:60020 1200095852282 requests: 0 regions: 26
> 192.168.105.13:60020 1200095851830 requests: 0 regions: 27
> 192.168.105.73:60020 1200095856843 requests: 0 regions: 21
> 192.168.105.63:60020 1200095856776 requests: 0 regions: 19
> 192.168.105.35:60020 1200095853520 requests: 0 regions: 24
> 192.168.105.25:60020 1200095853591 requests: 1 regions: 25
> 192.168.105.75:60020 1200095856884 requests: 0 regions: 37
> 192.168.105.65:60020 1200095856817 requests: 0 regions: 19
> 192.168.105.55:60020 1200095856625 requests: 0 regions: 31
> 192.168.105.27:60020 1200095853490 requests: 0 regions: 23 192.168.105.37:60020 1200095853876 requests: 0 regions: 21
> 192.168.105.17:60020 1200095851832 requests: 0 regions: 25
> 192.168.105.47:60020 1200095856542 requests: 0 regions: 20
> 192.168.105.77:60020 1200095856853 requests: 0 regions: 22
> 192.168.105.67:60020 1200095856853 requests: 0 regions: 19
> 192.168.105.57:60020 1200095856719 requests: 0 regions: 25
> 192.168.105.19:60020 1200095851818 requests: 0 regions: 29
> 192.168.105.39:60020 1200095853864 requests: 0 regions: 2
> 192.168.105.29:60020 1200095853521 requests: 0 regions: 25
> 192.168.105.69:60020 1200095856831 requests: 0 regions: 21
> 192.168.105.49:60020 1200095856553 requests: 0 regions: 36
> 192.168.105.59:60020 1200095856736 requests: 0 regions: 19
> Total: servers: 33 requests: 1 regions: 1017
>
> Load is requests per hbase.regionsserver.msginterval (3 second(s)) and count of regions loaded
>
> After another restart, the tables display as they should. Reloading the page does not help.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-2232) Add option to disable nagles algorithm in the IPC Server

2008-01-23T12:43:34Z

[ https://issues.apache.org/jira/browse/HADOOP-2232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561816#action_12561816 ]

Mukund Madhugiri commented on HADOOP-2232:
------------------------------------------

Ok. Here is the data from the latest trunk. I did a run last night on trunk and *see* the exceptions on trunk as well. Here are the new trunk numbers and comparing them with the trunk + patch numbers, this patch looks good.

|*500 nodes*|*trunk*|*trunk + patch*|*Difference (%)*|
|randomWriter (mins)|29|28| ( -1.8% )
|sort (mins)|110|113| ( 2% )

> Add option to disable nagles algorithm in the IPC Server
> --------------------------------------------------------
>
> Key: HADOOP-2232
> URL: https://issues.apache.org/jira/browse/HADOOP-2232
> Project: Hadoop Core
> Issue Type: Improvement
> Components: ipc
> Affects Versions: 0.16.0
> Reporter: Clint Morgan
> Assignee: Clint Morgan
> Attachments: 2232-3.patch, HADOOP-2232-1.patch, HADOOP-2232-2.patch
>
>
> While investigating hbase performance, I found a bottleneck caused by
> Nagles algorithm. For some reads I would get a bi-modal distribution
> of read times, with about half the times being around 20ms, and half
> around 200ms. I tracked this down to the well-known interaction between
> Nagle's algorithm and TCP delayed acknowledgments.
> I found that calling setTcpNoDelay(true) on the server's socket
> connection dropped all of my read times back to a constant 20 ms.
> I propose a patch to have this TCP_NODELAY option be configurable. The
> attacked patch allows one to set the TCP_NODELAY option on both the
> client and the server side. Currently this is defaulted to false
> (i.e., with Nagle's enabled).
> To see the effect, I have included a Test which provokes the issue by
> sending a MapWriteable over an IPC call. On my machine this test shows
> a speedup of 117 times when using TCP_NODELAY.
> These tests were done on OSX 10.4. Your milage may very with other
> TCP/IP implementation stacks.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-2576) Namenode performance degradation over time

2008-01-23T12:41:34Z

[ https://issues.apache.org/jira/browse/HADOOP-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561814#action_12561814 ]

dhruba borthakur commented on HADOOP-2576:
------------------------------------------

I agree with Raghu that the throttle to delete blocks from a datanode could be done by the Datanode. Currently, the namenode does this throttling. See HADOOP-774 for more discussion on this topic.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HADOOP-2633) Revert change to fsck made as part of permissions implementation

2008-01-23T12:14:34Z

[ https://issues.apache.org/jira/browse/HADOOP-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-2633:
-------------------------------------------

Attachment: 2633_20080123.patch

2633_20080123.patch: put fsck unit tests back to one class, updated with current trunk.

> Revert change to fsck made as part of permissions implementation
> ----------------------------------------------------------------
>
> Key: HADOOP-2633
> URL: https://issues.apache.org/jira/browse/HADOOP-2633
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Reporter: Robert Chansler
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: 2633_20080116.patch, 2633_20080117.patch, 2633_20080117b.patch, 2633_20080117c.patch, 2633_20080118.patch, 2633_20080122.patch, 2633_20080123.patch
>
>
> Earlier change has unacceptable performance behavior.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-2690) Adding support into build.xml to build a special hadoop jar file that has the MiniDFSCluster and MiniMRCluster classes among others necessary for building and running the unit tests of Pig on the local mini cluster

2008-01-23T12:08:34Z

[ https://issues.apache.org/jira/browse/HADOOP-2690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561804#action_12561804 ]

Nigel Daley commented on HADOOP-2690:
-------------------------------------

This is a Pig packaging problem. Please close this as "will not fix"
and file a PIG issue.

> Adding support into build.xml to build a special hadoop jar file that has the MiniDFSCluster and MiniMRCluster classes among others necessary for building and running the unit tests of Pig on the local mini cluster
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-2690
> URL: https://issues.apache.org/jira/browse/HADOOP-2690
> Project: Hadoop Core
> Issue Type: Improvement
> Components: build
> Affects Versions: 0.15.2
> Reporter: Xu Zhang
> Attachments: build.xml, hadoop-0.15.3-dev-test.jar
>
>
> In order to build Pig and run its unit tests with ant, Pig needs to be built and run against a hadoop jar file that not only has the MiniDFSCluster and MiniMRCluster classes but also has classes from several 3rd party libraries. I added a target to the build.xml file of Hadoop for that purpose (see the "Compile code for Pig" section in the attached build.xml). I also attached the resulted hadoop jar with this report.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Assigned: (HADOOP-2646) SortValidator broken with fully-qualified working directories

2008-01-23T11:50:34Z

[ https://issues.apache.org/jira/browse/HADOOP-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy reassigned HADOOP-2646:
-------------------------------------

Assignee: Arun C Murthy

> SortValidator broken with fully-qualified working directories
> -------------------------------------------------------------
>
> Key: HADOOP-2646
> URL: https://issues.apache.org/jira/browse/HADOOP-2646
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: Doug Cutting
> Assignee: Arun C Murthy
> Attachments: HADOOP-2646.patch
>
>
> The sort validator is broken by HADOOP-2567. In particular, it no longer works when DistributedFileSystem#getWorkingDirectory() returns a fully-qualified path.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HADOOP-2690) Adding support into build.xml to build a special hadoop jar file that has the MiniDFSCluster and MiniMRCluster classes among others necessary for building and running the unit tests of Pig on the local mini cluster

2008-01-23T11:50:34Z

[ https://issues.apache.org/jira/browse/HADOOP-2690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xu Zhang updated HADOOP-2690:
-----------------------------

Attachment: hadoop-0.15.3-dev-test.jar

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-2596) add SequenceFile.createWriter() method that takes block size as parameter

2008-01-23T11:48:35Z

[ https://issues.apache.org/jira/browse/HADOOP-2596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561795#action_12561795 ]

Arun C Murthy commented on HADOOP-2596:
---------------------------------------

Sigh, looks like another unrelated test failed.

I'm ok about this patch, but then Doug's concern about too many SequenceFile.createWriter methods (HADOOP-938) springs to my mind... thoughts? Doug?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HADOOP-2690) Adding support into build.xml to build a special hadoop jar file that has the MiniDFSCluster and MiniMRCluster classes among others necessary for building and running the unit tests of Pig on the local mini cluster

2008-01-23T11:48:34Z

> Adding support into build.xml to build a special hadoop jar file that has the MiniDFSCluster and MiniMRCluster classes among others necessary for building and running the unit tests of Pig on the local mini cluster
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-2690
> URL: https://issues.apache.org/jira/browse/HADOOP-2690
> Project: Hadoop Core
> Issue Type: Improvement
> Components: build
> Affects Versions: 0.15.2
> Reporter: Xu Zhang
> Attachments: build.xml
>
>
> In order to build Pig and run its unit tests with ant, Pig needs to be built and run against a hadoop jar file that not only has the MiniDFSCluster and MiniMRCluster classes but also has classes from several 3rd party libraries. I added a target to the build.xml file of Hadoop for that purpose (see the "Compile code for Pig" section in the attached build.xml). I also attached the resulted hadoop jar with this report.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-1985) Abstract node to switch mapping into a topology service class used by namenode and jobtracker

2008-01-23T11:48:34Z

[ https://issues.apache.org/jira/browse/HADOOP-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561794#action_12561794 ]

Devaraj Das commented on HADOOP-1985:
-------------------------------------

I think the core tests failed due to HADOOP-2680 ("all datanodes are bad" ..). They pass on my machine.

> Abstract node to switch mapping into a topology service class used by namenode and jobtracker
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-1985
> URL: https://issues.apache.org/jira/browse/HADOOP-1985
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs, mapred
> Reporter: eric baldeschwieler
> Assignee: Devaraj Das
> Fix For: 0.16.0
>
> Attachments: 1985.new.patch, 1985.v1.patch, 1985.v10.patch, 1985.v2.patch, 1985.v3.patch, 1985.v4.patch, 1985.v5.patch, 1985.v6.patch, 1985.v9.patch, jobinprogress.patch
>
>
> In order to implement switch locality in MapReduce, we need to have switch location in both the namenode and job tracker. Currently the namenode asks the data nodes for this info and they run a local script to answer this question. In our environment and others that I know of there is no reason to push this to each node. It is easier to maintain a centralized script that maps node DNS names to switch strings.
> I propose that we build a new class that caches known DNS name to switch mappings and invokes a loadable class or a configurable system call to resolve unknown DNS to switch mappings. We can then add this to the namenode to support the current block to switch mapping needs and simplify the data nodes. We can also add this same callout to the job tracker and then implement rack locality logic there without needing to chane the filesystem API or the split planning API.
> Not only is this the least intrusive path to building racklocal MR I can ID, it is also future compatible to future infrastructures that may derive topology on the fly, etc, etc...

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.