| 
Apache
 » 
Lucene
 » 
Hadoop Lucene
 » 
Hadoop lucene-dev

[jira] Created: (HADOOP-2691) Some junit tests fail with the exception: All datanodes are bad. Aborting...

View: New views
4 Messages — Rating Filter:   Alert me  

[jira] Created: (HADOOP-2691) Some junit tests fail with the exception: All datanodes are bad. Aborting...

by JIRA jira@apache.org :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Some junit tests fail with the exception: All datanodes are bad. Aborting...
----------------------------------------------------------------------------

                 Key: HADOOP-2691
                 URL: https://issues.apache.org/jira/browse/HADOOP-2691
             Project: Hadoop Core
          Issue Type: Bug
          Components: dfs
    Affects Versions: 0.15.2
            Reporter: Hairong Kuang
             Fix For: 0.16.0


Some junit tests fail with the following exception:
java.io.IOException: All datanodes are bad. Aborting...
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:1831)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1100(DFSClient.java:1479)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1571)
The log contains the following message:
2008-01-19 23:00:25,557 INFO  dfs.StateChange (FSNamesystem.java:allocateBlock(1274)) - BLOCK* NameSystem.allocateBlock: /srcdat/three/3189919341591612220. blk_6989304691537873255
2008-01-19 23:00:25,559 INFO  fs.DFSClient (DFSClient.java:createBlockOutputStream(1982)) - pipeline = 127.0.0.1:40678
2008-01-19 23:00:25,559 INFO  fs.DFSClient (DFSClient.java:createBlockOutputStream(1982)) - pipeline = 127.0.0.1:40680
2008-01-19 23:00:25,559 INFO  fs.DFSClient (DFSClient.java:createBlockOutputStream(1985)) - Connecting to 127.0.0.1:40678
2008-01-19 23:00:25,570 INFO  dfs.DataNode (DataNode.java:writeBlock(1084)) - Receiving block blk_6989304691537873255 from /127.0.0.1
2008-01-19 23:00:25,572 INFO  dfs.DataNode (DataNode.java:writeBlock(1084)) - Receiving block blk_6989304691537873255 from /127.0.0.1
2008-01-19 23:00:25,573 INFO  dfs.DataNode (DataNode.java:writeBlock(1169)) - Datanode 0 forwarding connect ack to upstream firstbadlink is
2008-01-19 23:00:25,573 INFO  dfs.DataNode (DataNode.java:writeBlock(1150)) - Datanode 1 got response for connect ack  from downstream datanode with firstbadlink as
2008-01-19 23:00:25,573 INFO  dfs.DataNode (DataNode.java:writeBlock(1169)) - Datanode 1 forwarding connect ack to upstream firstbadlink is
2008-01-19 23:00:25,574 INFO  dfs.DataNode (DataNode.java:lastDataNodeRun(1802)) - Received block blk_6989304691537873255 of size 34 from /127.0.0.1
2008-01-19 23:00:25,575 INFO  dfs.DataNode (DataNode.java:lastDataNodeRun(1819)) - PacketResponder 0 for block blk_6989304691537873255 terminating
2008-01-19 23:00:25,575 INFO  dfs.StateChange (FSNamesystem.java:addStoredBlock(2467)) - BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:40680 is added to blk_6989304691537873255 size 34
2008-01-19 23:00:25,575 INFO  dfs.DataNode (DataNode.java:close(2013)) - BlockReceiver for block blk_6989304691537873255 waiting for last write to drain.
2008-01-19 23:01:31,577 WARN  fs.DFSClient (DFSClient.java:run(1764)) - DFSOutputStream ResponseProcessor exception  for block blk_6989304691537873255java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.io.DataInputStream.readFully(DataInputStream.java:176)
        at java.io.DataInputStream.readLong(DataInputStream.java:380)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:1726)

2008-01-19 23:01:31,578 INFO  fs.DFSClient (DFSClient.java:run(1653)) - Closing old block blk_6989304691537873255
2008-01-19 23:01:31,579 WARN  fs.DFSClient (DFSClient.java:processDatanodeError(1803)) - Error Recovery for block blk_6989304691537873255 bad datanode[0] 127.0.0.1:40678
2008-01-19 23:01:31,580 WARN  fs.DFSClient (DFSClient.java:processDatanodeError(1836)) - Error Recovery for block blk_6989304691537873255 bad datanode 127.0.0.1:40678
2008-01-19 23:01:31,580 INFO  fs.DFSClient (DFSClient.java:createBlockOutputStream(1982)) - pipeline = 127.0.0.1:40680
2008-01-19 23:01:31,580 INFO  fs.DFSClient (DFSClient.java:createBlockOutputStream(1985)) - Connecting to 127.0.0.1:40680
2008-01-19 23:01:31,582 INFO  dfs.DataNode (DataNode.java:writeBlock(1084)) - Receiving block blk_6989304691537873255 from /127.0.0.1
2008-01-19 23:01:31,584 INFO  dfs.DataNode (DataNode.java:writeBlock(1196)) - writeBlock blk_6989304691537873255 received exception java.io.IOException: Reopen Block blk_6989304691537873255 is valid, and cannot be written to.
2008-01-19 23:01:31,584 ERROR dfs.DataNode (DataNode.java:run(997)) - 127.0.0.1:40680:DataXceiver: java.io.IOException: Reopen Block blk_6989304691537873255 is valid, and cannot be written to.
        at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:613)
        at org.apache.hadoop.dfs.DataNode$BlockReceiver.<init>(DataNode.java:1996)
        at org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:1109)
        at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:982)
        at java.lang.Thread.run(Thread.java:595)

2008-01-19 23:01:31,585 INFO  fs.DFSClient (DFSClient.java:createBlockOutputStream(2024)) - Exception in createBlockOutputStream java.io.EOFException

The log shows that blk_6989304691537873255 was successfully written to two datanodes. But dfsclient timed out waiting for a response from the first datanode. It tried to recover from the failure by resending the data to the second datanode. However, the recovery failed because the second datanode threw an IOException when it detected that it already had the block. It would be nice that the second datanode does not throw an exception for a finalized block during a recovery.


--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Assigned: (HADOOP-2691) Some junit tests fail with the exception: All datanodes are bad. Aborting...

by JIRA jira@apache.org :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message


     [ https://issues.apache.org/jira/browse/HADOOP-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur reassigned HADOOP-2691:
----------------------------------------

    Assignee: dhruba borthakur

> Some junit tests fail with the exception: All datanodes are bad. Aborting...
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-2691
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2691
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.15.2
>            Reporter: Hairong Kuang
>            Assignee: dhruba borthakur
>             Fix For: 0.16.0
>
>         Attachments: datanodesBad.patch
>
>
> Some junit tests fail with the following exception:
> java.io.IOException: All datanodes are bad. Aborting...
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:1831)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1100(DFSClient.java:1479)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1571)
> The log contains the following message:
> 2008-01-19 23:00:25,557 INFO  dfs.StateChange (FSNamesystem.java:allocateBlock(1274)) - BLOCK* NameSystem.allocateBlock: /srcdat/three/3189919341591612220. blk_6989304691537873255
> 2008-01-19 23:00:25,559 INFO  fs.DFSClient (DFSClient.java:createBlockOutputStream(1982)) - pipeline = 127.0.0.1:40678
> 2008-01-19 23:00:25,559 INFO  fs.DFSClient (DFSClient.java:createBlockOutputStream(1982)) - pipeline = 127.0.0.1:40680
> 2008-01-19 23:00:25,559 INFO  fs.DFSClient (DFSClient.java:createBlockOutputStream(1985)) - Connecting to 127.0.0.1:40678
> 2008-01-19 23:00:25,570 INFO  dfs.DataNode (DataNode.java:writeBlock(1084)) - Receiving block blk_6989304691537873255 from /127.0.0.1
> 2008-01-19 23:00:25,572 INFO  dfs.DataNode (DataNode.java:writeBlock(1084)) - Receiving block blk_6989304691537873255 from /127.0.0.1
> 2008-01-19 23:00:25,573 INFO  dfs.DataNode (DataNode.java:writeBlock(1169)) - Datanode 0 forwarding connect ack to upstream firstbadlink is
> 2008-01-19 23:00:25,573 INFO  dfs.DataNode (DataNode.java:writeBlock(1150)) - Datanode 1 got response for connect ack  from downstream datanode with firstbadlink as
> 2008-01-19 23:00:25,573 INFO  dfs.DataNode (DataNode.java:writeBlock(1169)) - Datanode 1 forwarding connect ack to upstream firstbadlink is
> 2008-01-19 23:00:25,574 INFO  dfs.DataNode (DataNode.java:lastDataNodeRun(1802)) - Received block blk_6989304691537873255 of size 34 from /127.0.0.1
> 2008-01-19 23:00:25,575 INFO  dfs.DataNode (DataNode.java:lastDataNodeRun(1819)) - PacketResponder 0 for block blk_6989304691537873255 terminating
> 2008-01-19 23:00:25,575 INFO  dfs.StateChange (FSNamesystem.java:addStoredBlock(2467)) - BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:40680 is added to blk_6989304691537873255 size 34
> 2008-01-19 23:00:25,575 INFO  dfs.DataNode (DataNode.java:close(2013)) - BlockReceiver for block blk_6989304691537873255 waiting for last write to drain.
> 2008-01-19 23:01:31,577 WARN  fs.DFSClient (DFSClient.java:run(1764)) - DFSOutputStream ResponseProcessor exception  for block blk_6989304691537873255java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at java.io.DataInputStream.readFully(DataInputStream.java:176)
> at java.io.DataInputStream.readLong(DataInputStream.java:380)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:1726)
> 2008-01-19 23:01:31,578 INFO  fs.DFSClient (DFSClient.java:run(1653)) - Closing old block blk_6989304691537873255
> 2008-01-19 23:01:31,579 WARN  fs.DFSClient (DFSClient.java:processDatanodeError(1803)) - Error Recovery for block blk_6989304691537873255 bad datanode[0] 127.0.0.1:40678
> 2008-01-19 23:01:31,580 WARN  fs.DFSClient (DFSClient.java:processDatanodeError(1836)) - Error Recovery for block blk_6989304691537873255 bad datanode 127.0.0.1:40678
> 2008-01-19 23:01:31,580 INFO  fs.DFSClient (DFSClient.java:createBlockOutputStream(1982)) - pipeline = 127.0.0.1:40680
> 2008-01-19 23:01:31,580 INFO  fs.DFSClient (DFSClient.java:createBlockOutputStream(1985)) - Connecting to 127.0.0.1:40680
> 2008-01-19 23:01:31,582 INFO  dfs.DataNode (DataNode.java:writeBlock(1084)) - Receiving block blk_6989304691537873255 from /127.0.0.1
> 2008-01-19 23:01:31,584 INFO  dfs.DataNode (DataNode.java:writeBlock(1196)) - writeBlock blk_6989304691537873255 received exception java.io.IOException: Reopen Block blk_6989304691537873255 is valid, and cannot be written to.
> 2008-01-19 23:01:31,584 ERROR dfs.DataNode (DataNode.java:run(997)) - 127.0.0.1:40680:DataXceiver: java.io.IOException: Reopen Block blk_6989304691537873255 is valid, and cannot be written to.
> at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:613)
> at org.apache.hadoop.dfs.DataNode$BlockReceiver.<init>(DataNode.java:1996)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:1109)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:982)
> at java.lang.Thread.run(Thread.java:595)
> 2008-01-19 23:01:31,585 INFO  fs.DFSClient (DFSClient.java:createBlockOutputStream(2024)) - Exception in createBlockOutputStream java.io.EOFException
> The log shows that blk_6989304691537873255 was successfully written to two datanodes. But dfsclient timed out waiting for a response from the first datanode. It tried to recover from the failure by resending the data to the second datanode. However, the recovery failed because the second datanode threw an IOException when it detected that it already had the block. It would be nice that the second datanode does not throw an exception for a finalized block during a recovery.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-2691) Some junit tests fail with the exception: All datanodes are bad. Aborting...

by JIRA jira@apache.org :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message


     [ https://issues.apache.org/jira/browse/HADOOP-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-2691:
-------------------------------------

    Attachment: datanodesBad.patch

Triggering hudson test because this problem occurs only on the Hudson test runs.

> Some junit tests fail with the exception: All datanodes are bad. Aborting...
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-2691
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2691
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.15.2
>            Reporter: Hairong Kuang
>             Fix For: 0.16.0
>
>         Attachments: datanodesBad.patch
>
>
> Some junit tests fail with the following exception:
> java.io.IOException: All datanodes are bad. Aborting...
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:1831)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1100(DFSClient.java:1479)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1571)
> The log contains the following message:
> 2008-01-19 23:00:25,557 INFO  dfs.StateChange (FSNamesystem.java:allocateBlock(1274)) - BLOCK* NameSystem.allocateBlock: /srcdat/three/3189919341591612220. blk_6989304691537873255
> 2008-01-19 23:00:25,559 INFO  fs.DFSClient (DFSClient.java:createBlockOutputStream(1982)) - pipeline = 127.0.0.1:40678
> 2008-01-19 23:00:25,559 INFO  fs.DFSClient (DFSClient.java:createBlockOutputStream(1982)) - pipeline = 127.0.0.1:40680
> 2008-01-19 23:00:25,559 INFO  fs.DFSClient (DFSClient.java:createBlockOutputStream(1985)) - Connecting to 127.0.0.1:40678
> 2008-01-19 23:00:25,570 INFO  dfs.DataNode (DataNode.java:writeBlock(1084)) - Receiving block blk_6989304691537873255 from /127.0.0.1
> 2008-01-19 23:00:25,572 INFO  dfs.DataNode (DataNode.java:writeBlock(1084)) - Receiving block blk_6989304691537873255 from /127.0.0.1
> 2008-01-19 23:00:25,573 INFO  dfs.DataNode (DataNode.java:writeBlock(1169)) - Datanode 0 forwarding connect ack to upstream firstbadlink is
> 2008-01-19 23:00:25,573 INFO  dfs.DataNode (DataNode.java:writeBlock(1150)) - Datanode 1 got response for connect ack  from downstream datanode with firstbadlink as
> 2008-01-19 23:00:25,573 INFO  dfs.DataNode (DataNode.java:writeBlock(1169)) - Datanode 1 forwarding connect ack to upstream firstbadlink is
> 2008-01-19 23:00:25,574 INFO  dfs.DataNode (DataNode.java:lastDataNodeRun(1802)) - Received block blk_6989304691537873255 of size 34 from /127.0.0.1
> 2008-01-19 23:00:25,575 INFO  dfs.DataNode (DataNode.java:lastDataNodeRun(1819)) - PacketResponder 0 for block blk_6989304691537873255 terminating
> 2008-01-19 23:00:25,575 INFO  dfs.StateChange (FSNamesystem.java:addStoredBlock(2467)) - BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:40680 is added to blk_6989304691537873255 size 34
> 2008-01-19 23:00:25,575 INFO  dfs.DataNode (DataNode.java:close(2013)) - BlockReceiver for block blk_6989304691537873255 waiting for last write to drain.
> 2008-01-19 23:01:31,577 WARN  fs.DFSClient (DFSClient.java:run(1764)) - DFSOutputStream ResponseProcessor exception  for block blk_6989304691537873255java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at java.io.DataInputStream.readFully(DataInputStream.java:176)
> at java.io.DataInputStream.readLong(DataInputStream.java:380)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:1726)
> 2008-01-19 23:01:31,578 INFO  fs.DFSClient (DFSClient.java:run(1653)) - Closing old block blk_6989304691537873255
> 2008-01-19 23:01:31,579 WARN  fs.DFSClient (DFSClient.java:processDatanodeError(1803)) - Error Recovery for block blk_6989304691537873255 bad datanode[0] 127.0.0.1:40678
> 2008-01-19 23:01:31,580 WARN  fs.DFSClient (DFSClient.java:processDatanodeError(1836)) - Error Recovery for block blk_6989304691537873255 bad datanode 127.0.0.1:40678
> 2008-01-19 23:01:31,580 INFO  fs.DFSClient (DFSClient.java:createBlockOutputStream(1982)) - pipeline = 127.0.0.1:40680
> 2008-01-19 23:01:31,580 INFO  fs.DFSClient (DFSClient.java:createBlockOutputStream(1985)) - Connecting to 127.0.0.1:40680
> 2008-01-19 23:01:31,582 INFO  dfs.DataNode (DataNode.java:writeBlock(1084)) - Receiving block blk_6989304691537873255 from /127.0.0.1
> 2008-01-19 23:01:31,584 INFO  dfs.DataNode (DataNode.java:writeBlock(1196)) - writeBlock blk_6989304691537873255 received exception java.io.IOException: Reopen Block blk_6989304691537873255 is valid, and cannot be written to.
> 2008-01-19 23:01:31,584 ERROR dfs.DataNode (DataNode.java:run(997)) - 127.0.0.1:40680:DataXceiver: java.io.IOException: Reopen Block blk_6989304691537873255 is valid, and cannot be written to.
> at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:613)
> at org.apache.hadoop.dfs.DataNode$BlockReceiver.<init>(DataNode.java:1996)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:1109)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:982)
> at java.lang.Thread.run(Thread.java:595)
> 2008-01-19 23:01:31,585 INFO  fs.DFSClient (DFSClient.java:createBlockOutputStream(2024)) - Exception in createBlockOutputStream java.io.EOFException
> The log shows that blk_6989304691537873255 was successfully written to two datanodes. But dfsclient timed out waiting for a response from the first datanode. It tried to recover from the failure by resending the data to the second datanode. However, the recovery failed because the second datanode threw an IOException when it detected that it already had the block. It would be nice that the second datanode does not throw an exception for a finalized block during a recovery.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-2691) Some junit tests fail with the exception: All datanodes are bad. Aborting...

by JIRA jira@apache.org :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message


     [ https://issues.apache.org/jira/browse/HADOOP-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-2691:
-------------------------------------

    Status: Patch Available  (was: Open)

> Some junit tests fail with the exception: All datanodes are bad. Aborting...
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-2691
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2691
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.15.2
>            Reporter: Hairong Kuang
>            Assignee: dhruba borthakur
>             Fix For: 0.16.0
>
>         Attachments: datanodesBad.patch
>
>
> Some junit tests fail with the following exception:
> java.io.IOException: All datanodes are bad. Aborting...
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:1831)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1100(DFSClient.java:1479)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1571)
> The log contains the following message:
> 2008-01-19 23:00:25,557 INFO  dfs.StateChange (FSNamesystem.java:allocateBlock(1274)) - BLOCK* NameSystem.allocateBlock: /srcdat/three/3189919341591612220. blk_6989304691537873255
> 2008-01-19 23:00:25,559 INFO  fs.DFSClient (DFSClient.java:createBlockOutputStream(1982)) - pipeline = 127.0.0.1:40678
> 2008-01-19 23:00:25,559 INFO  fs.DFSClient (DFSClient.java:createBlockOutputStream(1982)) - pipeline = 127.0.0.1:40680
> 2008-01-19 23:00:25,559 INFO  fs.DFSClient (DFSClient.java:createBlockOutputStream(1985)) - Connecting to 127.0.0.1:40678
> 2008-01-19 23:00:25,570 INFO  dfs.DataNode (DataNode.java:writeBlock(1084)) - Receiving block blk_6989304691537873255 from /127.0.0.1
> 2008-01-19 23:00:25,572 INFO  dfs.DataNode (DataNode.java:writeBlock(1084)) - Receiving block blk_6989304691537873255 from /127.0.0.1
> 2008-01-19 23:00:25,573 INFO  dfs.DataNode (DataNode.java:writeBlock(1169)) - Datanode 0 forwarding connect ack to upstream firstbadlink is
> 2008-01-19 23:00:25,573 INFO  dfs.DataNode (DataNode.java:writeBlock(1150)) - Datanode 1 got response for connect ack  from downstream datanode with firstbadlink as
> 2008-01-19 23:00:25,573 INFO  dfs.DataNode (DataNode.java:writeBlock(1169)) - Datanode 1 forwarding connect ack to upstream firstbadlink is
> 2008-01-19 23:00:25,574 INFO  dfs.DataNode (DataNode.java:lastDataNodeRun(1802)) - Received block blk_6989304691537873255 of size 34 from /127.0.0.1
> 2008-01-19 23:00:25,575 INFO  dfs.DataNode (DataNode.java:lastDataNodeRun(1819)) - PacketResponder 0 for block blk_6989304691537873255 terminating
> 2008-01-19 23:00:25,575 INFO  dfs.StateChange (FSNamesystem.java:addStoredBlock(2467)) - BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:40680 is added to blk_6989304691537873255 size 34
> 2008-01-19 23:00:25,575 INFO  dfs.DataNode (DataNode.java:close(2013)) - BlockReceiver for block blk_6989304691537873255 waiting for last write to drain.
> 2008-01-19 23:01:31,577 WARN  fs.DFSClient (DFSClient.java:run(1764)) - DFSOutputStream ResponseProcessor exception  for block blk_6989304691537873255java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at java.io.DataInputStream.readFully(DataInputStream.java:176)
> at java.io.DataInputStream.readLong(DataInputStream.java:380)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:1726)
> 2008-01-19 23:01:31,578 INFO  fs.DFSClient (DFSClient.java:run(1653)) - Closing old block blk_6989304691537873255
> 2008-01-19 23:01:31,579 WARN  fs.DFSClient (DFSClient.java:processDatanodeError(1803)) - Error Recovery for block blk_6989304691537873255 bad datanode[0] 127.0.0.1:40678
> 2008-01-19 23:01:31,580 WARN  fs.DFSClient (DFSClient.java:processDatanodeError(1836)) - Error Recovery for block blk_6989304691537873255 bad datanode 127.0.0.1:40678
> 2008-01-19 23:01:31,580 INFO  fs.DFSClient (DFSClient.java:createBlockOutputStream(1982)) - pipeline = 127.0.0.1:40680
> 2008-01-19 23:01:31,580 INFO  fs.DFSClient (DFSClient.java:createBlockOutputStream(1985)) - Connecting to 127.0.0.1:40680
> 2008-01-19 23:01:31,582 INFO  dfs.DataNode (DataNode.java:writeBlock(1084)) - Receiving block blk_6989304691537873255 from /127.0.0.1
> 2008-01-19 23:01:31,584 INFO  dfs.DataNode (DataNode.java:writeBlock(1196)) - writeBlock blk_6989304691537873255 received exception java.io.IOException: Reopen Block blk_6989304691537873255 is valid, and cannot be written to.
> 2008-01-19 23:01:31,584 ERROR dfs.DataNode (DataNode.java:run(997)) - 127.0.0.1:40680:DataXceiver: java.io.IOException: Reopen Block blk_6989304691537873255 is valid, and cannot be written to.
> at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:613)
> at org.apache.hadoop.dfs.DataNode$BlockReceiver.<init>(DataNode.java:1996)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:1109)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:982)
> at java.lang.Thread.run(Thread.java:595)
> 2008-01-19 23:01:31,585 INFO  fs.DFSClient (DFSClient.java:createBlockOutputStream(2024)) - Exception in createBlockOutputStream java.io.EOFException
> The log shows that blk_6989304691537873255 was successfully written to two datanodes. But dfsclient timed out waiting for a response from the first datanode. It tried to recover from the failure by resending the data to the second datanode. However, the recovery failed because the second datanode threw an IOException when it detected that it already had the block. It would be nice that the second datanode does not throw an exception for a finalized block during a recovery.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Free embeddable forum powered by Nabble Forum Help