EOFException while trying to read 65557 bytes

[bhavin pandya-3]

Hi,

I got following exception in my datanode log file while udating db.

2009-11-03 04:39:24,273 ERROR datanode.DataNode -
DatanodeRegistration(192.168.101.152:50010,
storageID=DS-1706374374-192.168.101.152-50010-1255721446274,
infoPort=50075, ipcPort=50020):DataXceiver
java.io.EOFException: while trying to read 65557 bytes
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:264)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:308)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:372)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:524)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:357)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
        at java.lang.Thread.run(Thread.java:619)

Process stuck at this exception.

I am running 3 node in cluster ( 1 m + 2 s)

is this memory issue?  My hadoop heap size is 1 GB.

I am using hadoop-0.19 core with nutch 1.0.

Here are few settings.

<property>
  <name>mapred.map.tasks</name>
  <value>3</value>
  <description>
    define mapred.map tasks to be number of slave hosts
  </description>
</property>

<property>
  <name>mapred.reduce.tasks</name>
  <value>3</value>
  <description>
    define mapred.reduce tasks to be number of slave hosts
  </description>
</property>

<property>
  <name>dfs.replication</name>
  <value>2</value>
</property>


- Bhavin

[bhavin pandya-3]

Hello everyone,

Here is more info about the exception.

On both slave node i got exception.

2009-11-03 04:39:24,273 ERROR datanode.DataNode -
DatanodeRegistration(192.168.101.152:50010,
storageID=DS-1706374374-192.168.101.152-50010-1255721446274,
infoPort=50075, ipcPort=50020):DataXceiver
java.io.EOFException: while trying to read 65557 bytes
       at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:264)
       at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:308)
       at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:372)
       at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:524)
       at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:357)
       at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
       at java.lang.Thread.run(Thread.java:619)


On master node, namenode log file has warning...
2009-11-03 04:47:19,863 WARN  hdfs.StateChange - BLOCK*
INodeFileUnderConstruction.initLeaseRecovery: No blocks found, lease
removed.

Process stucks and on master node, above namenode warning repeating at
every 1 hour.

On one of the slave node where i got this exception, i checked log
through http://192.168.101.152:50060/ and its showing something like
this.

Running task:         attempt_200911021146_0004_m_000051_0
UNASSIGNED           0.00%

Non running task:
attempt_200911021146_0004_m_000000_0         SUCCEEDED
attempt_200911021146_0004_m_000002_0         SUCCEEDED

Task from running job:
attempt_200911021146_0004_m_000000_0         SUCCEEDED  100.00%
attempt_200911021146_0004_m_000002_0         SUCCEEDED  100.00%


- Bhavin


On Tue, Nov 3, 2009 at 11:07 AM, bhavin pandya <bvnpandya@...> wrote:

> Hi,
>
> I got following exception in my datanode log file while udating db.
>
> 2009-11-03 04:39:24,273 ERROR datanode.DataNode -
> DatanodeRegistration(192.168.101.152:50010,
> storageID=DS-1706374374-192.168.101.152-50010-1255721446274,
> infoPort=50075, ipcPort=50020):DataXceiver
> java.io.EOFException: while trying to read 65557 bytes
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:264)
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:308)
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:372)
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:524)
>        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:357)
>        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
>        at java.lang.Thread.run(Thread.java:619)
>
> Process stuck at this exception.
>
> I am running 3 node in cluster ( 1 m + 2 s)
>
> is this memory issue?  My hadoop heap size is 1 GB.
>
> I am using hadoop-0.19 core with nutch 1.0.
>
> Here are few settings.
>
> <property>
>  <name>mapred.map.tasks</name>
>  <value>3</value>
>  <description>
>    define mapred.map tasks to be number of slave hosts
>  </description>
> </property>
>
> <property>
>  <name>mapred.reduce.tasks</name>
>  <value>3</value>
>  <description>
>    define mapred.reduce tasks to be number of slave hosts
>  </description>
> </property>
>
> <property>
>  <name>dfs.replication</name>
>  <value>2</value>
> </property>
>
>
> - Bhavin
>

--
- Bhavin