unbalanced fetching

[Jesse Hires]

I have a two datanode and one namenode setup. One of my datanodes is slower
than the other, causing the fetch to run significantly longer on it. Is
there a way to balance this out?

Jesse

int GetRandomNumber()
{
   return 4; // Chosen by fair roll of dice
                // Guaranteed to be random
} // xkcd.com

[Andrzej Bialecki]

Jesse Hires wrote:
> I have a two datanode and one namenode setup. One of my datanodes is slower
> than the other, causing the fetch to run significantly longer on it. Is
> there a way to balance this out?

Most likely the number of URLs/host is unbalanced, meaning that the
tasktracker that takes the longest is assigned a lot of URLs from a
single host.

A workaround for this is to limit the max number of URLs per host (in
nutch-site.xml) to a more reasonable number, e.g. 100 or 1000, whatever
works best for you.

--
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com

[Jesse Hires]

Thanks, I'll give that a shot!
Jesse

int GetRandomNumber()
{
   return 4; // Chosen by fair roll of dice
                // Guaranteed to be random
} // xkcd.com



On Thu, Oct 29, 2009 at 5:53 AM, Andrzej Bialecki <ab@...> wrote:

> Jesse Hires wrote:
>
>> I have a two datanode and one namenode setup. One of my datanodes is
>> slower
>> than the other, causing the fetch to run significantly longer on it. Is
>> there a way to balance this out?
>>
>
> Most likely the number of URLs/host is unbalanced, meaning that the
> tasktracker that takes the longest is assigned a lot of URLs from a single
> host.
>
> A workaround for this is to limit the max number of URLs per host (in
> nutch-site.xml) to a more reasonable number, e.g. 100 or 1000, whatever
> works best for you.
>
> --
> Best regards,
> Andrzej Bialecki     <><
>  ___. ___ ___ ___ _ _   __________________________________
> [__ || __|__/|__||\/|  Information Retrieval, Semantic Web
> ___|||__||  \|  ||  |  Embedded Unix, System Integration
> http://www.sigram.com  Contact: info at sigram dot com
>
>