MergeSegments - java.lang.OutOfMemoryError

[kevin chen-6]

Hi, I have using a trunk version of nutch since Jul 2007. It's being
running fine since.

Recently I am experimenting with nutch 1.0. Everything worked great and
better until I start to use MergeSegments.  I was merging segments with
around 20k urls and it gave me OutOfMemoryError. I have tried to
increase the java heap max to 3G, I still got OutOfMemoryError.  In
contrast, in my older version of nutch,  same merge works with the
default java heap max setting of only 1G.

Dose anybody have the same experience? Is there any work around this?

Thanks
Kevin Chen

[Fadzi Ushewokunze-2]

i have a similar issue; i havent been able to get to the bottom of it.

On Sat, 2009-11-07 at 23:31 -0500, kevin chen wrote:

> Hi, I have using a trunk version of nutch since Jul 2007. It's being
> running fine since.
>
> Recently I am experimenting with nutch 1.0. Everything worked great and
> better until I start to use MergeSegments.  I was merging segments with
> around 20k urls and it gave me OutOfMemoryError. I have tried to
> increase the java heap max to 3G, I still got OutOfMemoryError.  In
> contrast, in my older version of nutch,  same merge works with the
> default java heap max setting of only 1G.
>
> Dose anybody have the same experience? Is there any work around this?
>
> Thanks
> Kevin Chen
>

[Julien Nioche-4]

Hi guys,

Could you send a stack trace of the process? Have you tried using a profiler
to check where the memory was used?
Check http://hadoop.apache.org/common/docs/current/mapred_tutorial.html for
instructions on how to profile with Hadoop in (pseudo) distributed mode.

Julien
--
DigitalPebble Ltd
http://www.digitalpebble.com

2009/11/8 Fadzi Ushewokunze <fadzi@...>

> i have a similar issue; i havent been able to get to the bottom of it.
>
> On Sat, 2009-11-07 at 23:31 -0500, kevin chen wrote:
> > Hi, I have using a trunk version of nutch since Jul 2007. It's being
> > running fine since.
> >
> > Recently I am experimenting with nutch 1.0. Everything worked great and
> > better until I start to use MergeSegments.  I was merging segments with
> > around 20k urls and it gave me OutOfMemoryError. I have tried to
> > increase the java heap max to 3G, I still got OutOfMemoryError.  In
> > contrast, in my older version of nutch,  same merge works with the
> > default java heap max setting of only 1G.
> >
> > Dose anybody have the same experience? Is there any work around this?
> >
> > Thanks
> > Kevin Chen
> >
>
>

[Subhojit Roy]

We had encountered a similar issue once that got solved by increasing the
swap space on out Linux machine. Did you try doing that?

-sroy

On Sun, Nov 8, 2009 at 10:01 AM, kevin chen <kevinchen@...> wrote:

> Hi, I have using a trunk version of nutch since Jul 2007. It's being
> running fine since.
>
> Recently I am experimenting with nutch 1.0. Everything worked great and
> better until I start to use MergeSegments.  I was merging segments with
> around 20k urls and it gave me OutOfMemoryError. I have tried to
> increase the java heap max to 3G, I still got OutOfMemoryError.  In
> contrast, in my older version of nutch,  same merge works with the
> default java heap max setting of only 1G.
>
> Dose anybody have the same experience? Is there any work around this?
>
> Thanks
> Kevin Chen
>
>

--
Subhojit Roy
Profound Technologies
(Search Solutions based on Open Source)
email: sroy@...
http://www.profound.in