Nabble - Nutch - User

Re: Nutch indexes less pages, then it fetches

2009-11-27T05:56:30Z

Does anybody know how to solve this problem?

Re: 100 fetches per second?

2009-11-27T01:35:28Z

MilleBii wrote:

> Interesting updates on the current run of 450K urls :
> + 30minutes @ 3Mbits/s
> + drop to 1Mbit/s (1/X shape)
> + gradual improvement to 1.5 Mbit/s and steady for 7 hours
> + sudden drop to 0.9 Mbits/s and steady for 4 hours
> + up to 1.7 Mbits for 1hour
> + staircasing down to 0.5 Mbit/s by steps of 1 hour
>
> I don't know what to take as a conclusion, but it is quite strange to have
> those sudden variation of bandwidth and overall very slow.
> I can post the graph if people are interested.

This most likely comes from the allocation of urls to map tasks, and the
maximum number of map tasks that you can run on your cluster. when tasks
finish their run, you see a sudden drop in speed, until the next task
starts running. Initially, I suspect that you have more tasks available
than the capacity of your cluster, so it's easy to fill the slots and
max the speed. Later on, slow map tasks tend to hang around, but still
some of them finish and make space for new tasks. As time goes on,
majority of your tasks becomes slow tasks, so the overall speed
continues to drop down.

--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com

Re: Encoding the content got from Fetcher

2009-11-27T01:16:55Z

I had already tried with:

<property>
<name>parser.character.encoding.default</name>
<value>UTF-8</value>
<description>The character encoding to fall back to when no other information
is available</description>
</property>

and System.out.println(content.toString());
is still the HTML code with the incorrect encoding...

Re: Encoding the content got from Fetcher

2009-11-27T00:45:46Z

Santiago Pérez wrote:

> Yes, I tried in that configuration file setting with the latin encoding
> Windows-1250, but the value of this property does not affect to the encoding
> of the content (I also tried with unexistent encoding and the result is the
> same...)
>
> <property>
> <name>parser.character.encoding.default</name>
> <value>Windows-1250</value>
> <description>The character encoding to fall back to when no other
> information
> is available</description>
> </property>
>
> Has anyone had the same problem? (Hungarian o Polish people sure...)

The appearance of characters that you quoted in your other email
indicates that the problem may be the opposite - your pages seem to use
UTF-8, and you are trying to convert them using Windows-1250 ... Try
putting UTF-8 in this property, and see what happens.

Generally speaking, pages should declare their encoding, either in HTTP
headers or in <meta> tags, but often this declaration is either missing
or completely wrong. Nutch uses ICU4J CharsetDetector plus its own
heuristic (in util.EncodingDetector and in HtmlParser) that tries to
detect character encoding if it's missing or even if it's wrong - but
this is a tricky issue and sometimes results are unpredictable.

--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com

Re: Encoding the content got from Fetcher

2009-11-27T00:17:04Z

Yes, I tried in that configuration file setting with the latin encoding Windows-1250, but the value of this property does not affect to the encoding of the content (I also tried with unexistent encoding and the result is the same...)

<property>
<name>parser.character.encoding.default</name>
<value>Windows-1250</value>
<description>The character encoding to fall back to when no other information
is available</description>
</property>

Has anyone had the same problem? (Hungarian o Polish people sure...)

Thanks

Re: Nutch near future - strategic directions

2009-11-26T23:50:38Z

Andrzej Bialecki wrote:
> Sami Siren wrote:
>> Lots of good thoughts and ideas, easy to agree with.
>>
>> Something for the "ease of use" category:
>> -allow running on top of plain vanilla hadoop
>
> What does it mean "plain vanilla" here? Do you mean the current DB
> implementation? That's the idea, we should aim for an abstract layer
> that can accommodate both HBase and plain MapFile-s.

I was simply trying to say that we should not bundle Hadoop anymore with
Nutch and instead just mention the specific version it should run on top
of as a requirement. I am not totally sure anymore if this is a good idea...

I do not know details about the HBase branch. Would using HBase allow us
easy migration from one data model to another (without complex code we
now have in our datums). How easy is HBase to manage/setup/configure?

I think Avro looks promising as a data storage technology: has some
support for data model evolution, can be accessed "natively" from many
programming languages, is relatively well performing... The downside at
the moment is that it is not yet fully supported by hadoop mapred (I think).

>> -split into reusable components with nice and clean public api
>> -publish mvn artifacts so developers can directly use mvn, ivy etc to
>> pull required dependencies for their specific crawler
>
> +1, with slight preference towards ivy.

I was not clear here, I think I was referring to users of Nutch instead
of Developers. And in that case the choise of a tool would be up to the
user after the artifacts are in the repo.

Also, I think what I wanted to day is more about the model how would
people that want to do some customization operate instead of a
technology choice.

Creating new plugin:
-create your own build configuration (or use a template we provide)
-implement plugin code
-publish to m2 repository

Creating your custom crawler:
-create your own build configuration (or use a template we might
provide), specify the dependencies you need (plugins basically, from
apache or from anybody else as long as they are available through some
repository)
-potentially write some custom code

We could also still provide a "default" Nutch crawler also, as a build
configuration (basically just xml file + some config) if we wanted.

The new Hadoop maven artifacts also help with this vision since we could
also access hadoop apis (and dependencies) through similar mechanism.

>> My biggest concern is in execution of this (or any other) plan.
>> Some of the changes or improvements that have been proposed are quite
>> "heavy" in nature and would require large changes. I am just thinking
>> that would it still be better to take a fresh start instead of trying
>> to do this incrementally on top of existing code base.
>
> Well ... that's (almost) what Dogacan did with the HBase port. I agree
> that we should not feel too constrained by the existing code base, but
> it would be silly to throw everything away and start from scratch - we
> need to find a middle ground. The crawler-commons and Tika projects
> should help us to get rid of the ballast and significantly reduce the
> size of our code.

I am not aiming to throw everything away, just trying to relax the back
compatibility burden and give "innovation" a chance.

>> In the history of Nutch this approach is not something new (remember
>> map reduce?) and in my opinion it worked nicely then. Perhaps it is
>> different this time since the changes we are discussing now have many
>> abstract things hanging in the air, even fundamental ones.
>
> Nutch 0.7 to 0.8 reused a lot of the existing code.

I am hoping that this time it will not be different.

>>
>> Of course the rewrite approach means that it will take some time
>> before we actually get into the point where we can start adding real
>> substance (meaning new features etc).
>>
>> So to summarize, I would go ahead and put together a branch "nutch
>> N.0" that would consist of (a.k.a my wish list, hope I am not being
>> too aggressive here):
>>
>> -runs on top of plain hadoop
>
> See above - what do you mean by that?
>
>> -use osgi (or some other more optimal extension mechanism that fits
>> and is easy to use)
>> -basic http/https crawling functionality (with "db abstraction" or
>> hbase directly and smart data structures that allow flexible and
>> efficient usage of the data)
>> -basic solr integration for indexing/search
>> -basic parsing with tika
>>
>> After the basics are ok we would start adding and promoting any of the
>> hidden gems we might have, or some solutions for the interesting
>> challenges.
>
> I believe that's more or less where Dogacan's port is right now, except
> it's not merged with the OSGI port.

Are you sure OSGI is the way to go? I Know it has all these nice
features and all but for some reason I feel that we could live with
something simpler. From functional pow: just drop your jars info
classpath and you're all set. So 2 changes here: 1. plugins are jars 2.
no individual classloaders for plugins.

--
Sami Siren

Re: 100 fetches per second?

2009-11-26T22:52:47Z

Interesting updates on the current run of 450K urls :
+ 30minutes @ 3Mbits/s
+ drop to 1Mbit/s (1/X shape)
+ gradual improvement to 1.5 Mbit/s and steady for 7 hours
+ sudden drop to 0.9 Mbits/s and steady for 4 hours
+ up to 1.7 Mbits for 1hour
+ staircasing down to 0.5 Mbit/s by steps of 1 hour

I don't know what to take as a conclusion, but it is quite strange to have
those sudden variation of bandwidth and overall very slow.
I can post the graph if people are interested.

2009/11/26 MilleBii <millebii@...>

> Yep, I will try right after this run ends... Which is likely tomorrow
> by the sound of it.
>
> Still how come there is a factor 6+ difference from one run to the
> next ... Timing hosts blocking the queue maybe, but the probability to
> get one in the queue can not be so different from one run to run.
>
> 2009/11/26, Otis Gospodnetic <ogjunk-nutch@...>:
> > I think in the end what Ken Krugler did with Bixo (limiting crawl time)
> and
> > what Julien added in https://issues.apache.org/jira/browse/NUTCH-770(plus
> > https://issues.apache.org/jira/browse/NUTCH-769) are solutions to this
> > problem, in addition to what Andrzej described below.
> >
> > Can you try https://issues.apache.org/jira/browse/NUTCH-770 and
> > https://issues.apache.org/jira/browse/NUTCH-769 ?
> >
> > Otis
> > --
> > Sematext is hiring -- http://sematext.com/about/jobs.html?mls
> > Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
> >
> >
> >
> > ----- Original Message ----
> >> From: Andrzej Bialecki <ab@...>
> >> To: nutch-user@...
> >> Sent: Wed, November 25, 2009 6:13:07 PM
> >> Subject: Re: 100 fetches per second?
> >>
> >> MilleBii wrote:
> >> > I have to say that I'm still puzzled. Here is the latest. I just
> >> > restarted a
> >> > run and then guess what :
> >> >
> >> > got ultra-high speed : 8Mbits/s sustained for 1 hour where I could
> only
> >> > get
> >> > 3Mbit/s max before (nota bits and not bytes as I said before).
> >> > A few samples show that I was running at 50 Fetches/sec ... not bad.
> But
> >> > why
> >> > this high-speed on this run I haven't got the faintest idea.
> >> >
> >> >
> >> > Than it drops and I get that kind of logs
> >> >
> >> > 2009-11-25 23:28:28,584 INFO fetcher.Fetcher - -activeThreads=100,
> >> > spinWaiting=100, fetchQueues.totalSize=516
> >> > 2009-11-25 23:28:29,227 INFO fetcher.Fetcher - -activeThreads=100,
> >> > spinWaiting=100, fetchQueues.totalSize=120
> >> > 2009-11-25 23:28:29,584 INFO fetcher.Fetcher - -activeThreads=100,
> >> > spinWaiting=100, fetchQueues.totalSize=516
> >> > 2009-11-25 23:28:30,227 INFO fetcher.Fetcher - -activeThreads=100,
> >> > spinWaiting=100, fetchQueues.totalSize=120
> >> > 2009-11-25 23:28:30,585 INFO fetcher.Fetcher - -activeThreads=100,
> >> > spinWaiting=100, fetchQueues.totalSize=516
> >> >
> >> > Don't fully understand why it is oscillating between two queue size
> >> > never
> >> > mind.... but it is likely the end of the run since hadoop shows 99.99%
> >> > percent complete for the 2 map it generated.
> >> >
> >> > Would that be explained by a better URL mix ????
> >>
> >> I suspect that you have a bunch of hosts that slowly trickle the
> content,
> >> i.e.
> >> requests don't time out, crawl-delay is low, but the download speed is
> >> very very
> >> low due to the limits at their end (either physical or artificial).
> >>
> >> The solution in that case would be to track a minimum avg. speed per
> >> FetchQueue,
> >> and lock-out the queue if this number crosses the threshold (similarly
> to
> >> what
> >> we do when we discover a crawl-delay that is too high).
> >>
> >> In the meantime, you could add the number of FetchQueue-s to that
> >> diagnostic
> >> output, to see how many unique hosts are in the current working set.
> >>
> >> -- Best regards,
> >> Andrzej Bialecki <><
> >> ___. ___ ___ ___ _ _ __________________________________
> >> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> >> ___|||__|| \| || | Embedded Unix, System Integration
> >> http://www.sigram.com Contact: info at sigram dot com
> >
> >
>
>
> --
> -MilleBii-
>

--
-MilleBii-

add parse-wml plugin to Nutch!

2009-11-26T17:22:54Z

hi,
i have to add parse-wml plugin to Nutch, if it has been finished,pls
give me some advise.

Tks!

Re: Encoding the content got from Fetcher

2009-11-26T15:11:11Z

hi

have you tried to change this property:

parser.character.encoding.default

>
> Hej,
>
> I am a newbie in Nutch and I need some help with a problem because I do
> not
> find clear documentation.
>
> In crawling proccess when the each of the FetcherThread get the content,
> this is in formatted in a way which deletes the new line characters ("\n")
> and transform useful characters in Spanish as Ã¡,Ã©,Ã,Ã³,Ãº,Ã±,Ã¼ in the
> default
> encoding like: Ã?ÃÂ¡, Ã?ÃÂ³, Ã?ÃÂ, Ã?ÃÂ³, Ã?ÃÂº, Ã?ÃÂ±,
> Ã?ÃÂ¼.
>
> I would like to know if it is possible to set this default encoding (is
> UTF-8?) to the one that I need (ASCII I guess).
>
> Thanks in advance ;)
> --
> View this message in context:
> http://old.nabble.com/Encoding-the-content-got-from-Fetcher-tp26528468p26528468.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
>

Re: Broken segments ?

2009-11-26T12:34:03Z

Mischa Tuffield wrote:
> Hello All,

http://people.apache.org/~hossman/#threadhijack

"When starting a new discussion on a mailing list, please do not reply
to an existing message, instead start a fresh email. Even if you change
the subject line of your email, other mail headers still track which
thread you replied to and your question is "hidden" in that thread and
gets less attention. It makes following discussions in the mailing
list archives particularly difficult."

--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com

Broken segments ?

2009-11-26T11:54:48Z

Hello All,

I was wondering if there is any way to check the integrity of a segment? As it stands, I can't create the index I want due to a number of my segments freaking out like below :

Is there anyway to check if my segments are OK, I guess i could always re:fetch them if need be.

Regards, and thanks in advance :)

Mischa



On 26 Nov 2009, at 12:03, Santiago Pérez wrote:

>
> Hej,
>
> I am a newbie in Nutch and I need some help with a problem because I do not
> find clear documentation.
>
> In crawling proccess when the each of the FetcherThread get the content,
> this is in formatted in a way which deletes the new line characters ("\n")
> and transform useful characters in Spanish as á,é,í,ó,ú,ñ,ü in the default
> encoding like: Ã?Â¡, Ã?Â³, Ã?Â , Ã?Â³, Ã?Âº, Ã?Â±, Ã?Â¼.
>
> I would like to know if it is possible to set this default encoding (is
> UTF-8?) to the one that I need (ASCII I guess).
>
> Thanks in advance ;)
> --
> View this message in context: http://old.nabble.com/Encoding-the-content-got-from-Fetcher-tp26528468p26528468.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>

___________________________________
Mischa Tuffield
Email: mischa.tuffield@...
Homepage - http://mmt.me.uk/
Garlik Limited, 2 Sheen Road, Richmond, TW9 1AE, UK
+44(0)20 8973 2465 http://www.garlik.com/
Registered in England and Wales 535 7233 VAT # 849 0517 11
Registered office: Thames House, Portsmouth Road, Esher, Surrey, KT10 9AD

Re: 100 fetches per second?

2009-11-26T11:32:56Z

Yep, I will try right after this run ends... Which is likely tomorrow
by the sound of it.

Still how come there is a factor 6+ difference from one run to the
next ... Timing hosts blocking the queue maybe, but the probability to
get one in the queue can not be so different from one run to run.

2009/11/26, Otis Gospodnetic <ogjunk-nutch@...>:

> I think in the end what Ken Krugler did with Bixo (limiting crawl time) and
> what Julien added in https://issues.apache.org/jira/browse/NUTCH-770 (plus
> https://issues.apache.org/jira/browse/NUTCH-769) are solutions to this
> problem, in addition to what Andrzej described below.
>
> Can you try https://issues.apache.org/jira/browse/NUTCH-770 and
> https://issues.apache.org/jira/browse/NUTCH-769 ?
>
> Otis
> --
> Sematext is hiring -- http://sematext.com/about/jobs.html?mls
> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
>
>
>
> ----- Original Message ----
>> From: Andrzej Bialecki <ab@...>
>> To: nutch-user@...
>> Sent: Wed, November 25, 2009 6:13:07 PM
>> Subject: Re: 100 fetches per second?
>>
>> MilleBii wrote:
>> > I have to say that I'm still puzzled. Here is the latest. I just
>> > restarted a
>> > run and then guess what :
>> >
>> > got ultra-high speed : 8Mbits/s sustained for 1 hour where I could only
>> > get
>> > 3Mbit/s max before (nota bits and not bytes as I said before).
>> > A few samples show that I was running at 50 Fetches/sec ... not bad. But
>> > why
>> > this high-speed on this run I haven't got the faintest idea.
>> >
>> >
>> > Than it drops and I get that kind of logs
>> >
>> > 2009-11-25 23:28:28,584 INFO fetcher.Fetcher - -activeThreads=100,
>> > spinWaiting=100, fetchQueues.totalSize=516
>> > 2009-11-25 23:28:29,227 INFO fetcher.Fetcher - -activeThreads=100,
>> > spinWaiting=100, fetchQueues.totalSize=120
>> > 2009-11-25 23:28:29,584 INFO fetcher.Fetcher - -activeThreads=100,
>> > spinWaiting=100, fetchQueues.totalSize=516
>> > 2009-11-25 23:28:30,227 INFO fetcher.Fetcher - -activeThreads=100,
>> > spinWaiting=100, fetchQueues.totalSize=120
>> > 2009-11-25 23:28:30,585 INFO fetcher.Fetcher - -activeThreads=100,
>> > spinWaiting=100, fetchQueues.totalSize=516
>> >
>> > Don't fully understand why it is oscillating between two queue size
>> > never
>> > mind.... but it is likely the end of the run since hadoop shows 99.99%
>> > percent complete for the 2 map it generated.
>> >
>> > Would that be explained by a better URL mix ????
>>
>> I suspect that you have a bunch of hosts that slowly trickle the content,
>> i.e.
>> requests don't time out, crawl-delay is low, but the download speed is
>> very very
>> low due to the limits at their end (either physical or artificial).
>>
>> The solution in that case would be to track a minimum avg. speed per
>> FetchQueue,
>> and lock-out the queue if this number crosses the threshold (similarly to
>> what
>> we do when we discover a crawl-delay that is too high).
>>
>> In the meantime, you could add the number of FetchQueue-s to that
>> diagnostic
>> output, to see how many unique hosts are in the current working set.
>>
>> -- Best regards,
>> Andrzej Bialecki <><
>> ___. ___ ___ ___ _ _ __________________________________
>> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
>> ___|||__|| \| || | Embedded Unix, System Integration
>> http://www.sigram.com Contact: info at sigram dot com
>
>

--
-MilleBii-

Encoding the content got from Fetcher

2009-11-26T04:03:52Z

Hej,

I am a newbie in Nutch and I need some help with a problem because I do not find clear documentation.

In crawling proccess when the each of the FetcherThread get the content, this is in formatted in a way which deletes the new line characters ("\n") and transform useful characters in Spanish as á,é,í,ó,ú,ñ,ü in the default encoding like: Ã?Â¡, Ã?Â³, Ã?Â, Ã?Â³, Ã?Âº, Ã?Â±, Ã?Â¼.

I would like to know if it is possible to set this default encoding (is UTF-8?) to the one that I need (ASCII I guess).

Thanks in advance ;)

Re: 100 fetches per second?

2009-11-26T03:36:57Z

I think in the end what Ken Krugler did with Bixo (limiting crawl time) and what Julien added in https://issues.apache.org/jira/browse/NUTCH-770 (plus https://issues.apache.org/jira/browse/NUTCH-769) are solutions to this problem, in addition to what Andrzej described below.

Can you try https://issues.apache.org/jira/browse/NUTCH-770 and https://issues.apache.org/jira/browse/NUTCH-769 ?

Otis
--
Sematext is hiring -- http://sematext.com/about/jobs.html?mls
Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR

----- Original Message ----

> From: Andrzej Bialecki <ab@...>
> To: nutch-user@...
> Sent: Wed, November 25, 2009 6:13:07 PM
> Subject: Re: 100 fetches per second?
>
> MilleBii wrote:
> > I have to say that I'm still puzzled. Here is the latest. I just restarted a
> > run and then guess what :
> >
> > got ultra-high speed : 8Mbits/s sustained for 1 hour where I could only get
> > 3Mbit/s max before (nota bits and not bytes as I said before).
> > A few samples show that I was running at 50 Fetches/sec ... not bad. But why
> > this high-speed on this run I haven't got the faintest idea.
> >
> >
> > Than it drops and I get that kind of logs
> >
> > 2009-11-25 23:28:28,584 INFO fetcher.Fetcher - -activeThreads=100,
> > spinWaiting=100, fetchQueues.totalSize=516
> > 2009-11-25 23:28:29,227 INFO fetcher.Fetcher - -activeThreads=100,
> > spinWaiting=100, fetchQueues.totalSize=120
> > 2009-11-25 23:28:29,584 INFO fetcher.Fetcher - -activeThreads=100,
> > spinWaiting=100, fetchQueues.totalSize=516
> > 2009-11-25 23:28:30,227 INFO fetcher.Fetcher - -activeThreads=100,
> > spinWaiting=100, fetchQueues.totalSize=120
> > 2009-11-25 23:28:30,585 INFO fetcher.Fetcher - -activeThreads=100,
> > spinWaiting=100, fetchQueues.totalSize=516
> >
> > Don't fully understand why it is oscillating between two queue size never
> > mind.... but it is likely the end of the run since hadoop shows 99.99%
> > percent complete for the 2 map it generated.
> >
> > Would that be explained by a better URL mix ????
>
> I suspect that you have a bunch of hosts that slowly trickle the content, i.e.
> requests don't time out, crawl-delay is low, but the download speed is very very
> low due to the limits at their end (either physical or artificial).
>
> The solution in that case would be to track a minimum avg. speed per FetchQueue,
> and lock-out the queue if this number crosses the threshold (similarly to what
> we do when we discover a crawl-delay that is too high).
>
> In the meantime, you could add the number of FetchQueue-s to that diagnostic
> output, to see how many unique hosts are in the current working set.
>
> -- Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com

remove fields

2009-11-26T02:31:05Z

hi all,

there are 4 document fields in my index that i am not indexing anymore;

then i have 4 new fields i need to add to my index, so i created a new
indexing filter.

how i can add these new fields while preserving the removed fields in
the existing docs?

at the moment when i run bin/index all non-indexed fields get removed
from the index;

Re: 100 fetches per second?

2009-11-25T23:11:01Z

Dennis,

Interesting info, I don't use the standard OPIC scorer but a slightly
modified version which boost pages with content that I'm looking for... so
it could be that my pages are generally on slow servers.

Now heads-up, just started a new run with 450k URLs and it looks like I'm
back to the previous behaviour :
+ 4 Mb/s for a few minutes
+ steady 1.9 Mb/s ... for ages probably since it really means around 10-15
Fetch/s

Why did the previous run go so fast ???? I'm still wondering

2009/11/26 Dennis Kubes <kubes@...>

> One interesting thing we were seeing a while back on large crawls where we
> were fetching the best scoring pages first, then next best, and so on, is
> that lower scoring pages typically had worse response time rates and worse
> timeout rates.
>
> So while the best scoring pages would respond very quickly and would have <
> 1% timeout rate, the worst scoring pages would take x times as long (don't
> remember the exact ratio but it was multiples) and could have as high as a
> 50% timeout rate. Just something to think about.
>
> Dennis Kubes
>
>
> Andrzej Bialecki wrote:
>
>> MilleBii wrote:
>>
>>> I have to say that I'm still puzzled. Here is the latest. I just
>>> restarted a
>>> run and then guess what :
>>>
>>> got ultra-high speed : 8Mbits/s sustained for 1 hour where I could only
>>> get
>>> 3Mbit/s max before (nota bits and not bytes as I said before).
>>> A few samples show that I was running at 50 Fetches/sec ... not bad. But
>>> why
>>> this high-speed on this run I haven't got the faintest idea.
>>>
>>>
>>> Than it drops and I get that kind of logs
>>>
>>> 2009-11-25 23:28:28,584 INFO fetcher.Fetcher - -activeThreads=100,
>>> spinWaiting=100, fetchQueues.totalSize=516
>>> 2009-11-25 23:28:29,227 INFO fetcher.Fetcher - -activeThreads=100,
>>> spinWaiting=100, fetchQueues.totalSize=120
>>> 2009-11-25 23:28:29,584 INFO fetcher.Fetcher - -activeThreads=100,
>>> spinWaiting=100, fetchQueues.totalSize=516
>>> 2009-11-25 23:28:30,227 INFO fetcher.Fetcher - -activeThreads=100,
>>> spinWaiting=100, fetchQueues.totalSize=120
>>> 2009-11-25 23:28:30,585 INFO fetcher.Fetcher - -activeThreads=100,
>>> spinWaiting=100, fetchQueues.totalSize=516
>>>
>>> Don't fully understand why it is oscillating between two queue size never
>>> mind.... but it is likely the end of the run since hadoop shows 99.99%
>>> percent complete for the 2 map it generated.
>>>
>>> Would that be explained by a better URL mix ????
>>>
>>
>> I suspect that you have a bunch of hosts that slowly trickle the content,
>> i.e. requests don't time out, crawl-delay is low, but the download speed is
>> very very low due to the limits at their end (either physical or
>> artificial).
>>
>> The solution in that case would be to track a minimum avg. speed per
>> FetchQueue, and lock-out the queue if this number crosses the threshold
>> (similarly to what we do when we discover a crawl-delay that is too high).
>>
>> In the meantime, you could add the number of FetchQueue-s to that
>> diagnostic output, to see how many unique hosts are in the current working
>> set.
>>
>>

--
-MilleBii-

Re: 100 fetches per second?

2009-11-25T23:01:24Z

Did not think of that one... interesting

how & where do you control the number of FetchQueues I only use the default
so I assume there is only one.

How I should do if I want to analyze the content of a generated fetchlist ?

Is it possible to increase the number of fetcher on a single node
configuration ? If not than I may turn to a configuration with two low specs
servers vs one middle range... I will get more out my bucks ;-)

2009/11/26 Andrzej Bialecki <ab@...>

> MilleBii wrote:
>
>> I have to say that I'm still puzzled. Here is the latest. I just restarted
>> a
>> run and then guess what :
>>
>> got ultra-high speed : 8Mbits/s sustained for 1 hour where I could only
>> get
>> 3Mbit/s max before (nota bits and not bytes as I said before).
>> A few samples show that I was running at 50 Fetches/sec ... not bad. But
>> why
>> this high-speed on this run I haven't got the faintest idea.
>>
>>
>> Than it drops and I get that kind of logs
>>
>> 2009-11-25 23:28:28,584 INFO fetcher.Fetcher - -activeThreads=100,
>> spinWaiting=100, fetchQueues.totalSize=516
>> 2009-11-25 23:28:29,227 INFO fetcher.Fetcher - -activeThreads=100,
>> spinWaiting=100, fetchQueues.totalSize=120
>> 2009-11-25 23:28:29,584 INFO fetcher.Fetcher - -activeThreads=100,
>> spinWaiting=100, fetchQueues.totalSize=516
>> 2009-11-25 23:28:30,227 INFO fetcher.Fetcher - -activeThreads=100,
>> spinWaiting=100, fetchQueues.totalSize=120
>> 2009-11-25 23:28:30,585 INFO fetcher.Fetcher - -activeThreads=100,
>> spinWaiting=100, fetchQueues.totalSize=516
>>
>> Don't fully understand why it is oscillating between two queue size never
>> mind.... but it is likely the end of the run since hadoop shows 99.99%
>> percent complete for the 2 map it generated.
>>
>> Would that be explained by a better URL mix ????
>>
>
> I suspect that you have a bunch of hosts that slowly trickle the content,
> i.e. requests don't time out, crawl-delay is low, but the download speed is
> very very low due to the limits at their end (either physical or
> artificial).
>
> The solution in that case would be to track a minimum avg. speed per
> FetchQueue, and lock-out the queue if this number crosses the threshold
> (similarly to what we do when we discover a crawl-delay that is too high).
>
> In the meantime, you could add the number of FetchQueue-s to that
> diagnostic output, to see how many unique hosts are in the current working
> set.
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
>

--
-MilleBii-

Re: Exception while slicing and parsing old segments without fetching

2009-11-25T21:17:29Z

Hi Vishal,

I got the same prolem while runing updatedb and invertlinks.
Have you got the solution to the problem?
Please let me know if u get the solution.

Thank You,
Srinivas

On Mon, Aug 24, 2009 at 2:00 PM, vishal vachhani <vishal.ce@...>wrote:

> Hi All,
> I had a big segment(size= 25 GB). Using "mergesegs utility and
> slice=20000" , I have divided the segment into around 400 small segments. I
> re-paresed(using parse command) all the segments because we have made
> changes into the parsing modules of Nutch. Parsing was completed
> successfully for all segments. Linkdb is also generated successfully.
>
> I have following questions.
>
> 1. Do I need to run "Updatedb" on the parsed segments again? When I run
> Updatedb command on these segments, I am getting following exception.
>
> ----------------------------------------------------------------------------------------
> 2009-08-17 20:09:33,679 WARN fs.FSInputChecker - Problem reading checksum
> file: java.io.EOFException. Ignoring.
> 2009-08-17 20:09:33,700 WARN mapred.LocalJobRunner - job_fmwtmv
> java.lang.RuntimeException: Summer buffer overflow b.len=4096, off=0,
> summed=3584, read=4096, bytesPerSum=1, inSum=512
> at
>
> org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.readBuffer(ChecksumFileSystem.java:201)
> at
>
> org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.read(ChecksumFileSystem.java:167)
> at
>
> org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:41)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at
>
> org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:57)
> at
> org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:91)
> at
> org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1525)
> at
> org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1436)
> at
> org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1482)
> at
>
> org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:73)
> at org.apache.hadoop.mapred.MapTask$1.next(MapTask.java:157)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:46)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:175)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:126)
> Caused by: java.lang.ArrayIndexOutOfBoundsException
> at java.util.zip.CRC32.update(CRC32.java:43)
> at
>
> org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.readBuffer(ChecksumFileSystem.java:199)
> ... 16 more
> 2009-08-17 20:09:33,749 FATAL crawl.CrawlDb - CrawlDb update:
> java.io.IOException: Job failed!
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:604)
> at org.apache.nutch.crawl.CrawlDb.update(CrawlDb.java:97)
> at org.apache.nutch.crawl.CrawlDb.run(CrawlDb.java:199)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:189)
> at org.apache.nutch.crawl.CrawlDb.main(CrawlDb.java:152)
>
> -----------------------------------------------------------------------------------------------------------------------------
>
> 2. When I run the "index" command on the segments,crawldb and linkdb, I am
> getting "java heap space" error. While with single big segments and same
> configuration of Java heap, we were able to index the segments. Are we
> doing
> something wrong? We will be thankful if somebody could give us some
> pointers
> in the problems.
>
> --------------------------------------------------------------------------------
>
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2786)
> at
> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.Text.writeString(Text.java:399)
> at org.apache.nutch.metadata.Metadata.write(Metadata.java:225)
> at org.apache.nutch.parse.ParseData.write(ParseData.java:165)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:154)
> at org.apache.hadoop.io.ObjectWritable.write(ObjectWritable.java:65)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:315)
> at org.apache.nutch.indexer.Indexer.map(Indexer.java:362)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:175)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:126)
> 2009-08-23 23:19:28,569 FATAL indexer.Indexer - Indexer:
> java.io.IOException: Job failed!
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:604)
> at org.apache.nutch.indexer.Indexer.index(Indexer.java:329)
> at org.apache.nutch.indexer.Indexer.run(Indexer.java:351)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:189)
> at org.apache.nutch.indexer.Indexer.main(Indexer.java:334)
>
> ----------------------------------------------------------------------------------------------------------------
>
>
> --
> Thanks and Regards,
> Vishal Vachhani
>

--
http://cheyuta.wordpress.com

Re: 100 fetches per second?

2009-11-25T16:42:20Z

One interesting thing we were seeing a while back on large crawls where
we were fetching the best scoring pages first, then next best, and so
on, is that lower scoring pages typically had worse response time rates
and worse timeout rates.

So while the best scoring pages would respond very quickly and would
have < 1% timeout rate, the worst scoring pages would take x times as
long (don't remember the exact ratio but it was multiples) and could
have as high as a 50% timeout rate. Just something to think about.

Dennis Kubes

Andrzej Bialecki wrote:

> MilleBii wrote:
>> I have to say that I'm still puzzled. Here is the latest. I just
>> restarted a
>> run and then guess what :
>>
>> got ultra-high speed : 8Mbits/s sustained for 1 hour where I could
>> only get
>> 3Mbit/s max before (nota bits and not bytes as I said before).
>> A few samples show that I was running at 50 Fetches/sec ... not bad.
>> But why
>> this high-speed on this run I haven't got the faintest idea.
>>
>>
>> Than it drops and I get that kind of logs
>>
>> 2009-11-25 23:28:28,584 INFO fetcher.Fetcher - -activeThreads=100,
>> spinWaiting=100, fetchQueues.totalSize=516
>> 2009-11-25 23:28:29,227 INFO fetcher.Fetcher - -activeThreads=100,
>> spinWaiting=100, fetchQueues.totalSize=120
>> 2009-11-25 23:28:29,584 INFO fetcher.Fetcher - -activeThreads=100,
>> spinWaiting=100, fetchQueues.totalSize=516
>> 2009-11-25 23:28:30,227 INFO fetcher.Fetcher - -activeThreads=100,
>> spinWaiting=100, fetchQueues.totalSize=120
>> 2009-11-25 23:28:30,585 INFO fetcher.Fetcher - -activeThreads=100,
>> spinWaiting=100, fetchQueues.totalSize=516
>>
>> Don't fully understand why it is oscillating between two queue size never
>> mind.... but it is likely the end of the run since hadoop shows 99.99%
>> percent complete for the 2 map it generated.
>>
>> Would that be explained by a better URL mix ????
>
> I suspect that you have a bunch of hosts that slowly trickle the
> content, i.e. requests don't time out, crawl-delay is low, but the
> download speed is very very low due to the limits at their end (either
> physical or artificial).
>
> The solution in that case would be to track a minimum avg. speed per
> FetchQueue, and lock-out the queue if this number crosses the threshold
> (similarly to what we do when we discover a crawl-delay that is too high).
>
> In the meantime, you could add the number of FetchQueue-s to that
> diagnostic output, to see how many unique hosts are in the current
> working set.
>

Re: 100 fetches per second?

2009-11-25T15:13:07Z

MilleBii wrote:

> I have to say that I'm still puzzled. Here is the latest. I just restarted a
> run and then guess what :
>
> got ultra-high speed : 8Mbits/s sustained for 1 hour where I could only get
> 3Mbit/s max before (nota bits and not bytes as I said before).
> A few samples show that I was running at 50 Fetches/sec ... not bad. But why
> this high-speed on this run I haven't got the faintest idea.
>
>
> Than it drops and I get that kind of logs
>
> 2009-11-25 23:28:28,584 INFO fetcher.Fetcher - -activeThreads=100,
> spinWaiting=100, fetchQueues.totalSize=516
> 2009-11-25 23:28:29,227 INFO fetcher.Fetcher - -activeThreads=100,
> spinWaiting=100, fetchQueues.totalSize=120
> 2009-11-25 23:28:29,584 INFO fetcher.Fetcher - -activeThreads=100,
> spinWaiting=100, fetchQueues.totalSize=516
> 2009-11-25 23:28:30,227 INFO fetcher.Fetcher - -activeThreads=100,
> spinWaiting=100, fetchQueues.totalSize=120
> 2009-11-25 23:28:30,585 INFO fetcher.Fetcher - -activeThreads=100,
> spinWaiting=100, fetchQueues.totalSize=516
>
> Don't fully understand why it is oscillating between two queue size never
> mind.... but it is likely the end of the run since hadoop shows 99.99%
> percent complete for the 2 map it generated.
>
> Would that be explained by a better URL mix ????

I suspect that you have a bunch of hosts that slowly trickle the
content, i.e. requests don't time out, crawl-delay is low, but the
download speed is very very low due to the limits at their end (either
physical or artificial).

The solution in that case would be to track a minimum avg. speed per
FetchQueue, and lock-out the queue if this number crosses the threshold
(similarly to what we do when we discover a crawl-delay that is too high).

In the meantime, you could add the number of FetchQueue-s to that
diagnostic output, to see how many unique hosts are in the current
working set.

--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com

Re: 100 fetches per second?

2009-11-25T14:45:35Z

I have to say that I'm still puzzled. Here is the latest. I just restarted a
run and then guess what :

got ultra-high speed : 8Mbits/s sustained for 1 hour where I could only get
3Mbit/s max before (nota bits and not bytes as I said before).
A few samples show that I was running at 50 Fetches/sec ... not bad. But why
this high-speed on this run I haven't got the faintest idea.

Than it drops and I get that kind of logs

2009-11-25 23:28:28,584 INFO fetcher.Fetcher - -activeThreads=100,
spinWaiting=100, fetchQueues.totalSize=516
2009-11-25 23:28:29,227 INFO fetcher.Fetcher - -activeThreads=100,
spinWaiting=100, fetchQueues.totalSize=120
2009-11-25 23:28:29,584 INFO fetcher.Fetcher - -activeThreads=100,
spinWaiting=100, fetchQueues.totalSize=516
2009-11-25 23:28:30,227 INFO fetcher.Fetcher - -activeThreads=100,
spinWaiting=100, fetchQueues.totalSize=120
2009-11-25 23:28:30,585 INFO fetcher.Fetcher - -activeThreads=100,
spinWaiting=100, fetchQueues.totalSize=516

Don't fully understand why it is oscillating between two queue size never
mind.... but it is likely the end of the run since hadoop shows 99.99%
percent complete for the 2 map it generated.

Would that be explained by a better URL mix ????

2009/11/25 Mark Kerzner <markkerzner@...>

> Judging by how this discussion goes, there may be a need for URL mix
> optimizer and for a fast crawler based on that. Is this something worth
> pursuing. MilleBii, q'en pensez vous?
>
> Mark
>
> On Wed, Nov 25, 2009 at 3:44 PM, MilleBii <millebii@...> wrote:
>
> > The logs show that my fetch queue is full and my 100 threads are mostly
> > spin
> > waiting towards the end.
> >
> > Now the very last run (150kURLs) I can clearly see 4 phases:
> > + very high speed : 3MB/s for a few minutes
> > + sudden speed drop around 1MB/s and flat for several hours
> > + another speed drop to around 400kB/s for several hours
> > + another speed drop to around 200kB/s for a few hours two.
> >
> > So probably it is just a consequence of the url mix which isn't that good
> > nota: I have limited to 1000 URLS per host, and there are about 20-30
> hosts
> > in the mix which get limited that way.
> >
> > May be there is better mix of URLs possible ?
> >
> > 2009/11/25 Julien Nioche <lists.digitalpebble@...>
> >
> > > or it is stuck on a couple of hosts which time out? The logs should
> have
> > a
> > > trace with the number of active threads, which should give some
> > indication
> > > of what's happening.
> > >
> > > Julien
> > >
> > >
> > > 2009/11/25 Dennis Kubes <kubes@...>
> > >
> > > > If it is waiting and the box is idle, my first though is not dns. I
> > just
> > > > put that up as one of the things people will run into. Most likely
> it
> > is
> > > > uneven distribution of urls or something like that.
> > > >
> > > > Dennis
> > > >
> > > >
> > > > MilleBii wrote:
> > > >
> > > >> Get your point... Although I thought high number of threads would do
> > > >> exactly the same. Maybe I miss something.
> > > >>
> > > >> During my fetcher runs used bandwidth gets low pretty quickly, disk
> > > >> I/O is low, the CPU is low... So it must be waiting for something
> but
> > > >> what ?
> > > >>
> > > >> Could be the DNS cache wich is full and any new request gets
> forwarded
> > > >> to the master DNS of my ISP,
> > > >> Any idea how to check that ? I'm not familiar with Bind myself...
> What
> > > >> is the typical rate you can get how many dns request/s ?
> > > >>
> > > >>
> > > >>
> > > >> 2009/11/25, Dennis Kubes <kubes@...>:
> > > >>
> > > >>> It is not about the local DNS caching as much as having local DNS
> > > >>> servers. Too many fetchers hitting a centralized DNS server can
> act
> > as
> > > >>> a DOS attack and slow down the entire fetching system.
> > > >>>
> > > >>> For example say I have a single centralized DNS server for my
> > network.
> > > >>> And say I have 2 map task per machine, 50 machines, 20 threads per
> > > task.
> > > >>> That would be 50 * 2 * 20 = 2000 fetchers. Meaning a possibility
> of
> > > >>> 2000 DNS requests / sec. Most local DNS servers for smaller
> > networks
> > > >>> can't handle that. If everything is hitting a centralized DNS and
> > that
> > > >>> DNS takes 1-3 sec per request because of too many requests. The
> > entire
> > > >>> fetching system stalls.
> > > >>>
> > > >>> Hitting a secondary larger cache, such as OpenDNS, can have an
> effect
> > > >>> because you are making one hop to get the name versus multiple hops
> > to
> > > >>> root servers then domain servers.
> > > >>>
> > > >>> Working off of a single server these issues don't show up as much
> > > >>> because there aren't enough fetchers.
> > > >>>
> > > >>> Dennis Kubes
> > > >>>
> > > >>> MilleBii wrote:
> > > >>>
> > > >>>> Why would DNS local caching work... It only is working if you are
> > > >>>> going to crawl often the same site ... In which case you are hit
> by
> > > >>>> the politeness.
> > > >>>>
> > > >>>> if you have segments with only/mainly different sites it is
> > not/really
> > > >>>> going to help.
> > > >>>>
> > > >>>> So far I have not seen my quad core + 100mb/s + pseudo distributed
> > > >>>> hadoop going faster than 10 fetch / s... Let me check the DNS and
> I
> > > >>>> will tell you.
> > > >>>>
> > > >>>> I vote for 100 Fetch/s not sure how to get it though
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> 2009/11/24, Dennis Kubes <kubes@...>:
> > > >>>>
> > > >>>>> Hi Mark,
> > > >>>>>
> > > >>>>> I just put this up on the wiki. Hope it helps:
> > > >>>>>
> > > >>>>> http://wiki.apache.org/nutch/OptimizingCrawls
> > > >>>>>
> > > >>>>> Dennis
> > > >>>>>
> > > >>>>>
> > > >>>>> Mark Kerzner wrote:
> > > >>>>>
> > > >>>>>> Hi, guys,
> > > >>>>>>
> > > >>>>>> my goal is to do by crawls at 100 fetches per second, observing,
> > of
> > > >>>>>> course,
> > > >>>>>> polite crawling. But, when URLs are all different domains, what
> > > >>>>>> theoretically would stop some software from downloading from 100
> > > >>>>>> domains
> > > >>>>>> at
> > > >>>>>> once, achieving the desired speed?
> > > >>>>>>
> > > >>>>>> But, whatever I do, I can't make Nutch crawl at that speed. Even
> > if
> > > it
> > > >>>>>> starts at a few dozen URLs/second, it slows down at the end (as
> > > >>>>>> discussed
> > > >>>>>> by
> > > >>>>>> many and by Krugler).
> > > >>>>>>
> > > >>>>>> Should I write something of my own, or are their fast crawlers?
> > > >>>>>>
> > > >>>>>> Thanks!
> > > >>>>>>
> > > >>>>>> Mark
> > > >>>>>>
> > > >>>>>>
> > > >>
> > >
> > >
> > > --
> > > DigitalPebble Ltd
> > > http://www.digitalpebble.com
> > >
> >
> >
> >
> > --
> > -MilleBii-
> >
>

--
-MilleBii-

Re: 100 fetches per second?

2009-11-25T13:50:31Z

Judging by how this discussion goes, there may be a need for URL mix
optimizer and for a fast crawler based on that. Is this something worth
pursuing. MilleBii, q'en pensez vous?

Mark

On Wed, Nov 25, 2009 at 3:44 PM, MilleBii <millebii@...> wrote:

> The logs show that my fetch queue is full and my 100 threads are mostly
> spin
> waiting towards the end.
>
> Now the very last run (150kURLs) I can clearly see 4 phases:
> + very high speed : 3MB/s for a few minutes
> + sudden speed drop around 1MB/s and flat for several hours
> + another speed drop to around 400kB/s for several hours
> + another speed drop to around 200kB/s for a few hours two.
>
> So probably it is just a consequence of the url mix which isn't that good
> nota: I have limited to 1000 URLS per host, and there are about 20-30 hosts
> in the mix which get limited that way.
>
> May be there is better mix of URLs possible ?
>
> 2009/11/25 Julien Nioche <lists.digitalpebble@...>
>
> > or it is stuck on a couple of hosts which time out? The logs should have
> a
> > trace with the number of active threads, which should give some
> indication
> > of what's happening.
> >
> > Julien
> >
> >
> > 2009/11/25 Dennis Kubes <kubes@...>
> >
> > > If it is waiting and the box is idle, my first though is not dns. I
> just
> > > put that up as one of the things people will run into. Most likely it
> is
> > > uneven distribution of urls or something like that.
> > >
> > > Dennis
> > >
> > >
> > > MilleBii wrote:
> > >
> > >> Get your point... Although I thought high number of threads would do
> > >> exactly the same. Maybe I miss something.
> > >>
> > >> During my fetcher runs used bandwidth gets low pretty quickly, disk
> > >> I/O is low, the CPU is low... So it must be waiting for something but
> > >> what ?
> > >>
> > >> Could be the DNS cache wich is full and any new request gets forwarded
> > >> to the master DNS of my ISP,
> > >> Any idea how to check that ? I'm not familiar with Bind myself... What
> > >> is the typical rate you can get how many dns request/s ?
> > >>
> > >>
> > >>
> > >> 2009/11/25, Dennis Kubes <kubes@...>:
> > >>
> > >>> It is not about the local DNS caching as much as having local DNS
> > >>> servers. Too many fetchers hitting a centralized DNS server can act
> as
> > >>> a DOS attack and slow down the entire fetching system.
> > >>>
> > >>> For example say I have a single centralized DNS server for my
> network.
> > >>> And say I have 2 map task per machine, 50 machines, 20 threads per
> > task.
> > >>> That would be 50 * 2 * 20 = 2000 fetchers. Meaning a possibility of
> > >>> 2000 DNS requests / sec. Most local DNS servers for smaller
> networks
> > >>> can't handle that. If everything is hitting a centralized DNS and
> that
> > >>> DNS takes 1-3 sec per request because of too many requests. The
> entire
> > >>> fetching system stalls.
> > >>>
> > >>> Hitting a secondary larger cache, such as OpenDNS, can have an effect
> > >>> because you are making one hop to get the name versus multiple hops
> to
> > >>> root servers then domain servers.
> > >>>
> > >>> Working off of a single server these issues don't show up as much
> > >>> because there aren't enough fetchers.
> > >>>
> > >>> Dennis Kubes
> > >>>
> > >>> MilleBii wrote:
> > >>>
> > >>>> Why would DNS local caching work... It only is working if you are
> > >>>> going to crawl often the same site ... In which case you are hit by
> > >>>> the politeness.
> > >>>>
> > >>>> if you have segments with only/mainly different sites it is
> not/really
> > >>>> going to help.
> > >>>>
> > >>>> So far I have not seen my quad core + 100mb/s + pseudo distributed
> > >>>> hadoop going faster than 10 fetch / s... Let me check the DNS and I
> > >>>> will tell you.
> > >>>>
> > >>>> I vote for 100 Fetch/s not sure how to get it though
> > >>>>
> > >>>>
> > >>>>
> > >>>> 2009/11/24, Dennis Kubes <kubes@...>:
> > >>>>
> > >>>>> Hi Mark,
> > >>>>>
> > >>>>> I just put this up on the wiki. Hope it helps:
> > >>>>>
> > >>>>> http://wiki.apache.org/nutch/OptimizingCrawls
> > >>>>>
> > >>>>> Dennis
> > >>>>>
> > >>>>>
> > >>>>> Mark Kerzner wrote:
> > >>>>>
> > >>>>>> Hi, guys,
> > >>>>>>
> > >>>>>> my goal is to do by crawls at 100 fetches per second, observing,
> of
> > >>>>>> course,
> > >>>>>> polite crawling. But, when URLs are all different domains, what
> > >>>>>> theoretically would stop some software from downloading from 100
> > >>>>>> domains
> > >>>>>> at
> > >>>>>> once, achieving the desired speed?
> > >>>>>>
> > >>>>>> But, whatever I do, I can't make Nutch crawl at that speed. Even
> if
> > it
> > >>>>>> starts at a few dozen URLs/second, it slows down at the end (as
> > >>>>>> discussed
> > >>>>>> by
> > >>>>>> many and by Krugler).
> > >>>>>>
> > >>>>>> Should I write something of my own, or are their fast crawlers?
> > >>>>>>
> > >>>>>> Thanks!
> > >>>>>>
> > >>>>>> Mark
> > >>>>>>
> > >>>>>>
> > >>
> >
> >
> > --
> > DigitalPebble Ltd
> > http://www.digitalpebble.com
> >
>
>
>
> --
> -MilleBii-
>

Re: 100 fetches per second?

2009-11-25T13:44:19Z

The logs show that my fetch queue is full and my 100 threads are mostly spin
waiting towards the end.

Now the very last run (150kURLs) I can clearly see 4 phases:
+ very high speed : 3MB/s for a few minutes
+ sudden speed drop around 1MB/s and flat for several hours
+ another speed drop to around 400kB/s for several hours
+ another speed drop to around 200kB/s for a few hours two.

So probably it is just a consequence of the url mix which isn't that good
nota: I have limited to 1000 URLS per host, and there are about 20-30 hosts
in the mix which get limited that way.

May be there is better mix of URLs possible ?

2009/11/25 Julien Nioche <lists.digitalpebble@...>

> or it is stuck on a couple of hosts which time out? The logs should have a
> trace with the number of active threads, which should give some indication
> of what's happening.
>
> Julien
>
>
> 2009/11/25 Dennis Kubes <kubes@...>
>
> > If it is waiting and the box is idle, my first though is not dns. I just
> > put that up as one of the things people will run into. Most likely it is
> > uneven distribution of urls or something like that.
> >
> > Dennis
> >
> >
> > MilleBii wrote:
> >
> >> Get your point... Although I thought high number of threads would do
> >> exactly the same. Maybe I miss something.
> >>
> >> During my fetcher runs used bandwidth gets low pretty quickly, disk
> >> I/O is low, the CPU is low... So it must be waiting for something but
> >> what ?
> >>
> >> Could be the DNS cache wich is full and any new request gets forwarded
> >> to the master DNS of my ISP,
> >> Any idea how to check that ? I'm not familiar with Bind myself... What
> >> is the typical rate you can get how many dns request/s ?
> >>
> >>
> >>
> >> 2009/11/25, Dennis Kubes <kubes@...>:
> >>
> >>> It is not about the local DNS caching as much as having local DNS
> >>> servers. Too many fetchers hitting a centralized DNS server can act as
> >>> a DOS attack and slow down the entire fetching system.
> >>>
> >>> For example say I have a single centralized DNS server for my network.
> >>> And say I have 2 map task per machine, 50 machines, 20 threads per
> task.
> >>> That would be 50 * 2 * 20 = 2000 fetchers. Meaning a possibility of
> >>> 2000 DNS requests / sec. Most local DNS servers for smaller networks
> >>> can't handle that. If everything is hitting a centralized DNS and that
> >>> DNS takes 1-3 sec per request because of too many requests. The entire
> >>> fetching system stalls.
> >>>
> >>> Hitting a secondary larger cache, such as OpenDNS, can have an effect
> >>> because you are making one hop to get the name versus multiple hops to
> >>> root servers then domain servers.
> >>>
> >>> Working off of a single server these issues don't show up as much
> >>> because there aren't enough fetchers.
> >>>
> >>> Dennis Kubes
> >>>
> >>> MilleBii wrote:
> >>>
> >>>> Why would DNS local caching work... It only is working if you are
> >>>> going to crawl often the same site ... In which case you are hit by
> >>>> the politeness.
> >>>>
> >>>> if you have segments with only/mainly different sites it is not/really
> >>>> going to help.
> >>>>
> >>>> So far I have not seen my quad core + 100mb/s + pseudo distributed
> >>>> hadoop going faster than 10 fetch / s... Let me check the DNS and I
> >>>> will tell you.
> >>>>
> >>>> I vote for 100 Fetch/s not sure how to get it though
> >>>>
> >>>>
> >>>>
> >>>> 2009/11/24, Dennis Kubes <kubes@...>:
> >>>>
> >>>>> Hi Mark,
> >>>>>
> >>>>> I just put this up on the wiki. Hope it helps:
> >>>>>
> >>>>> http://wiki.apache.org/nutch/OptimizingCrawls
> >>>>>
> >>>>> Dennis
> >>>>>
> >>>>>
> >>>>> Mark Kerzner wrote:
> >>>>>
> >>>>>> Hi, guys,
> >>>>>>
> >>>>>> my goal is to do by crawls at 100 fetches per second, observing, of
> >>>>>> course,
> >>>>>> polite crawling. But, when URLs are all different domains, what
> >>>>>> theoretically would stop some software from downloading from 100
> >>>>>> domains
> >>>>>> at
> >>>>>> once, achieving the desired speed?
> >>>>>>
> >>>>>> But, whatever I do, I can't make Nutch crawl at that speed. Even if
> it
> >>>>>> starts at a few dozen URLs/second, it slows down at the end (as
> >>>>>> discussed
> >>>>>> by
> >>>>>> many and by Krugler).
> >>>>>>
> >>>>>> Should I write something of my own, or are their fast crawlers?
> >>>>>>
> >>>>>> Thanks!
> >>>>>>
> >>>>>> Mark
> >>>>>>
> >>>>>>
> >>
>
>
> --
> DigitalPebble Ltd
> http://www.digitalpebble.com
>

--
-MilleBii-

Re: 100 fetches per second?

2009-11-25T10:46:09Z

or it is stuck on a couple of hosts which time out? The logs should have a
trace with the number of active threads, which should give some indication
of what's happening.

Julien

2009/11/25 Dennis Kubes <kubes@...>

> If it is waiting and the box is idle, my first though is not dns. I just
> put that up as one of the things people will run into. Most likely it is
> uneven distribution of urls or something like that.
>
> Dennis
>
>
> MilleBii wrote:
>
>> Get your point... Although I thought high number of threads would do
>> exactly the same. Maybe I miss something.
>>
>> During my fetcher runs used bandwidth gets low pretty quickly, disk
>> I/O is low, the CPU is low... So it must be waiting for something but
>> what ?
>>
>> Could be the DNS cache wich is full and any new request gets forwarded
>> to the master DNS of my ISP,
>> Any idea how to check that ? I'm not familiar with Bind myself... What
>> is the typical rate you can get how many dns request/s ?
>>
>>
>>
>> 2009/11/25, Dennis Kubes <kubes@...>:
>>
>>> It is not about the local DNS caching as much as having local DNS
>>> servers. Too many fetchers hitting a centralized DNS server can act as
>>> a DOS attack and slow down the entire fetching system.
>>>
>>> For example say I have a single centralized DNS server for my network.
>>> And say I have 2 map task per machine, 50 machines, 20 threads per task.
>>> That would be 50 * 2 * 20 = 2000 fetchers. Meaning a possibility of
>>> 2000 DNS requests / sec. Most local DNS servers for smaller networks
>>> can't handle that. If everything is hitting a centralized DNS and that
>>> DNS takes 1-3 sec per request because of too many requests. The entire
>>> fetching system stalls.
>>>
>>> Hitting a secondary larger cache, such as OpenDNS, can have an effect
>>> because you are making one hop to get the name versus multiple hops to
>>> root servers then domain servers.
>>>
>>> Working off of a single server these issues don't show up as much
>>> because there aren't enough fetchers.
>>>
>>> Dennis Kubes
>>>
>>> MilleBii wrote:
>>>
>>>> Why would DNS local caching work... It only is working if you are
>>>> going to crawl often the same site ... In which case you are hit by
>>>> the politeness.
>>>>
>>>> if you have segments with only/mainly different sites it is not/really
>>>> going to help.
>>>>
>>>> So far I have not seen my quad core + 100mb/s + pseudo distributed
>>>> hadoop going faster than 10 fetch / s... Let me check the DNS and I
>>>> will tell you.
>>>>
>>>> I vote for 100 Fetch/s not sure how to get it though
>>>>
>>>>
>>>>
>>>> 2009/11/24, Dennis Kubes <kubes@...>:
>>>>
>>>>> Hi Mark,
>>>>>
>>>>> I just put this up on the wiki. Hope it helps:
>>>>>
>>>>> http://wiki.apache.org/nutch/OptimizingCrawls
>>>>>
>>>>> Dennis
>>>>>
>>>>>
>>>>> Mark Kerzner wrote:
>>>>>
>>>>>> Hi, guys,
>>>>>>
>>>>>> my goal is to do by crawls at 100 fetches per second, observing, of
>>>>>> course,
>>>>>> polite crawling. But, when URLs are all different domains, what
>>>>>> theoretically would stop some software from downloading from 100
>>>>>> domains
>>>>>> at
>>>>>> once, achieving the desired speed?
>>>>>>
>>>>>> But, whatever I do, I can't make Nutch crawl at that speed. Even if it
>>>>>> starts at a few dozen URLs/second, it slows down at the end (as
>>>>>> discussed
>>>>>> by
>>>>>> many and by Krugler).
>>>>>>
>>>>>> Should I write something of my own, or are their fast crawlers?
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> Mark
>>>>>>
>>>>>>
>>

--
DigitalPebble Ltd
http://www.digitalpebble.com

Re: 100 fetches per second?

2009-11-25T10:34:10Z

If it is waiting and the box is idle, my first though is not dns. I
just put that up as one of the things people will run into. Most likely
it is uneven distribution of urls or something like that.

Dennis

MilleBii wrote:

> Get your point... Although I thought high number of threads would do
> exactly the same. Maybe I miss something.
>
> During my fetcher runs used bandwidth gets low pretty quickly, disk
> I/O is low, the CPU is low... So it must be waiting for something but
> what ?
>
> Could be the DNS cache wich is full and any new request gets forwarded
> to the master DNS of my ISP,
> Any idea how to check that ? I'm not familiar with Bind myself... What
> is the typical rate you can get how many dns request/s ?
>
>
>
> 2009/11/25, Dennis Kubes <kubes@...>:
>> It is not about the local DNS caching as much as having local DNS
>> servers. Too many fetchers hitting a centralized DNS server can act as
>> a DOS attack and slow down the entire fetching system.
>>
>> For example say I have a single centralized DNS server for my network.
>> And say I have 2 map task per machine, 50 machines, 20 threads per task.
>> That would be 50 * 2 * 20 = 2000 fetchers. Meaning a possibility of
>> 2000 DNS requests / sec. Most local DNS servers for smaller networks
>> can't handle that. If everything is hitting a centralized DNS and that
>> DNS takes 1-3 sec per request because of too many requests. The entire
>> fetching system stalls.
>>
>> Hitting a secondary larger cache, such as OpenDNS, can have an effect
>> because you are making one hop to get the name versus multiple hops to
>> root servers then domain servers.
>>
>> Working off of a single server these issues don't show up as much
>> because there aren't enough fetchers.
>>
>> Dennis Kubes
>>
>> MilleBii wrote:
>>> Why would DNS local caching work... It only is working if you are
>>> going to crawl often the same site ... In which case you are hit by
>>> the politeness.
>>>
>>> if you have segments with only/mainly different sites it is not/really
>>> going to help.
>>>
>>> So far I have not seen my quad core + 100mb/s + pseudo distributed
>>> hadoop going faster than 10 fetch / s... Let me check the DNS and I
>>> will tell you.
>>>
>>> I vote for 100 Fetch/s not sure how to get it though
>>>
>>>
>>>
>>> 2009/11/24, Dennis Kubes <kubes@...>:
>>>> Hi Mark,
>>>>
>>>> I just put this up on the wiki. Hope it helps:
>>>>
>>>> http://wiki.apache.org/nutch/OptimizingCrawls
>>>>
>>>> Dennis
>>>>
>>>>
>>>> Mark Kerzner wrote:
>>>>> Hi, guys,
>>>>>
>>>>> my goal is to do by crawls at 100 fetches per second, observing, of
>>>>> course,
>>>>> polite crawling. But, when URLs are all different domains, what
>>>>> theoretically would stop some software from downloading from 100 domains
>>>>> at
>>>>> once, achieving the desired speed?
>>>>>
>>>>> But, whatever I do, I can't make Nutch crawl at that speed. Even if it
>>>>> starts at a few dozen URLs/second, it slows down at the end (as
>>>>> discussed
>>>>> by
>>>>> many and by Krugler).
>>>>>
>>>>> Should I write something of my own, or are their fast crawlers?
>>>>>
>>>>> Thanks!
>>>>>
>>>>> Mark
>>>>>
>

Re: 100 fetches per second?

2009-11-25T09:56:46Z

Get your point... Although I thought high number of threads would do
exactly the same. Maybe I miss something.

During my fetcher runs used bandwidth gets low pretty quickly, disk
I/O is low, the CPU is low... So it must be waiting for something but
what ?

Could be the DNS cache wich is full and any new request gets forwarded
to the master DNS of my ISP,
Any idea how to check that ? I'm not familiar with Bind myself... What
is the typical rate you can get how many dns request/s ?

2009/11/25, Dennis Kubes <kubes@...>:

> It is not about the local DNS caching as much as having local DNS
> servers. Too many fetchers hitting a centralized DNS server can act as
> a DOS attack and slow down the entire fetching system.
>
> For example say I have a single centralized DNS server for my network.
> And say I have 2 map task per machine, 50 machines, 20 threads per task.
> That would be 50 * 2 * 20 = 2000 fetchers. Meaning a possibility of
> 2000 DNS requests / sec. Most local DNS servers for smaller networks
> can't handle that. If everything is hitting a centralized DNS and that
> DNS takes 1-3 sec per request because of too many requests. The entire
> fetching system stalls.
>
> Hitting a secondary larger cache, such as OpenDNS, can have an effect
> because you are making one hop to get the name versus multiple hops to
> root servers then domain servers.
>
> Working off of a single server these issues don't show up as much
> because there aren't enough fetchers.
>
> Dennis Kubes
>
> MilleBii wrote:
>> Why would DNS local caching work... It only is working if you are
>> going to crawl often the same site ... In which case you are hit by
>> the politeness.
>>
>> if you have segments with only/mainly different sites it is not/really
>> going to help.
>>
>> So far I have not seen my quad core + 100mb/s + pseudo distributed
>> hadoop going faster than 10 fetch / s... Let me check the DNS and I
>> will tell you.
>>
>> I vote for 100 Fetch/s not sure how to get it though
>>
>>
>>
>> 2009/11/24, Dennis Kubes <kubes@...>:
>>> Hi Mark,
>>>
>>> I just put this up on the wiki. Hope it helps:
>>>
>>> http://wiki.apache.org/nutch/OptimizingCrawls
>>>
>>> Dennis
>>>
>>>
>>> Mark Kerzner wrote:
>>>> Hi, guys,
>>>>
>>>> my goal is to do by crawls at 100 fetches per second, observing, of
>>>> course,
>>>> polite crawling. But, when URLs are all different domains, what
>>>> theoretically would stop some software from downloading from 100 domains
>>>> at
>>>> once, achieving the desired speed?
>>>>
>>>> But, whatever I do, I can't make Nutch crawl at that speed. Even if it
>>>> starts at a few dozen URLs/second, it slows down at the end (as
>>>> discussed
>>>> by
>>>> many and by Krugler).
>>>>
>>>> Should I write something of my own, or are their fast crawlers?
>>>>
>>>> Thanks!
>>>>
>>>> Mark
>>>>
>>
>

--
Envoyé avec mon mobile

-MilleBii-

recrawl.sh stopped at depth 7/10 without error

2009-11-25T07:43:33Z

hi,

i'm running recrawl.sh and it stops every time at depth 7/10 without any error ! but when run the bin/crawl with the same crawl-urlfilter and the same seeds file it finishs softly in 1h50

i checked the hadoop.log, and dont find any error there...i just find the last url it was parsing
do fetching or crawling has a timeout ?
my recrawl takes 2 hours before it stops. i set the time fetch interval 24 hours and i'm running the generate with adddays = 1

best regards

_________________________________________________________________
Eligible CDN College & University students can upgrade to Windows 7 before Jan 3 for only $39.99. Upgrade now!
http://go.microsoft.com/?linkid=9691819

Re: dedup dont delete duplicates !

2009-11-25T07:37:24Z

Ok, my bad.

M
On 25 Nov 2009, at 15:35, BELLINI ADAM wrote:

>
> plz mischa, if your problem is not about delete duplicate just open another thread ! thx
>
>
> Andrzej, thx for all, i will try to run a diff command on the content of the 2 pages.
> i will give you news when done.
>
>
>
>
>> From: mischa.tuffield@...
>> Subject: Re: dedup dont delete duplicates !
>> Date: Wed, 25 Nov 2009 11:45:21 +0000
>> To: nutch-user@...
>>
>> Hello All,
>>
>> I am getting the following error in my hadoop.log (see below). It seems to happen everytime I run any of the nutch command line tools :(
>>
>> 
>>
>> Does anyone know what problem I am having ?
>>
>> Cheers,
>>
>> Mischa
>>
>> On 25 Nov 2009, at 09:15, Andrzej Bialecki wrote:
>>
>>> BELLINI ADAM wrote:
>>>> hi,
>>>> my two urls points to the same page !
>>>
>>> Please, no need to shout ...
>>>
>>> If the MD5 signatures are different, then the binary content of these pages is different, period.
>>>
>>> Use readseg -dump utility to retrieve the page content from the segment, extract just the two pages from the dump, and run a unix diff utility.
>>>
>>>> can you tell m eplz more about TextProfileSignature ? how should i
>>>> use it
>>>
>>> Configure this type of signature in your nutch-site.xml - please see the nutch-default.xml for instructions. Please note that you will have to re-parse segments and update the db in order to update the signatures.
>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Andrzej Bialecki <><
>>> ___. ___ ___ ___ _ _ __________________________________
>>> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
>>> ___|||__|| \| || | Embedded Unix, System Integration
>>> http://www.sigram.com Contact: info at sigram dot com
>>>
>>
>> ___________________________________
>> Mischa Tuffield
>> Email: mischa.tuffield@...
>> Homepage - http://mmt.me.uk/
>> Garlik Limited, 2 Sheen Road, Richmond, TW9 1AE, UK
>> +44(0)20 8973 2465 http://www.garlik.com/
>> Registered in England and Wales 535 7233 VAT # 849 0517 11
>> Registered office: Thames House, Portsmouth Road, Esher, Surrey, KT10 9AD
>>
>
> _________________________________________________________________
> Eligible CDN College & University students can upgrade to Windows 7 before Jan 3 for only $39.99. Upgrade now!
> http://go.microsoft.com/?linkid=9691819

RE: dedup dont delete duplicates !

2009-11-25T07:35:26Z

plz mischa, if your problem is not about delete duplicate just open another thread ! thx

Andrzej, thx for all, i will try to run a diff command on the content of the 2 pages.
i will give you news when done.

> From: mischa.tuffield@...
> Subject: Re: dedup dont delete duplicates !
> Date: Wed, 25 Nov 2009 11:45:21 +0000
> To: nutch-user@...
>
> Hello All,
>
> I am getting the following error in my hadoop.log (see below). It seems to happen everytime I run any of the nutch command line tools :(
>
> 
>
> Does anyone know what problem I am having ?
>
> Cheers,
>
> Mischa
>
> On 25 Nov 2009, at 09:15, Andrzej Bialecki wrote:
>
> > BELLINI ADAM wrote:
> >> hi,
> >> my two urls points to the same page !
> >
> > Please, no need to shout ...
> >
> > If the MD5 signatures are different, then the binary content of these pages is different, period.
> >
> > Use readseg -dump utility to retrieve the page content from the segment, extract just the two pages from the dump, and run a unix diff utility.
> >
> >> can you tell m eplz more about TextProfileSignature ? how should i
> >> use it
> >
> > Configure this type of signature in your nutch-site.xml - please see the nutch-default.xml for instructions. Please note that you will have to re-parse segments and update the db in order to update the signatures.
> >
> >
> >
> > --
> > Best regards,
> > Andrzej Bialecki <><
> > ___. ___ ___ ___ _ _ __________________________________
> > [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> > ___|||__|| \| || | Embedded Unix, System Integration
> > http://www.sigram.com Contact: info at sigram dot com
> >
>
> ___________________________________
> Mischa Tuffield
> Email: mischa.tuffield@...
> Homepage - http://mmt.me.uk/
> Garlik Limited, 2 Sheen Road, Richmond, TW9 1AE, UK
> +44(0)20 8973 2465 http://www.garlik.com/
> Registered in England and Wales 535 7233 VAT # 849 0517 11
> Registered office: Thames House, Portsmouth Road, Esher, Surrey, KT10 9AD
>

_________________________________________________________________
Eligible CDN College & University students can upgrade to Windows 7 before Jan 3 for only $39.99. Upgrade now!
http://go.microsoft.com/?linkid=9691819

Re: Nutch config IOException

2009-11-25T05:15:03Z

Hi Andrzej,

Yeah, I just noticed that this stack trace is for DEBUG purposes only I found it in the hadoop src, thanks for the info.

Regards,

Mischa
On 25 Nov 2009, at 13:11, Andrzej Bialecki wrote:

> Mischa Tuffield wrote:
>> Hello Again, Following my previous post below, I have noticed that I get the following IOException every time I atttempt to use nutch. 
>> Any pointers would be great, I wonder is there a way for me to validate my conf options before I deploy nutch?
>
> This exception is innocuous - it helps to debug at which points in the code the Configuration instances are being created. And you wouldn't have seen this if you didn't turn on the DEBUG logging. ;)
>
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>

Re: Nutch config IOException

2009-11-25T05:11:37Z

Mischa Tuffield wrote:

> Hello Again,
>
> Following my previous post below, I have noticed that I get the following IOException every time I atttempt to use nutch.
>
> 
>
> Any pointers would be great, I wonder is there a way for me to validate my conf options before I deploy nutch?

This exception is innocuous - it helps to debug at which points in the
code the Configuration instances are being created. And you wouldn't
have seen this if you didn't turn on the DEBUG logging. ;)

--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com

Re: 100 fetches per second?

2009-11-25T04:57:05Z

It is not about the local DNS caching as much as having local DNS
servers. Too many fetchers hitting a centralized DNS server can act as
a DOS attack and slow down the entire fetching system.

For example say I have a single centralized DNS server for my network.
And say I have 2 map task per machine, 50 machines, 20 threads per task.
That would be 50 * 2 * 20 = 2000 fetchers. Meaning a possibility of
2000 DNS requests / sec. Most local DNS servers for smaller networks
can't handle that. If everything is hitting a centralized DNS and that
DNS takes 1-3 sec per request because of too many requests. The entire
fetching system stalls.

Hitting a secondary larger cache, such as OpenDNS, can have an effect
because you are making one hop to get the name versus multiple hops to
root servers then domain servers.

Working off of a single server these issues don't show up as much
because there aren't enough fetchers.

Dennis Kubes

MilleBii wrote:

> Why would DNS local caching work... It only is working if you are
> going to crawl often the same site ... In which case you are hit by
> the politeness.
>
> if you have segments with only/mainly different sites it is not/really
> going to help.
>
> So far I have not seen my quad core + 100mb/s + pseudo distributed
> hadoop going faster than 10 fetch / s... Let me check the DNS and I
> will tell you.
>
> I vote for 100 Fetch/s not sure how to get it though
>
>
>
> 2009/11/24, Dennis Kubes <kubes@...>:
>> Hi Mark,
>>
>> I just put this up on the wiki. Hope it helps:
>>
>> http://wiki.apache.org/nutch/OptimizingCrawls
>>
>> Dennis
>>
>>
>> Mark Kerzner wrote:
>>> Hi, guys,
>>>
>>> my goal is to do by crawls at 100 fetches per second, observing, of
>>> course,
>>> polite crawling. But, when URLs are all different domains, what
>>> theoretically would stop some software from downloading from 100 domains
>>> at
>>> once, achieving the desired speed?
>>>
>>> But, whatever I do, I can't make Nutch crawl at that speed. Even if it
>>> starts at a few dozen URLs/second, it slows down at the end (as discussed
>>> by
>>> many and by Krugler).
>>>
>>> Should I write something of my own, or are their fast crawlers?
>>>
>>> Thanks!
>>>
>>> Mark
>>>
>

Nutch config IOException

2009-11-25T04:21:25Z

Hello Again,

Following my previous post below, I have noticed that I get the following IOException every time I atttempt to use nutch.



Any pointers would be great, I wonder is there a way for me to validate my conf options before I deploy nutch?

Regards,

Mischa
On 25 Nov 2009, at 11:45, Mischa Tuffield wrote:

> Hello All,
>
> I am getting the following error in my hadoop.log (see below). It seems to happen everytime I run any of the nutch command line tools :(
>
> 
>
> Does anyone know what problem I am having ?
>
> Cheers,
>
> Mischa
>
> On 25 Nov 2009, at 09:15, Andrzej Bialecki wrote:
>
>> BELLINI ADAM wrote:
>>> hi,
>>> my two urls points to the same page !
>>
>> Please, no need to shout ...
>>
>> If the MD5 signatures are different, then the binary content of these pages is different, period.
>>
>> Use readseg -dump utility to retrieve the page content from the segment, extract just the two pages from the dump, and run a unix diff utility.
>>
>>> can you tell m eplz more about TextProfileSignature ? how should i
>>> use it
>>
>> Configure this type of signature in your nutch-site.xml - please see the nutch-default.xml for instructions. Please note that you will have to re-parse segments and update the db in order to update the signatures.
>>
>>
>>
>> --
>> Best regards,
>> Andrzej Bialecki <><
>> ___. ___ ___ ___ _ _ __________________________________
>> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
>> ___|||__|| \| || | Embedded Unix, System Integration
>> http://www.sigram.com Contact: info at sigram dot com
>>
>
> ___________________________________
> Mischa Tuffield
> Email: mischa.tuffield@...
> Homepage - http://mmt.me.uk/
> Garlik Limited, 2 Sheen Road, Richmond, TW9 1AE, UK
> +44(0)20 8973 2465 http://www.garlik.com/
> Registered in England and Wales 535 7233 VAT # 849 0517 11
> Registered office: Thames House, Portsmouth Road, Esher, Surrey, KT10 9AD
>

Re: dedup dont delete duplicates !

2009-11-25T03:45:21Z

Hello All,

I am getting the following error in my hadoop.log (see below). It seems to happen everytime I run any of the nutch command line tools :(



Does anyone know what problem I am having ?

Cheers,

Mischa

On 25 Nov 2009, at 09:15, Andrzej Bialecki wrote:

> BELLINI ADAM wrote:
>> hi,
>> my two urls points to the same page !
>
> Please, no need to shout ...
>
> If the MD5 signatures are different, then the binary content of these pages is different, period.
>
> Use readseg -dump utility to retrieve the page content from the segment, extract just the two pages from the dump, and run a unix diff utility.
>
>> can you tell m eplz more about TextProfileSignature ? how should i
>> use it
>
> Configure this type of signature in your nutch-site.xml - please see the nutch-default.xml for instructions. Please note that you will have to re-parse segments and update the db in order to update the signatures.
>
>
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>

Re: dedup dont delete duplicates !

2009-11-25T02:52:18Z

Andrzej Bialecki schrieb:

> BELLINI ADAM wrote:
>> hi,
>>
>> my two urls points to the same page !
>
> Please, no need to shout ...
>
> If the MD5 signatures are different, then the binary content of these
> pages is different, period.
>
> Use readseg -dump utility to retrieve the page content from the
> segment, extract just the two pages from the dump, and run a unix diff
> utility.
>
>> can you tell m eplz more about TextProfileSignature ? how should i
>> use it
>
> Configure this type of signature in your nutch-site.xml - please see
> the nutch-default.xml for instructions. Please note that you will have
> to re-parse segments and update the db in order to update the signatures.
>
>
>

http://lucene.apache.org/nutch/apidocs/org/apache/nutch/crawl/TextProfileSignature.html

the class documentation describes how the algorithm works.
it is based on the frequencies of tokens.