tag:old.nabble.com,2006:forum-362Nabble - Nutch2009-12-05T21:26:15ZNutch is web search software. It builds on the Apache Lucene search library, adding a crawler, web database (including full link graph), plugins for various document formats, user interface, etc. Nutch home is <a href="http://lucene.apache.org/nutch/" target="_top" rel="nofollow">here</a>.tag:old.nabble.com,2006:post-26662396RE: Indexing with solrindexer -> OutOfMemoryError2009-12-05T21:26:15Z2009-12-05T21:26:15Zmiagomiago<br>hi,
<br>u have to make your segments smaller than that, just cut every segment in small pieces
<br><br><br><br><div class='shrinkable-quote'><br>> Subject: Indexing with solrindexer -> OutOfMemoryError
<br>> From: <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26662396&i=0" target="_top" rel="nofollow">felizimm@...</a>
<br>> To: <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26662396&i=1" target="_top" rel="nofollow">nutch-user@...</a>
<br>> Date: Sun, 6 Dec 2009 01:35:04 +0100
<br>>
<br>> Hi,
<br>>
<br>> when trying to index four segments (~5 GB) with solrindexer, I get this
<br>> error in hadoop.log. There is no error in the logs of Tomcat, where I
<br>> deployed Solr. I crawled with "crawl"-command.
<br>>
<br>> I`ve read that increasing the hadoop heap space will change nothing.
<br>> What can I do?
<br>>
<br>> Thanks for help!
<br>> Felix.
<br>>
<br>>
<br>> 2009-12-06 00:21:51,061 WARN mapred.LocalJobRunner - job_local_0001
<br>> java.lang.OutOfMemoryError: Java heap space
<br>> at java.util.Arrays.copyOf(Arrays.java:2882)
<br>> at
<br>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
<br>> at
<br>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572)
<br>> at java.lang.StringBuffer.append(StringBuffer.java:320)
<br>> at java.io.StringWriter.write(StringWriter.java:60)
<br>> at org.apache.solr.common.util.XML.escape(XML.java:180)
<br>> at org.apache.solr.common.util.XML.escapeCharData(XML.java:78)
<br>> at org.apache.solr.common.util.XML.writeXML(XML.java:148)
<br>> at
<br>> org.apache.solr.client.solrj.util.ClientUtils.writeXML(ClientUtils.java:117)
<br>> at
<br>> org.apache.solr.client.solrj.request.UpdateRequest.getXML(UpdateRequest.java:169)
<br>> at
<br>> org.apache.solr.client.solrj.request.UpdateRequest.getContentStreams(UpdateRequest.java:160)
<br>> at
<br>> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:191)
<br>> at
<br>> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:183)
<br>> at
<br>> org.apache.solr.client.solrj.request.UpdateRequest.process(UpdateRequest.java:217)
<br>> at
<br>> org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:48)
<br>> at
<br>> org.apache.nutch.indexer.solr.SolrWriter.write(SolrWriter.java:58)
<br>> at org.apache.nutch.indexer.IndexerOutputFormat
<br>> $1.write(IndexerOutputFormat.java:54)
<br>> at org.apache.nutch.indexer.IndexerOutputFormat
<br>> $1.write(IndexerOutputFormat.java:44)
<br>> at org.apache.hadoop.mapred.ReduceTask
<br>> $3.collect(ReduceTask.java:410)
<br>> at
<br>> org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:158)
<br>> at
<br>> org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:50)
<br>> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
<br>> at org.apache.hadoop.mapred.LocalJobRunner
<br>> $Job.run(LocalJobRunner.java:170)
<br>> 2009-12-06 00:21:51,650 FATAL solr.SolrIndexer - SolrIndexer:
<br>> java.io.IOException: Job failed!
<br>> at
<br>> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
<br>> at
<br>> org.apache.nutch.indexer.solr.SolrIndexer.indexSolr(SolrIndexer.java:73)
<br>> at
<br>> org.apache.nutch.indexer.solr.SolrIndexer.run(SolrIndexer.java:95)
<br>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
<br>> at
<br>> org.apache.nutch.indexer.solr.SolrIndexer.main(SolrIndexer.java:104)
<br>>
<br>>
<br>>
</div>
<br>_________________________________________________________________
<br>Windows Live: Make it easier for your friends to see what you’re up to on Facebook.
<br><a href="http://go.microsoft.com/?linkid=9691816" target="_top" rel="nofollow">http://go.microsoft.com/?linkid=9691816</a><p>From forum: <a href="http://old.nabble.com/Nutch---User-f375.html" embed="fixTarget[375]" target="_top" >Nutch - User</a></p>tag:old.nabble.com,2006:post-26660972Indexing with solrindexer -> OutOfMemoryError2009-12-05T16:35:04Z2009-12-05T16:35:04ZFelix Zimmermann-2Hi,
<br><br>when trying to index four segments (~5 GB) with solrindexer, I get this
<br>error in hadoop.log. There is no error in the logs of Tomcat, where I
<br>deployed Solr. I crawled with "crawl"-command.
<br><br>I`ve read that increasing the hadoop heap space will change nothing.
<br>What can I do?
<br><br>Thanks for help!
<br>Felix.
<br><br><br>2009-12-06 00:21:51,061 WARN mapred.LocalJobRunner - job_local_0001
<br>java.lang.OutOfMemoryError: Java heap space
<br> at java.util.Arrays.copyOf(Arrays.java:2882)
<br> at
<br>java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
<br> at
<br>java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572)
<br> at java.lang.StringBuffer.append(StringBuffer.java:320)
<br> at java.io.StringWriter.write(StringWriter.java:60)
<br> at org.apache.solr.common.util.XML.escape(XML.java:180)
<br> at org.apache.solr.common.util.XML.escapeCharData(XML.java:78)
<br> at org.apache.solr.common.util.XML.writeXML(XML.java:148)
<br> at
<br>org.apache.solr.client.solrj.util.ClientUtils.writeXML(ClientUtils.java:117)
<br> at
<br>org.apache.solr.client.solrj.request.UpdateRequest.getXML(UpdateRequest.java:169)
<br> at
<br>org.apache.solr.client.solrj.request.UpdateRequest.getContentStreams(UpdateRequest.java:160)
<br> at
<br>org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:191)
<br> at
<br>org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:183)
<br> at
<br>org.apache.solr.client.solrj.request.UpdateRequest.process(UpdateRequest.java:217)
<br> at
<br>org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:48)
<br> at
<br>org.apache.nutch.indexer.solr.SolrWriter.write(SolrWriter.java:58)
<br> at org.apache.nutch.indexer.IndexerOutputFormat
<br>$1.write(IndexerOutputFormat.java:54)
<br> at org.apache.nutch.indexer.IndexerOutputFormat
<br>$1.write(IndexerOutputFormat.java:44)
<br> at org.apache.hadoop.mapred.ReduceTask
<br>$3.collect(ReduceTask.java:410)
<br> at
<br>org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:158)
<br> at
<br>org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:50)
<br> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
<br> at org.apache.hadoop.mapred.LocalJobRunner
<br>$Job.run(LocalJobRunner.java:170)
<br>2009-12-06 00:21:51,650 FATAL solr.SolrIndexer - SolrIndexer:
<br>java.io.IOException: Job failed!
<br> at
<br>org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
<br> at
<br>org.apache.nutch.indexer.solr.SolrIndexer.indexSolr(SolrIndexer.java:73)
<br> at
<br>org.apache.nutch.indexer.solr.SolrIndexer.run(SolrIndexer.java:95)
<br> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
<br> at
<br>org.apache.nutch.indexer.solr.SolrIndexer.main(SolrIndexer.java:104)
<br><br><br><br><p>From forum: <a href="http://old.nabble.com/Nutch---User-f375.html" embed="fixTarget[375]" target="_top" >Nutch - User</a></p>tag:old.nabble.com,2006:post-26657756Re: HTTP Header problem2009-12-05T09:54:48Z2009-12-05T09:54:48ZKirk GillockThank you for the quick reply, Dennis. It was worth a shot. :-)
<br><br>People are not typically searching for our own name on our own site but,
<br>in case it did happen, we wanted to have the results be as clean as
<br>possible. For our next crawls we'll change the agent name and version to
<br>something else.
<br><br>Thanks again,
<br>Kirk
<br><br><br>Dennis Kubes wrote:
<div class='shrinkable-quote'><br>> There isn't a way to stop this from happening really except to change
<br>> the agent name in the Nutch configuration. When an http request is
<br>> made, the agent name is sent as a header. There are many pages as you
<br>> say that simply have logs of different user-agents hitting their sites
<br>> or have a script to spit back the user agent when a crawler is detected.
<br>>
<br>> Dennis
<br>>
<br>> Kirk Gillock wrote:
<br>>> Hi fellow Nutch users.
<br>>>
<br>>> Long time crawler, first time poster. :-)
<br>>>
<br>>> We're 23m pages into a 100m page crawl and our preliminary tests have
<br>>> shown that a lot of pages contain our agent name, description, etc.,
<br>>> in their page content. Meaning, sites that have a script which show
<br>>> http headers (typically to show browser information) causes the Nutch
<br>>> crawler to store its own header information within the content of
<br>>> that page. So when we search our index for "Isara" (our agent name)
<br>>> we get thousands of results and they all have "Isara/Isara-1.0 (A
<br>>> non-profit search engine benefiting charity.; <a href="http://www.isara.org;" target="_top" rel="nofollow">http://www.isara.org;</a>
<br>>> <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26657756&i=0" target="_top" rel="nofollow">e-mail@...</a>", which is the content of our nutch-default.xml
<br>>> file: http.agent.name, http.agent.description, http.agent.url,
<br>>> http.agent.email, and http.agent.version .
<br>>>
<br>>> I've searched around and haven't found any information on how to stop
<br>>> this from happening. Is there a solution and, if so, will it mean we
<br>>> need to recrawl all those pages again or can we filter the current
<br>>> database? Any suggestions would be greatly appreciated.
<br>>>
<br>>> Thank you for developing such an important open-source application,
<br>>> Kirk Gillock
<br>>> Isara Charity Foundation
<br>>> Nong Khai, Thailand
<br>>> <a href="http://www.isara.org" target="_top" rel="nofollow">http://www.isara.org</a><br>> ------------------------------------------------------------------------
<br>>
<br>>
<br>> No virus found in this incoming message.
<br>> Checked by AVG - www.avg.com
<br>> Version: 8.5.426 / Virus Database: 270.14.95/2546 - Release Date: 12/05/09 08:13:00
<br>>
<br>>
</div><br><p>From forum: <a href="http://old.nabble.com/Nutch---Agent-f372.html" embed="fixTarget[372]" target="_top" >Nutch - Agent</a></p>tag:old.nabble.com,2006:post-26657229[jira] Commented: (NUTCH-770) Timebomb for Fetcher2009-12-05T08:51:21Z2009-12-05T08:51:21ZJIRA jira@apache.org<br> [ <a href="https://issues.apache.org/jira/browse/NUTCH-770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12786443#action_12786443" target="_top" rel="nofollow">https://issues.apache.org/jira/browse/NUTCH-770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12786443#action_12786443</a> ]
<br><br>MilleBii commented on NUTCH-770:
<br>--------------------------------
<br><br>Tried it succesfully on a windows platform.
<br><br>It does not work on a Ubuntu, pseudo-distributed hadoop configuration with mappers running in parallel ????
<br><br><br><div class='shrinkable-quote'><br>> Timebomb for Fetcher
<br>> --------------------
<br>>
<br>> Key: NUTCH-770
<br>> URL: <a href="https://issues.apache.org/jira/browse/NUTCH-770" target="_top" rel="nofollow">https://issues.apache.org/jira/browse/NUTCH-770</a><br>> Project: Nutch
<br>> Issue Type: Improvement
<br>> Reporter: Julien Nioche
<br>> Assignee: Andrzej Bialecki
<br>> Fix For: 1.1
<br>>
<br>> Attachments: log-770, NUTCH-770-v2.patch, NUTCH-770-v3.patch, NUTCH-770.patch
<br>>
<br>>
<br>> This patch provides the Fetcher with a timebomb mechanism. By default the timebomb is not activated; it can be set using the parameter fetcher.timebomb.mins. The number of minutes is relative to the start of the Fetch job. When the number of minutes is reached, the QueueFeeder skips all remaining entries then all active queues are purged. This allows to keep the Fetch step under comtrol and works well in combination with NUTCH-769
</div><br>--
<br>This message is automatically generated by JIRA.
<br>-
<br>You can reply to this email to add a comment to the issue online.
<br><br><p>From forum: <a href="http://old.nabble.com/Nutch---Dev-f373.html" embed="fixTarget[373]" target="_top" >Nutch - Dev</a></p>tag:old.nabble.com,2006:post-26657230[jira] Issue Comment Edited: (NUTCH-770) Timebomb for Fetcher2009-12-05T08:51:21Z2009-12-05T08:51:21ZJIRA jira@apache.org<br> [ <a href="https://issues.apache.org/jira/browse/NUTCH-770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12786443#action_12786443" target="_top" rel="nofollow">https://issues.apache.org/jira/browse/NUTCH-770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12786443#action_12786443</a> ]
<br><br>MilleBii edited comment on NUTCH-770 at 12/5/09 4:50 PM:
<br>---------------------------------------------------------
<br><br>Tried it succesfully on a windows platform.
<br><br>It does not work on a Ubuntu, pseudo-distributed hadoop configuration with two mappers running in parallel ????
<br><br><br><br> was (Author: millebii):
<br> Tried it succesfully on a windows platform.
<br><br>It does not work on a Ubuntu, pseudo-distributed hadoop configuration with mappers running in parallel ????
<br><br><br>
<div class='shrinkable-quote'><br>> Timebomb for Fetcher
<br>> --------------------
<br>>
<br>> Key: NUTCH-770
<br>> URL: <a href="https://issues.apache.org/jira/browse/NUTCH-770" target="_top" rel="nofollow">https://issues.apache.org/jira/browse/NUTCH-770</a><br>> Project: Nutch
<br>> Issue Type: Improvement
<br>> Reporter: Julien Nioche
<br>> Assignee: Andrzej Bialecki
<br>> Fix For: 1.1
<br>>
<br>> Attachments: log-770, NUTCH-770-v2.patch, NUTCH-770-v3.patch, NUTCH-770.patch
<br>>
<br>>
<br>> This patch provides the Fetcher with a timebomb mechanism. By default the timebomb is not activated; it can be set using the parameter fetcher.timebomb.mins. The number of minutes is relative to the start of the Fetch job. When the number of minutes is reached, the QueueFeeder skips all remaining entries then all active queues are purged. This allows to keep the Fetch step under comtrol and works well in combination with NUTCH-769
</div><br>--
<br>This message is automatically generated by JIRA.
<br>-
<br>You can reply to this email to add a comment to the issue online.
<br><br><p>From forum: <a href="http://old.nabble.com/Nutch---Dev-f373.html" embed="fixTarget[373]" target="_top" >Nutch - Dev</a></p>tag:old.nabble.com,2006:post-26656906Re: HTTP Header problem2009-12-05T07:53:01Z2009-12-05T07:53:01ZDennis Kubes-2There isn't a way to stop this from happening really except to change
<br>the agent name in the Nutch configuration. When an http request is
<br>made, the agent name is sent as a header. There are many pages as you
<br>say that simply have logs of different user-agents hitting their sites
<br>or have a script to spit back the user agent when a crawler is detected.
<br><br>Dennis
<br><br>Kirk Gillock wrote:
<div class='shrinkable-quote'><br>> Hi fellow Nutch users.
<br>>
<br>> Long time crawler, first time poster. :-)
<br>>
<br>> We're 23m pages into a 100m page crawl and our preliminary tests have
<br>> shown that a lot of pages contain our agent name, description, etc., in
<br>> their page content. Meaning, sites that have a script which show http
<br>> headers (typically to show browser information) causes the Nutch crawler
<br>> to store its own header information within the content of that page. So
<br>> when we search our index for "Isara" (our agent name) we get thousands
<br>> of results and they all have "Isara/Isara-1.0 (A non-profit search
<br>> engine benefiting charity.; <a href="http://www.isara.org;" target="_top" rel="nofollow">http://www.isara.org;</a> <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26656906&i=0" target="_top" rel="nofollow">e-mail@...</a>",
<br>> which is the content of our nutch-default.xml file: http.agent.name,
<br>> http.agent.description, http.agent.url, http.agent.email, and
<br>> http.agent.version .
<br>>
<br>> I've searched around and haven't found any information on how to stop
<br>> this from happening. Is there a solution and, if so, will it mean we
<br>> need to recrawl all those pages again or can we filter the current
<br>> database? Any suggestions would be greatly appreciated.
<br>>
<br>> Thank you for developing such an important open-source application,
<br>> Kirk Gillock
<br>> Isara Charity Foundation
<br>> Nong Khai, Thailand
<br>> <a href="http://www.isara.org" target="_top" rel="nofollow">http://www.isara.org</a><br></div><p>From forum: <a href="http://old.nabble.com/Nutch---Agent-f372.html" embed="fixTarget[372]" target="_top" >Nutch - Agent</a></p>tag:old.nabble.com,2006:post-26656318State of nutchbase2009-12-05T06:56:49Z2009-12-05T06:56:49ZAlban MoutonHello,<br><br>I have looked a little into nutch code and mailing lists. I think the nutchbase branch (<a href="http://issues.apache.org/jira/browse/NUTCH-650" target="_top" rel="nofollow">http://issues.apache.org/jira/browse/NUTCH-650</a>) is very interesting, with a good potential to improve code clarity and flexibility (I find data structure quite obscure in current version). The issue is untouched since last august, so my question is : can nutchbase really be part of nutch 1.1 ? Is there still much work to do or is it almost ready ? Is it a worthy issue for an interested developer with a (still !) limited knowledge of the project ?<br>
<br>So far I have only tried to run nutchbase in eclipse by applying the tutorial (<a href="http://wiki.apache.org/nutch/RunNutchInEclipse1.0" target="_top" rel="nofollow">http://wiki.apache.org/nutch/RunNutchInEclipse1.0</a>) but I run in errors when building, mostly from Parser and tests. I may start by cleaning this up.<br>
<br>Eclipse build errors:<br><br>Description Resource Path Location Type<br>FetcherOutputFormat cannot be resolved to a type ArcSegmentCreator.java /nutchbase/src/java/org/apache/nutch/tools/arc line 362 Java Problem<br>
Generator.GENERATE_MAX_PER_HOST_BY_IP cannot be resolved TestGenerator.java /nutchbase/src/test/org/apache/nutch/crawl line 202 Java Problem<br>ParseImpl cannot be resolved to a type ArcSegmentCreator.java /nutchbase/src/java/org/apache/nutch/tools/arc line 229 Java Problem<br>
ParseImpl cannot be resolved to a type BasicFields.java /nutchbase/src/java/org/apache/nutch/indexer/field line 335 Java Problem<br>ParseImpl cannot be resolved to a type ExtParser.java /nutchbase/src/plugin/parse-ext/src/java/org/apache/nutch/parse/ext line 138 Java Problem<br>
ParseImpl cannot be resolved to a type MSBaseParser.java /nutchbase/src/plugin/lib-parsems/src/java/org/apache/nutch/parse/ms line 108 Java Problem<br>ParseImpl cannot be resolved to a type OOParser.java /nutchbase/src/plugin/parse-oo/src/java/org/apache/nutch/parse/oo line 103 Java Problem<br>
ParseImpl cannot be resolved to a type PdfParser.java /nutchbase/src/plugin/parse-pdf/src/java/org/apache/nutch/parse/pdf line 155 Java Problem<br>ParseImpl cannot be resolved to a type RSSParser.java /nutchbase/src/plugin/parse-rss/src/java/org/apache/nutch/parse/rss line 187 Java Problem<br>
ParseImpl cannot be resolved to a type SWFParser.java /nutchbase/src/plugin/parse-swf/src/java/org/apache/nutch/parse/swf line 113 Java Problem<br>ParseImpl cannot be resolved to a type TestIndexingFilters.java /nutchbase/src/test/org/apache/nutch/indexer line 45 Java Problem<br>
ParseImpl cannot be resolved to a type TestMoreIndexingFilter.java /nutchbase/src/plugin/index-more/src/test/org/apache/nutch/indexer/more line 61 Java Problem<br>ParseImpl cannot be resolved to a type TextParser.java /nutchbase/src/plugin/parse-text/src/java/org/apache/nutch/parse/text line 55 Java Problem<br>
ParseImpl cannot be resolved to a type ZipParser.java /nutchbase/src/plugin/parse-zip/src/java/org/apache/nutch/parse/zip line 105 Java Problem<br>ParseResult cannot be resolved ExtParser.java /nutchbase/src/plugin/parse-ext/src/java/org/apache/nutch/parse/ext line 137 Java Problem<br>
ParseResult cannot be resolved MSBaseParser.java /nutchbase/src/plugin/lib-parsems/src/java/org/apache/nutch/parse/ms line 107 Java Problem<br>ParseResult cannot be resolved OOParser.java /nutchbase/src/plugin/parse-oo/src/java/org/apache/nutch/parse/oo line 103 Java Problem<br>
ParseResult cannot be resolved PdfParser.java /nutchbase/src/plugin/parse-pdf/src/java/org/apache/nutch/parse/pdf line 155 Java Problem<br>ParseResult cannot be resolved RSSParser.java /nutchbase/src/plugin/parse-rss/src/java/org/apache/nutch/parse/rss line 187 Java Problem<br>
ParseResult cannot be resolved SWFParser.java /nutchbase/src/plugin/parse-swf/src/java/org/apache/nutch/parse/swf line 113 Java Problem<br>ParseResult cannot be resolved TextParser.java /nutchbase/src/plugin/parse-text/src/java/org/apache/nutch/parse/text line 55 Java Problem<br>
ParseResult cannot be resolved ZipParser.java /nutchbase/src/plugin/parse-zip/src/java/org/apache/nutch/parse/zip line 105 Java Problem<br>ParseResult cannot be resolved to a type ArcSegmentCreator.java /nutchbase/src/java/org/apache/nutch/tools/arc line 159 Java Problem<br>
ParseResult cannot be resolved to a type CCParseFilter.java /nutchbase/src/plugin/creativecommons/src/java/org/creativecommons/nutch line 267 Java Problem<br>ParseResult cannot be resolved to a type CCParseFilter.java /nutchbase/src/plugin/creativecommons/src/java/org/creativecommons/nutch line 267 Java Problem<br>
ParseResult cannot be resolved to a type ExtParser.java /nutchbase/src/plugin/parse-ext/src/java/org/apache/nutch/parse/ext line 69 Java Problem<br>ParseResult cannot be resolved to a type FeedParser.java /nutchbase/src/plugin/feed/src/java/org/apache/nutch/parse/feed line 106 Java Problem<br>
ParseResult cannot be resolved to a type FeedParser.java /nutchbase/src/plugin/feed/src/java/org/apache/nutch/parse/feed line 108 Java Problem<br>ParseResult cannot be resolved to a type FeedParser.java /nutchbase/src/plugin/feed/src/java/org/apache/nutch/parse/feed line 108 Java Problem<br>
ParseResult cannot be resolved to a type FeedParser.java /nutchbase/src/plugin/feed/src/java/org/apache/nutch/parse/feed line 211 Java Problem<br>ParseResult cannot be resolved to a type FeedParser.java /nutchbase/src/plugin/feed/src/java/org/apache/nutch/parse/feed line 221 Java Problem<br>
ParseResult cannot be resolved to a type HTMLLanguageParser.java /nutchbase/src/plugin/languageidentifier/src/java/org/apache/nutch/analysis/lang line 90 Java Problem<br>ParseResult cannot be resolved to a type HTMLLanguageParser.java /nutchbase/src/plugin/languageidentifier/src/java/org/apache/nutch/analysis/lang line 90 Java Problem<br>
ParseResult cannot be resolved to a type MSBaseParser.java /nutchbase/src/plugin/lib-parsems/src/java/org/apache/nutch/parse/ms line 64 Java Problem<br>ParseResult cannot be resolved to a type MSExcelParser.java /nutchbase/src/plugin/parse-msexcel/src/java/org/apache/nutch/parse/msexcel line 40 Java Problem<br>
ParseResult cannot be resolved to a type MSPowerPointParser.java /nutchbase/src/plugin/parse-mspowerpoint/src/java/org/apache/nutch/parse/mspowerpoint line 44 Java Problem<br>ParseResult cannot be resolved to a type MSWordParser.java /nutchbase/src/plugin/parse-msword/src/java/org/apache/nutch/parse/msword line 43 Java Problem<br>
ParseResult cannot be resolved to a type OOParser.java /nutchbase/src/plugin/parse-oo/src/java/org/apache/nutch/parse/oo line 63 Java Problem<br>ParseResult cannot be resolved to a type PdfParser.java /nutchbase/src/plugin/parse-pdf/src/java/org/apache/nutch/parse/pdf line 69 Java Problem<br>
ParseResult cannot be resolved to a type RSSParser.java /nutchbase/src/plugin/parse-rss/src/java/org/apache/nutch/parse/rss line 80 Java Problem<br>ParseResult cannot be resolved to a type RelTagParser.java /nutchbase/src/plugin/microformats-reltag/src/java/org/apache/nutch/microformats/reltag line 68 Java Problem<br>
ParseResult cannot be resolved to a type RelTagParser.java /nutchbase/src/plugin/microformats-reltag/src/java/org/apache/nutch/microformats/reltag line 68 Java Problem<br>ParseResult cannot be resolved to a type SWFParser.java /nutchbase/src/plugin/parse-swf/src/java/org/apache/nutch/parse/swf line 64 Java Problem<br>
ParseResult cannot be resolved to a type SWFParser.java /nutchbase/src/plugin/parse-swf/src/java/org/apache/nutch/parse/swf line 125 Java Problem<br>ParseResult cannot be resolved to a type TestFeedParser.java /nutchbase/src/plugin/feed/src/test/org/apache/nutch/parse/feed line 94 Java Problem<br>
ParseResult cannot be resolved to a type TextParser.java /nutchbase/src/plugin/parse-text/src/java/org/apache/nutch/parse/text line 41 Java Problem<br>ParseResult cannot be resolved to a type ZipParser.java /nutchbase/src/plugin/parse-zip/src/java/org/apache/nutch/parse/zip line 55 Java Problem<br>
The constructor Fetcher(Configuration) is undefined TestFetcher.java /nutchbase/src/test/org/apache/nutch/fetcher line 100 Java Problem<br>The constructor Fetcher(Configuration) is undefined TestFetcher.java /nutchbase/src/test/org/apache/nutch/fetcher line 177 Java Problem<br>
The constructor Generator(Configuration) is undefined TestFetcher.java /nutchbase/src/test/org/apache/nutch/fetcher line 94 Java Problem<br>The constructor Generator(Configuration) is undefined TestGenerator.java /nutchbase/src/test/org/apache/nutch/crawl line 312 Java Problem<br>
The constructor Injector(Configuration) is undefined TestFetcher.java /nutchbase/src/test/org/apache/nutch/fetcher line 90 Java Problem<br>The constructor Injector(Configuration) is undefined TestInjector.java /nutchbase/src/test/org/apache/nutch/crawl line 70 Java Problem<br>
The constructor NutchWritable(ParseImpl) is undefined ArcSegmentCreator.java /nutchbase/src/java/org/apache/nutch/tools/arc line 229 Java Problem<br>The import org.apache.nutch.fetcher.FetcherOutputFormat cannot be resolved ArcSegmentCreator.java /nutchbase/src/java/org/apache/nutch/tools/arc line 44 Java Problem<br>
The import org.apache.nutch.parse.ParseImpl cannot be resolved ArcSegmentCreator.java /nutchbase/src/java/org/apache/nutch/tools/arc line 50 Java Problem<br>The import org.apache.nutch.parse.ParseImpl cannot be resolved BasicFields.java /nutchbase/src/java/org/apache/nutch/indexer/field line 61 Java Problem<br>
The import org.apache.nutch.parse.ParseImpl cannot be resolved ExtParser.java /nutchbase/src/plugin/parse-ext/src/java/org/apache/nutch/parse/ext line 26 Java Problem<br>The import org.apache.nutch.parse.ParseImpl cannot be resolved MSBaseParser.java /nutchbase/src/plugin/lib-parsems/src/java/org/apache/nutch/parse/ms line 39 Java Problem<br>
The import org.apache.nutch.parse.ParseImpl cannot be resolved PdfParser.java /nutchbase/src/plugin/parse-pdf/src/java/org/apache/nutch/parse/pdf line 41 Java Problem<br>The import org.apache.nutch.parse.ParseImpl cannot be resolved RSSParser.java /nutchbase/src/plugin/parse-rss/src/java/org/apache/nutch/parse/rss line 41 Java Problem<br>
The import org.apache.nutch.parse.ParseImpl cannot be resolved TestExtParser.java /nutchbase/src/plugin/parse-ext/src/test/org/apache/nutch/parse/ext line 26 Java Problem<br>The import org.apache.nutch.parse.ParseImpl cannot be resolved TestIndexingFilters.java /nutchbase/src/test/org/apache/nutch/indexer line 26 Java Problem<br>
The import org.apache.nutch.parse.ParseImpl cannot be resolved TestMSWordParser.java /nutchbase/src/plugin/parse-msword/src/test/org/apache/nutch/parse/msword line 26 Java Problem<br>The import org.apache.nutch.parse.ParseImpl cannot be resolved TestMoreIndexingFilter.java /nutchbase/src/plugin/index-more/src/test/org/apache/nutch/indexer/more line 29 Java Problem<br>
The import org.apache.nutch.parse.ParseImpl cannot be resolved TestZipParser.java /nutchbase/src/plugin/parse-zip/src/test/org/apache/nutch/parse/zip line 26 Java Problem<br>The import org.apache.nutch.parse.ParseImpl cannot be resolved ZipParser.java /nutchbase/src/plugin/parse-zip/src/java/org/apache/nutch/parse/zip line 33 Java Problem<br>
The import org.apache.nutch.parse.ParseImpl cannot be resolved ZipTextExtractor.java /nutchbase/src/plugin/parse-zip/src/java/org/apache/nutch/parse/zip line 41 Java Problem<br>The import org.apache.nutch.parse.ParseResult cannot be resolved ArcSegmentCreator.java /nutchbase/src/java/org/apache/nutch/tools/arc line 51 Java Problem<br>
The import org.apache.nutch.parse.ParseResult cannot be resolved ExtParser.java /nutchbase/src/plugin/parse-ext/src/java/org/apache/nutch/parse/ext line 21 Java Problem<br>The import org.apache.nutch.parse.ParseResult cannot be resolved FeedParser.java /nutchbase/src/plugin/feed/src/java/org/apache/nutch/parse/feed line 43 Java Problem<br>
The import org.apache.nutch.parse.ParseResult cannot be resolved HTMLLanguageParser.java /nutchbase/src/plugin/languageidentifier/src/java/org/apache/nutch/analysis/lang line 33 Java Problem<br>The import org.apache.nutch.parse.ParseResult cannot be resolved MSBaseParser.java /nutchbase/src/plugin/lib-parsems/src/java/org/apache/nutch/parse/ms line 40 Java Problem<br>
The import org.apache.nutch.parse.ParseResult cannot be resolved MSExcelParser.java /nutchbase/src/plugin/parse-msexcel/src/java/org/apache/nutch/parse/msexcel line 20 Java Problem<br>The import org.apache.nutch.parse.ParseResult cannot be resolved MSPowerPointParser.java /nutchbase/src/plugin/parse-mspowerpoint/src/java/org/apache/nutch/parse/mspowerpoint line 20 Java Problem<br>
The import org.apache.nutch.parse.ParseResult cannot be resolved MSWordParser.java /nutchbase/src/plugin/parse-msword/src/java/org/apache/nutch/parse/msword line 21 Java Problem<br>The import org.apache.nutch.parse.ParseResult cannot be resolved PdfParser.java /nutchbase/src/plugin/parse-pdf/src/java/org/apache/nutch/parse/pdf line 37 Java Problem<br>
The import org.apache.nutch.parse.ParseResult cannot be resolved RSSParser.java /nutchbase/src/plugin/parse-rss/src/java/org/apache/nutch/parse/rss line 36 Java Problem<br>The import org.apache.nutch.parse.ParseResult cannot be resolved RelTagParser.java /nutchbase/src/plugin/microformats-reltag/src/java/org/apache/nutch/microformats/reltag line 38 Java Problem<br>
The import org.apache.nutch.parse.ParseResult cannot be resolved TestFeedParser.java /nutchbase/src/plugin/feed/src/test/org/apache/nutch/parse/feed line 32 Java Problem<br>The import org.apache.nutch.parse.ParseResult cannot be resolved ZipParser.java /nutchbase/src/plugin/parse-zip/src/java/org/apache/nutch/parse/zip line 34 Java Problem<br>
The method calculate(WebTableRow, Parse) in the type Signature is not applicable for the arguments (Content, Parse) ArcSegmentCreator.java /nutchbase/src/java/org/apache/nutch/tools/arc line 187 Java Problem<br>
The method calculate(WebTableRow, Parse) in the type Signature is not applicable for the arguments (Content, Parse) ArcSegmentCreator.java /nutchbase/src/java/org/apache/nutch/tools/arc line 208 Java Problem<br>
The method fetch(String, int, boolean) from the type Fetcher is not visible TestFetcher.java /nutchbase/src/test/org/apache/nutch/fetcher line 178 Java Problem<br>The method fetch(String, int, boolean) in the type Fetcher is not applicable for the arguments (Path, int, boolean) TestFetcher.java /nutchbase/src/test/org/apache/nutch/fetcher line 101 Java Problem<br>
The method generate(String, long, long, boolean) in the type Generator is not applicable for the arguments (Path, Path, int, int, long, boolean, boolean) TestGenerator.java /nutchbase/src/test/org/apache/nutch/crawl line 313 Java Problem<br>
The method generate(String, long, long, boolean) in the type Generator is not applicable for the arguments (Path, Path, int, long, long, boolean, boolean) TestFetcher.java /nutchbase/src/test/org/apache/nutch/fetcher line 95 Java Problem<br>
The method getData() is undefined for the type Parse ArcSegmentCreator.java /nutchbase/src/java/org/apache/nutch/tools/arc line 200 Java Problem<br>The method getData() is undefined for the type Parse ArcSegmentCreator.java /nutchbase/src/java/org/apache/nutch/tools/arc line 211 Java Problem<br>
The method getData() is undefined for the type Parse ArcSegmentCreator.java /nutchbase/src/java/org/apache/nutch/tools/arc line 213 Java Problem<br>The method getData() is undefined for the type Parse ArcSegmentCreator.java /nutchbase/src/java/org/apache/nutch/tools/arc line 216 Java Problem<br>
The method getData() is undefined for the type Parse ArcSegmentCreator.java /nutchbase/src/java/org/apache/nutch/tools/arc line 230 Java Problem<br>The method getData() is undefined for the type Parse ArcSegmentCreator.java /nutchbase/src/java/org/apache/nutch/tools/arc line 244 Java Problem<br>
The method getData() is undefined for the type Parse BasicFields.java /nutchbase/src/java/org/apache/nutch/indexer/field line 386 Java Problem<br>The method getData() is undefined for the type Parse BasicFields.java /nutchbase/src/java/org/apache/nutch/indexer/field line 395 Java Problem<br>
The method getData() is undefined for the type Parse CCIndexingFilter.java /nutchbase/src/plugin/creativecommons/src/java/org/creativecommons/nutch line 55 Java Problem<br>The method getData() is undefined for the type Parse CCParseFilter.java /nutchbase/src/plugin/creativecommons/src/java/org/creativecommons/nutch line 280 Java Problem<br>
The method getData() is undefined for the type Parse CCParseFilter.java /nutchbase/src/plugin/creativecommons/src/java/org/creativecommons/nutch line 286 Java Problem<br>The method getData() is undefined for the type Parse CCParseFilter.java /nutchbase/src/plugin/creativecommons/src/java/org/creativecommons/nutch line 291 Java Problem<br>
The method getData() is undefined for the type Parse FeedIndexingFilter.java /nutchbase/src/plugin/feed/src/java/org/apache/nutch/indexer/feed line 76 Java Problem<br><br>
<p>From forum: <a href="http://old.nabble.com/Nutch---Dev-f373.html" embed="fixTarget[373]" target="_top" >Nutch - Dev</a></p>tag:old.nabble.com,2006:post-26656109HTTP Header problem2009-12-05T06:29:20Z2009-12-05T06:29:20ZKirk GillockHi fellow Nutch users.
<br><br>Long time crawler, first time poster. :-)
<br><br>We're 23m pages into a 100m page crawl and our preliminary tests have
<br>shown that a lot of pages contain our agent name, description, etc., in
<br>their page content. Meaning, sites that have a script which show http
<br>headers (typically to show browser information) causes the Nutch crawler
<br>to store its own header information within the content of that page. So
<br>when we search our index for "Isara" (our agent name) we get thousands
<br>of results and they all have "Isara/Isara-1.0 (A non-profit search
<br>engine benefiting charity.; <a href="http://www.isara.org;" target="_top" rel="nofollow">http://www.isara.org;</a> <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26656109&i=0" target="_top" rel="nofollow">e-mail@...</a>",
<br>which is the content of our nutch-default.xml file: http.agent.name,
<br>http.agent.description, http.agent.url, http.agent.email, and
<br>http.agent.version .
<br><br>I've searched around and haven't found any information on how to stop
<br>this from happening. Is there a solution and, if so, will it mean we
<br>need to recrawl all those pages again or can we filter the current
<br>database? Any suggestions would be greatly appreciated.
<br><br>Thank you for developing such an important open-source application,
<br>Kirk Gillock
<br>Isara Charity Foundation
<br>Nong Khai, Thailand
<br><a href="http://www.isara.org" target="_top" rel="nofollow">http://www.isara.org</a><br><p>From forum: <a href="http://old.nabble.com/Nutch---Agent-f372.html" embed="fixTarget[372]" target="_top" >Nutch - Agent</a></p>tag:old.nabble.com,2006:post-26656071[jira] Updated: (NUTCH-767) Update Tika to v0.5 for the MimeType detection2009-12-05T06:25:20Z2009-12-05T06:25:20ZJIRA jira@apache.org<br> [ <a href="https://issues.apache.org/jira/browse/NUTCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel" target="_top" rel="nofollow">https://issues.apache.org/jira/browse/NUTCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel</a> ]
<br><br>Julien Nioche updated NUTCH-767:
<br>--------------------------------
<br><br> Attachment: NUTCH-767-part3.patch
<br><br>the problems with the test comes from the fact that tika's detection of the mimetypes based on content returns "text/plain" when no mimetype can be identified, e.g. in our case because we have an empty byte array as content.
<br><br>Tika's MimeTypes used to have a default value which was used in MimeUtil to determine when to use the type guessed by Tika but it has been removed since. The best course of action is probably to take into account Tika's guess only if it is not "text/plain" or "application/octet-stream", which is what this patch implements.
<br><br>The expected mime types in the test class are set to their original values (pre patch v2) apart from the one which used Tika's default Mime Type.
<br><br>J.
<br><div class='shrinkable-quote'><br>> Update Tika to v0.5 for the MimeType detection
<br>> -----------------------------------------------
<br>>
<br>> Key: NUTCH-767
<br>> URL: <a href="https://issues.apache.org/jira/browse/NUTCH-767" target="_top" rel="nofollow">https://issues.apache.org/jira/browse/NUTCH-767</a><br>> Project: Nutch
<br>> Issue Type: Improvement
<br>> Reporter: Julien Nioche
<br>> Assignee: Andrzej Bialecki
<br>> Fix For: 1.1
<br>>
<br>> Attachments: NUTCH-767-part2.patch, NUTCH-767-part3.patch, NUTCH-767.patch
<br>>
<br>> Original Estimate: 0h
<br>> Remaining Estimate: 0h
<br>>
<br>> The version 0.5 of TIka requires a few changes to the MimeType implementation. Tika is now split in several jars, we need to place the tika-core.jar in the main nutch lib.
</div><br>--
<br>This message is automatically generated by JIRA.
<br>-
<br>You can reply to this email to add a comment to the issue online.
<br><br><p>From forum: <a href="http://old.nabble.com/Nutch---Dev-f373.html" embed="fixTarget[373]" target="_top" >Nutch - Dev</a></p>tag:old.nabble.com,2006:post-26656014[jira] Reopened: (NUTCH-767) Update Tika to v0.5 for the MimeType detection2009-12-05T06:19:20Z2009-12-05T06:19:20ZJIRA jira@apache.org<br> [ <a href="https://issues.apache.org/jira/browse/NUTCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel" target="_top" rel="nofollow">https://issues.apache.org/jira/browse/NUTCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel</a> ]
<br><br>Julien Nioche reopened NUTCH-767:
<br>---------------------------------
<br><br><br>the problem with the test class has been investigated. am reopening the issue so that we can mark it as definitely fixed
<br><div class='shrinkable-quote'><br>> Update Tika to v0.5 for the MimeType detection
<br>> -----------------------------------------------
<br>>
<br>> Key: NUTCH-767
<br>> URL: <a href="https://issues.apache.org/jira/browse/NUTCH-767" target="_top" rel="nofollow">https://issues.apache.org/jira/browse/NUTCH-767</a><br>> Project: Nutch
<br>> Issue Type: Improvement
<br>> Reporter: Julien Nioche
<br>> Assignee: Andrzej Bialecki
<br>> Fix For: 1.1
<br>>
<br>> Attachments: NUTCH-767-part2.patch, NUTCH-767.patch
<br>>
<br>> Original Estimate: 0h
<br>> Remaining Estimate: 0h
<br>>
<br>> The version 0.5 of TIka requires a few changes to the MimeType implementation. Tika is now split in several jars, we need to place the tika-core.jar in the main nutch lib.
</div><br>--
<br>This message is automatically generated by JIRA.
<br>-
<br>You can reply to this email to add a comment to the issue online.
<br><br><p>From forum: <a href="http://old.nabble.com/Nutch---Dev-f373.html" embed="fixTarget[373]" target="_top" >Nutch - Dev</a></p>tag:old.nabble.com,2006:post-26655130Re: Fetch failing ?2009-12-05T04:17:45Z2009-12-05T04:17:45ZMilleBiiThx again Julien,
<br><br>Yes I'm going to buy myself the Hadoop book, because I thought I could do
<br>without but I realize that I need to make good use of hadooop.
<br><br>Didn't know you could split fetching & parsing: so I suppose you just issue
<br>nutch fetch <segment> -noParsing, followed by nutch parse <segment>. I will
<br>try on my next run.
<br><br><br><br>2009/12/5 Julien Nioche <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26655130&i=0" target="_top" rel="nofollow">lists.digitalpebble@...</a>>
<br><div class='shrinkable-quote'><br>> HADOOP_HEAPSIZE specifies the memory to be used by the hadoop demons and
<br>> does NOT affect the memory used for the map/ reduce jobs. Maybe you should
<br>> invest a bit of time reading about Hadoop first?
<br>>
<br>> As for your memory problem it could be due to the parsing and not the
<br>> fetching. If you don't already do so I suggest that you separate the
<br>> fetching from the parsing. First that will tell you which part fails + if
<br>> it
<br>> does fail in the parsing then you would not need to refetch the content
<br>>
<br>> J.
<br>>
<br>> 2009/12/5 MilleBii <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26655130&i=1" target="_top" rel="nofollow">millebii@...</a>>
<br>>
<br>> > My fetch cycle failed on the following initial error :
<br>> >
<br>> > java.io.IOException: Task process exit with nonzero status of 65.
<br>> > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:425)
<br>> >
<br>> > Than it makes a second attempt and after 3 hours I bump on that error
<br>> > (altough I had double HADOOP_HEAPSIZE):
<br>> >
<br>> > java.lang.OutOfMemoryError: GC overhead limit exceeded
<br>> >
<br>> >
<br>> > Any idea what the initial error is or could be ?
<br>> > For the second one, I'm going to reduce number of threads... but I'm
<br>> > wondering if there could be a memory leak ? And I don't how to trace
<br>> that.
<br>> >
<br>> > --
<br>> > -MilleBii-
<br>> >
<br>>
<br>>
<br>>
<br>> --
<br>> DigitalPebble Ltd
<br>> <a href="http://www.digitalpebble.com" target="_top" rel="nofollow">http://www.digitalpebble.com</a><br>>
</div><br><br><br>--
<br>-MilleBii-
<br><p>From forum: <a href="http://old.nabble.com/Nutch---User-f375.html" embed="fixTarget[375]" target="_top" >Nutch - User</a></p>tag:old.nabble.com,2006:post-26654970Re: Fetch failing ?2009-12-05T03:56:50Z2009-12-05T03:56:50ZJulien Nioche-4HADOOP_HEAPSIZE specifies the memory to be used by the hadoop demons and
<br>does NOT affect the memory used for the map/ reduce jobs. Maybe you should
<br>invest a bit of time reading about Hadoop first?
<br><br>As for your memory problem it could be due to the parsing and not the
<br>fetching. If you don't already do so I suggest that you separate the
<br>fetching from the parsing. First that will tell you which part fails + if it
<br>does fail in the parsing then you would not need to refetch the content
<br><br>J.
<br><br>2009/12/5 MilleBii <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26654970&i=0" target="_top" rel="nofollow">millebii@...</a>>
<br><div class='shrinkable-quote'><br>> My fetch cycle failed on the following initial error :
<br>>
<br>> java.io.IOException: Task process exit with nonzero status of 65.
<br>> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:425)
<br>>
<br>> Than it makes a second attempt and after 3 hours I bump on that error
<br>> (altough I had double HADOOP_HEAPSIZE):
<br>>
<br>> java.lang.OutOfMemoryError: GC overhead limit exceeded
<br>>
<br>>
<br>> Any idea what the initial error is or could be ?
<br>> For the second one, I'm going to reduce number of threads... but I'm
<br>> wondering if there could be a memory leak ? And I don't how to trace that.
<br>>
<br>> --
<br>> -MilleBii-
<br>>
</div><br><br><br>--
<br>DigitalPebble Ltd
<br><a href="http://www.digitalpebble.com" target="_top" rel="nofollow">http://www.digitalpebble.com</a><br><p>From forum: <a href="http://old.nabble.com/Nutch---User-f375.html" embed="fixTarget[375]" target="_top" >Nutch - User</a></p>tag:old.nabble.com,2006:post-26653874Fetch failing ?2009-12-05T00:50:34Z2009-12-05T00:50:34ZMilleBiiMy fetch cycle failed on the following initial error :
<br><br>java.io.IOException: Task process exit with nonzero status of 65.
<br> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:425)
<br><br>Than it makes a second attempt and after 3 hours I bump on that error
<br>(altough I had double HADOOP_HEAPSIZE):
<br><br>java.lang.OutOfMemoryError: GC overhead limit exceeded
<br><br><br>Any idea what the initial error is or could be ?
<br>For the second one, I'm going to reduce number of threads... but I'm
<br>wondering if there could be a memory leak ? And I don't how to trace that.
<br><br>--
<br>-MilleBii-
<br><p>From forum: <a href="http://old.nabble.com/Nutch---User-f375.html" embed="fixTarget[375]" target="_top" >Nutch - User</a></p>tag:old.nabble.com,2006:post-26653828Re: How to drop page content at fetch stages ?2009-12-05T00:42:11Z2009-12-05T00:42:11ZMilleBiiThx, a bit too complex for me right now. I don't -yet- fully understand this
<br>map/reduce technique.
<br>But I'll keep the idea for a future development.
<br><br>2009/12/4 Dennis Kubes <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26653828&i=0" target="_top" rel="nofollow">kubes@...</a>>
<br><div class='shrinkable-quote'><br>> Sorry, segments, not indexes.
<br>>
<br>>
<br>> Dennis Kubes wrote:
<br>>
<br>>> You would need to write a custom MapReduce job to run through the indexes
<br>>> and only keeps the ones identified by your plugin. Be sure to update the
<br>>> CrawlDb with the extracted urls before you drop the content from the
<br>>> segments.
<br>>>
<br>>> Dennis
<br>>>
<br>>> MilleBii wrote:
<br>>>
<br>>>> Hi guys,
<br>>>>
<br>>>> I'm looking if I can optimize the size occupied on disk by my segments.
<br>>>> I have implemented a topical-scoring plugin... this means I know at that
<br>>>> steps if I should keep that page content or not.
<br>>>> Is there a way to drop some pages content after parsing it, but of course
<br>>>> keep the links because I want to follow the graph ?
<br>>>>
<br>>>> PS: Prune is no option to me because it only cleans up the indexes, not
<br>>>> the
<br>>>> segments and my indexer does that clean-up very well.
<br>>>>
<br>>>>
</div><br><br>--
<br>-MilleBii-
<br><p>From forum: <a href="http://old.nabble.com/Nutch---User-f375.html" embed="fixTarget[375]" target="_top" >Nutch - User</a></p>tag:old.nabble.com,2006:post-26653825Nutch - create my own repository2009-12-05T00:41:02Z2009-12-05T00:41:02ZzzeranHi,
<br><br>I'm developing my own set of tools, plugins and some minor code changes to
<br>Nutch.
<br><br>I still want to get updates from the main Nutch repository, but I would
<br>like to keep my own SVN for tracking my local code changes.
<br><br>I'm using normal shell SVN (I have no expirence with GIT) to track my
<br>changes.
<br><br>My question is - can I create a branch from the main repository to my own
<br>repository, which will only track my changes and keep getting updates from
<br>Nutch main repository with easy merge?
<br><br>Thanks,
<br>Eran
<br><p>From forum: <a href="http://old.nabble.com/Nutch---User-f375.html" embed="fixTarget[375]" target="_top" >Nutch - User</a></p>tag:old.nabble.com,2006:post-26653645Re: unsubscribe from nutch-user2009-12-04T23:59:55Z2009-12-04T23:59:55ZM S RamI did it many times. But I am still receiving these mails.
<br><br>prashant ullegaddi wrote:
<div class='shrinkable-quote'><br>> Take a look at it:
<br>>
<br>> <a href="http://lucene.apache.org/nutch/mailing_lists.html" target="_top" rel="nofollow">http://lucene.apache.org/nutch/mailing_lists.html</a><br>>
<br>> or
<br>> probably sending a blank mail to:
<br>> <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26653645&i=0" target="_top" rel="nofollow">nutch-user-unsubscribe@...</a> also
<br>> work.
<br>>
<br>> Thanks,
<br>> Prashant.
<br>>
<br>> On Fri, Dec 4, 2009 at 8:30 PM, M S Ram <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26653645&i=1" target="_top" rel="nofollow">msram@...</a>> wrote:
<br>>
<br>>
<br>>> Same here. Please remove my ID also from the mailing list.
<br>>>
<br>>> Thanks,
<br>>> MSR
<br>>>
<br>>> rengan xu wrote:
<br>>>
<br>>>
<br>>>> To whom it may concern,
<br>>>>
<br>>>> Hello! Because I will use this E-mail for special purpose. I will use
<br>>>> another E-mail to subscribe in nutch-user. So I want to unsubscribe from
<br>>>> nutch-user.
<br>>>>
<br>>>> Thank you!
<br>>>>
<br>>>>
<br>>>>
<br>>>>
<br>>>>
<br>>>
<br>>>
<br>>
<br>>
<br>>
</div><br><br><p>From forum: <a href="http://old.nabble.com/Nutch---User-f375.html" embed="fixTarget[375]" target="_top" >Nutch - User</a></p>tag:old.nabble.com,2006:post-26653542Nutch image extraction2009-12-04T23:36:18Z2009-12-04T23:36:18ZmanishkbawneHi,
<br>I am using nutch to crawl the data from the web. Now I want to extract the images using nutch. Can somebody please suggest me some way how to do that or sugeest me some url?
<br><br>Regards,
<br>Manish Bawne
<br>Software Engineer
<br>Biz Integra Systems Pvt Ltd
<br><a href="http://www.bizhandel.com" target="_top" rel="nofollow">http://www.bizhandel.com</a><br><p>From forum: <a href="http://old.nabble.com/Nutch---User-f375.html" embed="fixTarget[375]" target="_top" >Nutch - User</a></p>tag:old.nabble.com,2006:post-26653021Hudson build is back to normal: Nutch-trunk #10022009-12-04T20:54:23Z2009-12-04T20:54:23ZApache Hudson ServerSee <<a href="http://hudson.zones.apache.org/hudson/job/Nutch-trunk/1002/changes" target="_top" rel="nofollow">http://hudson.zones.apache.org/hudson/job/Nutch-trunk/1002/changes</a>>
<br><br><br><p>From forum: <a href="http://old.nabble.com/Nutch---Dev-f373.html" embed="fixTarget[373]" target="_top" >Nutch - Dev</a></p>tag:old.nabble.com,2006:post-26653020[jira] Commented: (NUTCH-767) Update Tika to v0.5 for the MimeType detection2009-12-04T20:54:20Z2009-12-04T20:54:20ZJIRA jira@apache.org<br> [ <a href="https://issues.apache.org/jira/browse/NUTCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12786339#action_12786339" target="_top" rel="nofollow">https://issues.apache.org/jira/browse/NUTCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12786339#action_12786339</a> ]
<br><br>Hudson commented on NUTCH-767:
<br>------------------------------
<br><br>Integrated in Nutch-trunk #1002 (See [<a href="http://hudson.zones.apache.org/hudson/job/Nutch-trunk/1002/" target="_top" rel="nofollow">http://hudson.zones.apache.org/hudson/job/Nutch-trunk/1002/</a>])
<br> Fix a failing test - still needs more work.
<br><br><div class='shrinkable-quote'><br>> Update Tika to v0.5 for the MimeType detection
<br>> -----------------------------------------------
<br>>
<br>> Key: NUTCH-767
<br>> URL: <a href="https://issues.apache.org/jira/browse/NUTCH-767" target="_top" rel="nofollow">https://issues.apache.org/jira/browse/NUTCH-767</a><br>> Project: Nutch
<br>> Issue Type: Improvement
<br>> Reporter: Julien Nioche
<br>> Assignee: Andrzej Bialecki
<br>> Fix For: 1.1
<br>>
<br>> Attachments: NUTCH-767-part2.patch, NUTCH-767.patch
<br>>
<br>> Original Estimate: 0h
<br>> Remaining Estimate: 0h
<br>>
<br>> The version 0.5 of TIka requires a few changes to the MimeType implementation. Tika is now split in several jars, we need to place the tika-core.jar in the main nutch lib.
</div><br>--
<br>This message is automatically generated by JIRA.
<br>-
<br>You can reply to this email to add a comment to the issue online.
<br><br><p>From forum: <a href="http://old.nabble.com/Nutch---Dev-f373.html" embed="fixTarget[373]" target="_top" >Nutch - Dev</a></p>tag:old.nabble.com,2006:post-26650632Re: How to drop page content at fetch stages ?2009-12-04T14:55:50Z2009-12-04T14:55:50ZDennis Kubes-2Sorry, segments, not indexes.
<br><br>Dennis Kubes wrote:
<div class='shrinkable-quote'><br>> You would need to write a custom MapReduce job to run through the
<br>> indexes and only keeps the ones identified by your plugin. Be sure to
<br>> update the CrawlDb with the extracted urls before you drop the content
<br>> from the segments.
<br>>
<br>> Dennis
<br>>
<br>> MilleBii wrote:
<br>>> Hi guys,
<br>>>
<br>>> I'm looking if I can optimize the size occupied on disk by my segments.
<br>>> I have implemented a topical-scoring plugin... this means I know at that
<br>>> steps if I should keep that page content or not.
<br>>> Is there a way to drop some pages content after parsing it, but of course
<br>>> keep the links because I want to follow the graph ?
<br>>>
<br>>> PS: Prune is no option to me because it only cleans up the indexes,
<br>>> not the
<br>>> segments and my indexer does that clean-up very well.
<br>>>
<br></div><p>From forum: <a href="http://old.nabble.com/Nutch---User-f375.html" embed="fixTarget[375]" target="_top" >Nutch - User</a></p>tag:old.nabble.com,2006:post-26650506Re: How to drop page content at fetch stages ?2009-12-04T14:47:31Z2009-12-04T14:47:31ZDennis Kubes-2You would need to write a custom MapReduce job to run through the
<br>indexes and only keeps the ones identified by your plugin. Be sure to
<br>update the CrawlDb with the extracted urls before you drop the content
<br>from the segments.
<br><br>Dennis
<br><br>MilleBii wrote:
<div class='shrinkable-quote'><br>> Hi guys,
<br>>
<br>> I'm looking if I can optimize the size occupied on disk by my segments.
<br>> I have implemented a topical-scoring plugin... this means I know at that
<br>> steps if I should keep that page content or not.
<br>> Is there a way to drop some pages content after parsing it, but of course
<br>> keep the links because I want to follow the graph ?
<br>>
<br>> PS: Prune is no option to me because it only cleans up the indexes, not the
<br>> segments and my indexer does that clean-up very well.
<br>>
<br></div><p>From forum: <a href="http://old.nabble.com/Nutch---User-f375.html" embed="fixTarget[375]" target="_top" >Nutch - User</a></p>tag:old.nabble.com,2006:post-26650126How to drop page content at fetch stages ?2009-12-04T14:18:23Z2009-12-04T14:18:23ZMilleBiiHi guys,
<br><br>I'm looking if I can optimize the size occupied on disk by my segments.
<br>I have implemented a topical-scoring plugin... this means I know at that
<br>steps if I should keep that page content or not.
<br>Is there a way to drop some pages content after parsing it, but of course
<br>keep the links because I want to follow the graph ?
<br><br>PS: Prune is no option to me because it only cleans up the indexes, not the
<br>segments and my indexer does that clean-up very well.
<br><br>--
<br>-MilleBii-
<br><p>From forum: <a href="http://old.nabble.com/Nutch---User-f375.html" embed="fixTarget[375]" target="_top" >Nutch - User</a></p>tag:old.nabble.com,2006:post-26648656Re: What is the best choice: nutch/lucene or nutch/solr?2009-12-04T12:20:26Z2009-12-04T12:20:26ZOtis Gospodnetic-2Sounds like Nutch for crawling to gather the data, custom tools to read the gathered data, call the KV store, construct SolrInputDocuments, and index those to Solr. If you want Solr and not Lucene, which is a bigger question that I can't answer without knowing the details.
<br><br> Otis
<br>--
<br>Sematext -- <a href="http://sematext.com/" target="_top" rel="nofollow">http://sematext.com/</a> -- Solr - Lucene - Nutch
<br><br><br><br>----- Original Message ----
<div class='shrinkable-quote'><br>> From: Mr Hadoop <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26648656&i=0" target="_top" rel="nofollow">mrhadoop@...</a>>
<br>> To: <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26648656&i=1" target="_top" rel="nofollow">nutch-user@...</a>
<br>> Sent: Fri, December 4, 2009 2:51:47 PM
<br>> Subject: What is the best choice: nutch/lucene or nutch/solr?
<br>>
<br>> I am going over mailing list and still didn't find an answer.
<br>>
<br>> For a project, I need to crawl the web, index it and merge that content with
<br>> another site's content which is stored inside the key-value storage system.
<br>>
<br>> What is the best approach to merge these two contents in to a lucene index,
<br>> solr index or keep the index separate but merge during the search query
<br>> results?
</div><br><p>From forum: <a href="http://old.nabble.com/Nutch---User-f375.html" embed="fixTarget[375]" target="_top" >Nutch - User</a></p>tag:old.nabble.com,2006:post-26648251What is the best choice: nutch/lucene or nutch/solr?2009-12-04T11:51:47Z2009-12-04T11:51:47ZclusterboyI am going over mailing list and still didn't find an answer.
<br><br>For a project, I need to crawl the web, index it and merge that content with
<br>another site's content which is stored inside the key-value storage system.
<br><br>What is the best approach to merge these two contents in to a lucene index,
<br>solr index or keep the index separate but merge during the search query
<br>results?
<br><p>From forum: <a href="http://old.nabble.com/Nutch---User-f375.html" embed="fixTarget[375]" target="_top" >Nutch - User</a></p>tag:old.nabble.com,2006:post-26647740RE: Problems with a new Installation of Nutch2009-12-04T11:15:57Z2009-12-04T11:15:57ZTom LandvoigtHi,
<br><br>Does anyone know what packages I have to install in Suse to get Nutch running?
<br><br>I have another installation with nutch where everything is fine. So I copied the hole installation. It's also an Suse linux but it is in 64 bit and I don’t installed it.
<br><br>But the same problem.
<br>At the moment I installed the following packages:
<br>Tomcat 6
<br>Openjdk devel 1.6
<br>Sun java devel 1.6
<br>Ant 1.7
<br><br>Now it is enough for today.
<br><br>Hope someone can help.
<br><br>Tom
<br><br>-----Original Message-----
<br>From: MilleBii [mailto:<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26647740&i=0" target="_top" rel="nofollow">millebii@...</a>]
<br>Sent: Freitag, 4. Dezember 2009 17:31
<br>To: <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26647740&i=1" target="_top" rel="nofollow">nutch-user@...</a>
<br>Subject: Re: Problems with a new Installation of Nutch
<br><br>I don't know that hadoop uses tomcat... But I think it uses Jetty
<br>instead. The nodes communicate via http: so you need some kind of web
<br>server... And for monitorin its the best way
<br><br>2009/12/4, Tom Landvoigt <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26647740&i=2" target="_top" rel="nofollow">tom.landvoigt@...</a>>:
<div class='shrinkable-quote'><br>> Hi,
<br>>
<br>> I don't have tomcat on this system because I don't want to use the
<br>> websearch. But if it is necessary for hadoop what I don’t think I will
<br>> install it.
<br>>
<br>> nutch@ip-10-224-113-210:/nutch/search> ./bin/hadoop fs -ls /
<br>> Found 1 items
<br>> -rw-r--r-- 2 nutch supergroup 0 2009-12-04 14:04 /url.txt
<br>> nutch@ip-10-224-113-210:/nutch/search>
<br>>
<br>> I get the normal answer but the file is empty.
<br>>
<br>> -----Original Message-----
<br>> From: MilleBii [mailto:<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26647740&i=3" target="_top" rel="nofollow">millebii@...</a>]
<br>> Sent: Freitag, 4. Dezember 2009 15:06
<br>> To: <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26647740&i=4" target="_top" rel="nofollow">nutch-user@...</a>
<br>> Subject: Re: Problems with a new Installation of Nutch
<br>>
<br>> Did you check with the web interface ? It gives a lot of info you can
<br>> even browse the file system.
<br>>
<br>> Try hadoop fs -ls to see what it gives you ?
<br>>
<br>> 2009/12/4, Tom Landvoigt <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26647740&i=5" target="_top" rel="nofollow">tom.landvoigt@...</a>>:
<br>>> Hallo,
<br>>>
<br>>>
<br>>>
<br>>> I hope someone can help me.
<br>>>
<br>>>
<br>>>
<br>>> I installed nutch on 2 Amazon EC2 computers. Everything is fine but I
<br>>> can't put data in the hdfs.
<br>>>
<br>>>
<br>>>
<br>>> I formatted the namenode and start the hdfs with start all.
<br>>>
<br>>>
<br>>>
<br>>> All java processes start properly, but when I want to make hadoop fs
<br>>> -put something / I get these logs:
<br>>>
<br>>>
<br>>>
<br>>>
<br>>>
<br>>>
<br>>>
<br>>> nutch@bla:/nutch/search> ./bin/hadoop fs -put
<br>>> /tmp/hadoop-nutch-tasktracker.pid blub
<br>>>
<br>>> put: Protocol not available
<br>>>
<br>>>
<br>>>
<br>>> DATA NODE LOG on the master:
<br>>>
<br>>> 2009-12-04 12:50:15,566 INFO http.HttpServer - Version Jetty/5.1.4
<br>>>
<br>>> 2009-12-04 12:50:15,582 INFO util.Credential - Checking Resource
<br>>> aliases
<br>>>
<br>>> 2009-12-04 12:50:16,483 INFO util.Container - Started
<br>>> org.mortbay.jetty.servlet.WebApplicationHandler@e45b5e
<br>>>
<br>>> 2009-12-04 12:50:16,614 INFO util.Container - Started
<br>>> WebApplicationContext[/static,/static]
<br>>>
<br>>> 2009-12-04 12:50:16,882 INFO util.Container - Started
<br>>> org.mortbay.jetty.servlet.WebApplicationHandler@1284fd4
<br>>>
<br>>> 2009-12-04 12:50:16,883 INFO util.Container - Started
<br>>> WebApplicationContext[/logs,/logs]
<br>>>
<br>>> 2009-12-04 12:50:17,827 INFO util.Container - Started
<br>>> org.mortbay.jetty.servlet.WebApplicationHandler@39c8c1
<br>>>
<br>>> 2009-12-04 12:50:17,849 INFO util.Container - Started
<br>>> WebApplicationContext[/,/]
<br>>>
<br>>> 2009-12-04 12:50:18,485 INFO http.SocketListener - Started
<br>>> SocketListener on 0.0.0.0:50075
<br>>>
<br>>> 2009-12-04 12:50:18,485 INFO util.Container - Started
<br>>> org.mortbay.jetty.Server@36527f
<br>>>
<br>>> 2009-12-04 12:54:20,745 ERROR datanode.DataNode -
<br>>> DatanodeRegistration(10.224.113.210:50010,
<br>>> storageID=DS-1135263253-10.224.113.210-50010-1259926637370,
<br>>> infoPort=50075, ipcPort=50020):DataXceiver
<br>>>
<br>>> java.io.EOFException
<br>>>
<br>>> at java.io.DataInputStream.readShort(DataInputStream.java:315)
<br>>>
<br>>> at
<br>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:
<br>>> 79)
<br>>>
<br>>> at java.lang.Thread.run(Thread.java:636)
<br>>>
<br>>> 2009-12-04 12:54:20,746 ERROR datanode.DataNode -
<br>>> DatanodeRegistration(10.224.113.210:50010,
<br>>> storageID=DS-1135263253-10.224.113.210-50010-1259926637370,
<br>>> infoPort=50075, ipcPort=50020):DataXceiver
<br>>>
<br>>> java.io.EOFException
<br>>>
<br>>> at java.io.DataInputStream.readShort(DataInputStream.java:315)
<br>>>
<br>>> at
<br>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:
<br>>> 79)
<br>>>
<br>>> at java.lang.Thread.run(Thread.java:636)
<br>>>
<br>>> 2009-12-04 12:54:20,747 ERROR datanode.DataNode -
<br>>> DatanodeRegistration(10.224.113.210:50010,
<br>>> storageID=DS-1135263253-10.224.113.210-50010-1259926637370,
<br>>> infoPort=50075, ipcPort=50020):DataXceiver
<br>>>
<br>>> java.io.EOFException
<br>>>
<br>>> at java.io.DataInputStream.readShort(DataInputStream.java:315)
<br>>>
<br>>> at
<br>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:
<br>>> 79)
<br>>>
<br>>> at java.lang.Thread.run(Thread.java:636)
<br>>>
<br>>> 2009-12-04 12:54:20,747 ERROR datanode.DataNode -
<br>>> DatanodeRegistration(10.224.113.210:50010,
<br>>> storageID=DS-1135263253-10.224.113.210-50010-1259926637370,
<br>>> infoPort=50075, ipcPort=50020):DataXceiver
<br>>>
<br>>> java.io.EOFException
<br>>>
<br>>> at java.io.DataInputStream.readShort(DataInputStream.java:315)
<br>>>
<br>>> at
<br>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:
<br>>> 79)
<br>>>
<br>>> at java.lang.Thread.run(Thread.java:636)
<br>>>
<br>>>
<br>>>
<br>>> NAME NODE LOG
<br>>>
<br>>> 2009-12-04 12:50:11,539 INFO http.HttpServer - Version Jetty/5.1.4
<br>>>
<br>>> 2009-12-04 12:50:11,573 INFO util.Credential - Checking Resource
<br>>> aliases
<br>>>
<br>>> 2009-12-04 12:50:12,488 INFO util.Container - Started
<br>>> org.mortbay.jetty.servlet.WebApplicationHandler@19fe451
<br>>>
<br>>> 2009-12-04 12:50:12,565 INFO util.Container - Started
<br>>> WebApplicationContext[/static,/static]
<br>>>
<br>>> 2009-12-04 12:50:12,891 INFO util.Container - Started
<br>>> org.mortbay.jetty.servlet.WebApplicationHandler@1570945
<br>>>
<br>>> 2009-12-04 12:50:12,891 INFO util.Container - Started
<br>>> WebApplicationContext[/logs,/logs]
<br>>>
<br>>> 2009-12-04 12:50:13,569 INFO util.Container - Started
<br>>> org.mortbay.jetty.servlet.WebApplicationHandler@11410e5
<br>>>
<br>>> 2009-12-04 12:50:13,582 INFO util.Container - Started
<br>>> WebApplicationContext[/,/]
<br>>>
<br>>> 2009-12-04 12:50:13,613 INFO http.SocketListener - Started
<br>>> SocketListener on 0.0.0.0:50070
<br>>>
<br>>> 2009-12-04 12:50:13,613 INFO util.Container - Started
<br>>> org.mortbay.jetty.Server@173ec72
<br>>>
<br>>>
<br>>>
<br>>> SECONDARY NAMENODE LOG
<br>>>
<br>>> 2009-12-04 12:50:19,163 INFO http.HttpServer - Version Jetty/5.1.4
<br>>>
<br>>> 2009-12-04 12:50:19,207 INFO util.Credential - Checking Resource
<br>>> aliases
<br>>>
<br>>> 2009-12-04 12:50:20,365 INFO util.Container - Started
<br>>> org.mortbay.jetty.servlet.WebApplicationHandler@174d93a
<br>>>
<br>>> 2009-12-04 12:50:20,454 INFO util.Container - Started
<br>>> WebApplicationContext[/static,/static]
<br>>>
<br>>> 2009-12-04 12:50:21,396 INFO util.Container - Started
<br>>> org.mortbay.jetty.servlet.WebApplicationHandler@31f2a7
<br>>>
<br>>> 2009-12-04 12:50:21,396 INFO util.Container - Started
<br>>> WebApplicationContext[/logs,/logs]
<br>>>
<br>>> 2009-12-04 12:50:21,533 INFO servlet.XMLConfiguration - No
<br>>> WEB-INF/web.xml in file:/mnt/nutch/nutch-1.0/webapps/secondary. Serving
<br>>> files and default/dynamic servlets only
<br>>>
<br>>> 2009-12-04 12:50:22,206 INFO util.Container - Started
<br>>> org.mortbay.jetty.servlet.WebApplicationHandler@383118
<br>>>
<br>>> 2009-12-04 12:50:22,785 INFO util.Container - Started
<br>>> WebApplicationContext[/,/]
<br>>>
<br>>> 2009-12-04 12:50:22,787 INFO http.SocketListener - Started
<br>>> SocketListener on 0.0.0.0:50090
<br>>>
<br>>> 2009-12-04 12:50:22,787 INFO util.Container - Started
<br>>> org.mortbay.jetty.Server@297ffb
<br>>>
<br>>> 2009-12-04 12:50:22,787 WARN namenode.SecondaryNameNode - Checkpoint
<br>>> Period :3600 secs (60 min)
<br>>>
<br>>> 2009-12-04 12:50:22,787 WARN namenode.SecondaryNameNode - Log Size
<br>>> Trigger :67108864 bytes (65536 KB)
<br>>>
<br>>> 2009-12-04 12:55:23,908 WARN namenode.SecondaryNameNode - Checkpoint
<br>>> done. New Image Size: 1056
<br>>>
<br>>>
<br>>>
<br>>> HADOOP LOG
<br>>>
<br>>> 2009-12-04 12:54:20,708 WARN hdfs.DFSClient - DataStreamer Exception:
<br>>> java.io.IOException: Unable to create new block.
<br>>>
<br>>> at
<br>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(D
<br>>> FSClient.java:2722)
<br>>>
<br>>> at
<br>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.j
<br>>> ava:1996)
<br>>>
<br>>> at
<br>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCli
<br>>> ent.java:2183)
<br>>>
<br>>>
<br>>>
<br>>> 2009-12-04 12:54:20,709 WARN hdfs.DFSClient - Error Recovery for block
<br>>> blk_5506837520665828594_1002 bad datanode[0] nodes == null
<br>>>
<br>>> 2009-12-04 12:54:20,709 WARN hdfs.DFSClient - Could not get block
<br>>> locations. Source file "/user/nutch/blub/hadoop-nutch-tasktracker.pid" -
<br>>> Aborting...
<br>>>
<br>>>
<br>>>
<br>>> DATA NODE LOG on the slave
<br>>>
<br>>> 2009-12-04 12:49:49,433 INFO http.HttpServer - Version Jetty/5.1.4
<br>>>
<br>>> 2009-12-04 12:49:49,438 INFO util.Credential - Checking Resource
<br>>> aliases
<br>>>
<br>>> 2009-12-04 12:49:50,288 INFO util.Container - Started
<br>>> org.mortbay.jetty.servlet.WebApplicationHandler@e45b5e
<br>>>
<br>>> 2009-12-04 12:49:50,357 INFO util.Container - Started
<br>>> WebApplicationContext[/static,/static]
<br>>>
<br>>> 2009-12-04 12:49:50,555 INFO util.Container - Started
<br>>> org.mortbay.jetty.servlet.WebApplicationHandler@2016b0
<br>>>
<br>>> 2009-12-04 12:49:50,555 INFO util.Container - Started
<br>>> WebApplicationContext[/logs,/logs]
<br>>>
<br>>> 2009-12-04 12:49:50,816 INFO util.Container - Started
<br>>> org.mortbay.jetty.servlet.WebApplicationHandler@118278a
<br>>>
<br>>> 2009-12-04 12:49:50,820 INFO util.Container - Started
<br>>> WebApplicationContext[/,/]
<br>>>
<br>>> 2009-12-04 12:49:50,849 INFO http.SocketListener - Started
<br>>> SocketListener on 0.0.0.0:50075
<br>>>
<br>>> 2009-12-04 12:49:50,849 INFO util.Container - Started
<br>>> org.mortbay.jetty.Server@b02928
<br>>>
<br>>>
<br>>>
<br>>> HADOOP SITE XML
<br>>>
<br>>> <property>
<br>>>
<br>>> <name>fs.default.name</name>
<br>>>
<br>>> <value>hdfs://(yes here is the right ip):9000</value>
<br>>>
<br>>> <description>
<br>>>
<br>>> The name of the default file system. Either the literal string
<br>>>
<br>>> "local" or a host:port for NDFS.
<br>>>
<br>>> </description>
<br>>>
<br>>> </property>
<br>>>
<br>>>
<br>>>
<br>>> <!-- Gibt an wo der JobTracker (koordiniert die (MapReduce-)Auftraege)
<br>>> zu finden ist. -->
<br>>>
<br>>> <property>
<br>>>
<br>>> <name>mapred.job.tracker</name>
<br>>>
<br>>> <value>hdfs://(here to):9001</value>
<br>>>
<br>>> <description>
<br>>>
<br>>> The host and port that the MapReduce job tracker runs at. If
<br>>>
<br>>> "local", then jobs are run in-process as a single map and
<br>>>
<br>>> reduce task.
<br>>>
<br>>> </description>
<br>>>
<br>>> </property>
<br>>>
<br>>>
<br>>>
<br>>> <!-- Gibt an wieviele MapJobs gleichzeitig laufen duerfen-->
<br>>>
<br>>> <property>
<br>>>
<br>>> <name>mapred.tasktracker.map.tasks.maximum</name>
<br>>>
<br>>> <value>2</value>
<br>>>
<br>>> <description>
<br>>>
<br>>> define mapred.map tasks to be number of slave hosts
<br>>>
<br>>> </description>
<br>>>
<br>>> </property>
<br>>>
<br>>>
<br>>>
<br>>> <!-- Gibt an wieviele ReduceJobs gleichzeitig laufen duerfen-->
<br>>>
<br>>> <property>
<br>>>
<br>>> <name>mapred.tasktracker.reduce.tasks.maximum</name>
<br>>>
<br>>> <value>2</value>
<br>>>
<br>>> <description>
<br>>>
<br>>> define mapred.reduce tasks to be number of slave hosts
<br>>>
<br>>> </description>
<br>>>
<br>>> </property>
<br>>>
<br>>>
<br>>>
<br>>> <property>
<br>>>
<br>>> <name>mapred.child.java.opts</name>
<br>>>
<br>>> <value>-Xmx1500m</value>
<br>>>
<br>>> </property>
<br>>>
<br>>>
<br>>>
<br>>> <property>
<br>>>
<br>>> <name>mapred.jobtracker.restart.recover</name>
<br>>>
<br>>> <value>true</value>
<br>>>
<br>>> </property>
<br>>>
<br>>>
<br>>>
<br>>> <!-- Die naechsten Einstellungen geben an wo das HadoopFS seine Datein
<br>>> auf der Festplatte jeder Instanz speichert. -->
<br>>>
<br>>> <property>
<br>>>
<br>>> <name>dfs.name.dir</name>
<br>>>
<br>>> <value>/nutch/filesystem/name</value>
<br>>>
<br>>> </property>
<br>>>
<br>>>
<br>>>
<br>>> <property>
<br>>>
<br>>> <name>dfs.data.dir</name>
<br>>>
<br>>> <value>/nutch/filesystem/data</value>
<br>>>
<br>>> </property>
<br>>>
<br>>>
<br>>>
<br>>> <property>
<br>>>
<br>>> <name>mapred.system.dir</name>
<br>>>
<br>>> <value>/nutch/filesystem/mapreduce/system</value>
<br>>>
<br>>> </property>
<br>>>
<br>>>
<br>>>
<br>>> <property>
<br>>>
<br>>> <name>mapred.local.dir</name>
<br>>>
<br>>> <value>/nutch/filesystem/mapreduce/local</value>
<br>>>
<br>>> </property>
<br>>>
<br>>>
<br>>>
<br>>> <!-- Gibt an wieviele Replikate einer Datei im Dateisystem vorhanden
<br>>> sein muessen damit sie erreichbar ist. Am Anfang 1 -->
<br>>>
<br>>> <property>
<br>>>
<br>>> <name>dfs.replication</name>
<br>>>
<br>>> <value>2</value>
<br>>>
<br>>> </property>
<br>>>
<br>>>
<br>>>
<br>>> I hope someone can help me.
<br>>>
<br>>>
<br>>>
<br>>> Thanks
<br>>>
<br>>>
<br>>>
<br>>> Tom
<br>>>
<br>>>
<br>>
<br>>
<br>> --
<br>> -MilleBii-
<br>>
</div><br><br>--
<br>-MilleBii-
<br><p>From forum: <a href="http://old.nabble.com/Nutch---User-f375.html" embed="fixTarget[375]" target="_top" >Nutch - User</a></p>tag:old.nabble.com,2006:post-26645189Re: Problems with a new Installation of Nutch2009-12-04T08:30:58Z2009-12-04T08:30:58ZMilleBiiI don't know that hadoop uses tomcat... But I think it uses Jetty
<br>instead. The nodes communicate via http: so you need some kind of web
<br>server... And for monitorin its the best way
<br><br>2009/12/4, Tom Landvoigt <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26645189&i=0" target="_top" rel="nofollow">tom.landvoigt@...</a>>:
<div class='shrinkable-quote'><br>> Hi,
<br>>
<br>> I don't have tomcat on this system because I don't want to use the
<br>> websearch. But if it is necessary for hadoop what I don’t think I will
<br>> install it.
<br>>
<br>> nutch@ip-10-224-113-210:/nutch/search> ./bin/hadoop fs -ls /
<br>> Found 1 items
<br>> -rw-r--r-- 2 nutch supergroup 0 2009-12-04 14:04 /url.txt
<br>> nutch@ip-10-224-113-210:/nutch/search>
<br>>
<br>> I get the normal answer but the file is empty.
<br>>
<br>> -----Original Message-----
<br>> From: MilleBii [mailto:<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26645189&i=1" target="_top" rel="nofollow">millebii@...</a>]
<br>> Sent: Freitag, 4. Dezember 2009 15:06
<br>> To: <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26645189&i=2" target="_top" rel="nofollow">nutch-user@...</a>
<br>> Subject: Re: Problems with a new Installation of Nutch
<br>>
<br>> Did you check with the web interface ? It gives a lot of info you can
<br>> even browse the file system.
<br>>
<br>> Try hadoop fs -ls to see what it gives you ?
<br>>
<br>> 2009/12/4, Tom Landvoigt <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26645189&i=3" target="_top" rel="nofollow">tom.landvoigt@...</a>>:
<br>>> Hallo,
<br>>>
<br>>>
<br>>>
<br>>> I hope someone can help me.
<br>>>
<br>>>
<br>>>
<br>>> I installed nutch on 2 Amazon EC2 computers. Everything is fine but I
<br>>> can't put data in the hdfs.
<br>>>
<br>>>
<br>>>
<br>>> I formatted the namenode and start the hdfs with start all.
<br>>>
<br>>>
<br>>>
<br>>> All java processes start properly, but when I want to make hadoop fs
<br>>> -put something / I get these logs:
<br>>>
<br>>>
<br>>>
<br>>>
<br>>>
<br>>>
<br>>>
<br>>> nutch@bla:/nutch/search> ./bin/hadoop fs -put
<br>>> /tmp/hadoop-nutch-tasktracker.pid blub
<br>>>
<br>>> put: Protocol not available
<br>>>
<br>>>
<br>>>
<br>>> DATA NODE LOG on the master:
<br>>>
<br>>> 2009-12-04 12:50:15,566 INFO http.HttpServer - Version Jetty/5.1.4
<br>>>
<br>>> 2009-12-04 12:50:15,582 INFO util.Credential - Checking Resource
<br>>> aliases
<br>>>
<br>>> 2009-12-04 12:50:16,483 INFO util.Container - Started
<br>>> org.mortbay.jetty.servlet.WebApplicationHandler@e45b5e
<br>>>
<br>>> 2009-12-04 12:50:16,614 INFO util.Container - Started
<br>>> WebApplicationContext[/static,/static]
<br>>>
<br>>> 2009-12-04 12:50:16,882 INFO util.Container - Started
<br>>> org.mortbay.jetty.servlet.WebApplicationHandler@1284fd4
<br>>>
<br>>> 2009-12-04 12:50:16,883 INFO util.Container - Started
<br>>> WebApplicationContext[/logs,/logs]
<br>>>
<br>>> 2009-12-04 12:50:17,827 INFO util.Container - Started
<br>>> org.mortbay.jetty.servlet.WebApplicationHandler@39c8c1
<br>>>
<br>>> 2009-12-04 12:50:17,849 INFO util.Container - Started
<br>>> WebApplicationContext[/,/]
<br>>>
<br>>> 2009-12-04 12:50:18,485 INFO http.SocketListener - Started
<br>>> SocketListener on 0.0.0.0:50075
<br>>>
<br>>> 2009-12-04 12:50:18,485 INFO util.Container - Started
<br>>> org.mortbay.jetty.Server@36527f
<br>>>
<br>>> 2009-12-04 12:54:20,745 ERROR datanode.DataNode -
<br>>> DatanodeRegistration(10.224.113.210:50010,
<br>>> storageID=DS-1135263253-10.224.113.210-50010-1259926637370,
<br>>> infoPort=50075, ipcPort=50020):DataXceiver
<br>>>
<br>>> java.io.EOFException
<br>>>
<br>>> at java.io.DataInputStream.readShort(DataInputStream.java:315)
<br>>>
<br>>> at
<br>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:
<br>>> 79)
<br>>>
<br>>> at java.lang.Thread.run(Thread.java:636)
<br>>>
<br>>> 2009-12-04 12:54:20,746 ERROR datanode.DataNode -
<br>>> DatanodeRegistration(10.224.113.210:50010,
<br>>> storageID=DS-1135263253-10.224.113.210-50010-1259926637370,
<br>>> infoPort=50075, ipcPort=50020):DataXceiver
<br>>>
<br>>> java.io.EOFException
<br>>>
<br>>> at java.io.DataInputStream.readShort(DataInputStream.java:315)
<br>>>
<br>>> at
<br>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:
<br>>> 79)
<br>>>
<br>>> at java.lang.Thread.run(Thread.java:636)
<br>>>
<br>>> 2009-12-04 12:54:20,747 ERROR datanode.DataNode -
<br>>> DatanodeRegistration(10.224.113.210:50010,
<br>>> storageID=DS-1135263253-10.224.113.210-50010-1259926637370,
<br>>> infoPort=50075, ipcPort=50020):DataXceiver
<br>>>
<br>>> java.io.EOFException
<br>>>
<br>>> at java.io.DataInputStream.readShort(DataInputStream.java:315)
<br>>>
<br>>> at
<br>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:
<br>>> 79)
<br>>>
<br>>> at java.lang.Thread.run(Thread.java:636)
<br>>>
<br>>> 2009-12-04 12:54:20,747 ERROR datanode.DataNode -
<br>>> DatanodeRegistration(10.224.113.210:50010,
<br>>> storageID=DS-1135263253-10.224.113.210-50010-1259926637370,
<br>>> infoPort=50075, ipcPort=50020):DataXceiver
<br>>>
<br>>> java.io.EOFException
<br>>>
<br>>> at java.io.DataInputStream.readShort(DataInputStream.java:315)
<br>>>
<br>>> at
<br>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:
<br>>> 79)
<br>>>
<br>>> at java.lang.Thread.run(Thread.java:636)
<br>>>
<br>>>
<br>>>
<br>>> NAME NODE LOG
<br>>>
<br>>> 2009-12-04 12:50:11,539 INFO http.HttpServer - Version Jetty/5.1.4
<br>>>
<br>>> 2009-12-04 12:50:11,573 INFO util.Credential - Checking Resource
<br>>> aliases
<br>>>
<br>>> 2009-12-04 12:50:12,488 INFO util.Container - Started
<br>>> org.mortbay.jetty.servlet.WebApplicationHandler@19fe451
<br>>>
<br>>> 2009-12-04 12:50:12,565 INFO util.Container - Started
<br>>> WebApplicationContext[/static,/static]
<br>>>
<br>>> 2009-12-04 12:50:12,891 INFO util.Container - Started
<br>>> org.mortbay.jetty.servlet.WebApplicationHandler@1570945
<br>>>
<br>>> 2009-12-04 12:50:12,891 INFO util.Container - Started
<br>>> WebApplicationContext[/logs,/logs]
<br>>>
<br>>> 2009-12-04 12:50:13,569 INFO util.Container - Started
<br>>> org.mortbay.jetty.servlet.WebApplicationHandler@11410e5
<br>>>
<br>>> 2009-12-04 12:50:13,582 INFO util.Container - Started
<br>>> WebApplicationContext[/,/]
<br>>>
<br>>> 2009-12-04 12:50:13,613 INFO http.SocketListener - Started
<br>>> SocketListener on 0.0.0.0:50070
<br>>>
<br>>> 2009-12-04 12:50:13,613 INFO util.Container - Started
<br>>> org.mortbay.jetty.Server@173ec72
<br>>>
<br>>>
<br>>>
<br>>> SECONDARY NAMENODE LOG
<br>>>
<br>>> 2009-12-04 12:50:19,163 INFO http.HttpServer - Version Jetty/5.1.4
<br>>>
<br>>> 2009-12-04 12:50:19,207 INFO util.Credential - Checking Resource
<br>>> aliases
<br>>>
<br>>> 2009-12-04 12:50:20,365 INFO util.Container - Started
<br>>> org.mortbay.jetty.servlet.WebApplicationHandler@174d93a
<br>>>
<br>>> 2009-12-04 12:50:20,454 INFO util.Container - Started
<br>>> WebApplicationContext[/static,/static]
<br>>>
<br>>> 2009-12-04 12:50:21,396 INFO util.Container - Started
<br>>> org.mortbay.jetty.servlet.WebApplicationHandler@31f2a7
<br>>>
<br>>> 2009-12-04 12:50:21,396 INFO util.Container - Started
<br>>> WebApplicationContext[/logs,/logs]
<br>>>
<br>>> 2009-12-04 12:50:21,533 INFO servlet.XMLConfiguration - No
<br>>> WEB-INF/web.xml in file:/mnt/nutch/nutch-1.0/webapps/secondary. Serving
<br>>> files and default/dynamic servlets only
<br>>>
<br>>> 2009-12-04 12:50:22,206 INFO util.Container - Started
<br>>> org.mortbay.jetty.servlet.WebApplicationHandler@383118
<br>>>
<br>>> 2009-12-04 12:50:22,785 INFO util.Container - Started
<br>>> WebApplicationContext[/,/]
<br>>>
<br>>> 2009-12-04 12:50:22,787 INFO http.SocketListener - Started
<br>>> SocketListener on 0.0.0.0:50090
<br>>>
<br>>> 2009-12-04 12:50:22,787 INFO util.Container - Started
<br>>> org.mortbay.jetty.Server@297ffb
<br>>>
<br>>> 2009-12-04 12:50:22,787 WARN namenode.SecondaryNameNode - Checkpoint
<br>>> Period :3600 secs (60 min)
<br>>>
<br>>> 2009-12-04 12:50:22,787 WARN namenode.SecondaryNameNode - Log Size
<br>>> Trigger :67108864 bytes (65536 KB)
<br>>>
<br>>> 2009-12-04 12:55:23,908 WARN namenode.SecondaryNameNode - Checkpoint
<br>>> done. New Image Size: 1056
<br>>>
<br>>>
<br>>>
<br>>> HADOOP LOG
<br>>>
<br>>> 2009-12-04 12:54:20,708 WARN hdfs.DFSClient - DataStreamer Exception:
<br>>> java.io.IOException: Unable to create new block.
<br>>>
<br>>> at
<br>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(D
<br>>> FSClient.java:2722)
<br>>>
<br>>> at
<br>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.j
<br>>> ava:1996)
<br>>>
<br>>> at
<br>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCli
<br>>> ent.java:2183)
<br>>>
<br>>>
<br>>>
<br>>> 2009-12-04 12:54:20,709 WARN hdfs.DFSClient - Error Recovery for block
<br>>> blk_5506837520665828594_1002 bad datanode[0] nodes == null
<br>>>
<br>>> 2009-12-04 12:54:20,709 WARN hdfs.DFSClient - Could not get block
<br>>> locations. Source file "/user/nutch/blub/hadoop-nutch-tasktracker.pid" -
<br>>> Aborting...
<br>>>
<br>>>
<br>>>
<br>>> DATA NODE LOG on the slave
<br>>>
<br>>> 2009-12-04 12:49:49,433 INFO http.HttpServer - Version Jetty/5.1.4
<br>>>
<br>>> 2009-12-04 12:49:49,438 INFO util.Credential - Checking Resource
<br>>> aliases
<br>>>
<br>>> 2009-12-04 12:49:50,288 INFO util.Container - Started
<br>>> org.mortbay.jetty.servlet.WebApplicationHandler@e45b5e
<br>>>
<br>>> 2009-12-04 12:49:50,357 INFO util.Container - Started
<br>>> WebApplicationContext[/static,/static]
<br>>>
<br>>> 2009-12-04 12:49:50,555 INFO util.Container - Started
<br>>> org.mortbay.jetty.servlet.WebApplicationHandler@2016b0
<br>>>
<br>>> 2009-12-04 12:49:50,555 INFO util.Container - Started
<br>>> WebApplicationContext[/logs,/logs]
<br>>>
<br>>> 2009-12-04 12:49:50,816 INFO util.Container - Started
<br>>> org.mortbay.jetty.servlet.WebApplicationHandler@118278a
<br>>>
<br>>> 2009-12-04 12:49:50,820 INFO util.Container - Started
<br>>> WebApplicationContext[/,/]
<br>>>
<br>>> 2009-12-04 12:49:50,849 INFO http.SocketListener - Started
<br>>> SocketListener on 0.0.0.0:50075
<br>>>
<br>>> 2009-12-04 12:49:50,849 INFO util.Container - Started
<br>>> org.mortbay.jetty.Server@b02928
<br>>>
<br>>>
<br>>>
<br>>> HADOOP SITE XML
<br>>>
<br>>> <property>
<br>>>
<br>>> <name>fs.default.name</name>
<br>>>
<br>>> <value>hdfs://(yes here is the right ip):9000</value>
<br>>>
<br>>> <description>
<br>>>
<br>>> The name of the default file system. Either the literal string
<br>>>
<br>>> "local" or a host:port for NDFS.
<br>>>
<br>>> </description>
<br>>>
<br>>> </property>
<br>>>
<br>>>
<br>>>
<br>>> <!-- Gibt an wo der JobTracker (koordiniert die (MapReduce-)Auftraege)
<br>>> zu finden ist. -->
<br>>>
<br>>> <property>
<br>>>
<br>>> <name>mapred.job.tracker</name>
<br>>>
<br>>> <value>hdfs://(here to):9001</value>
<br>>>
<br>>> <description>
<br>>>
<br>>> The host and port that the MapReduce job tracker runs at. If
<br>>>
<br>>> "local", then jobs are run in-process as a single map and
<br>>>
<br>>> reduce task.
<br>>>
<br>>> </description>
<br>>>
<br>>> </property>
<br>>>
<br>>>
<br>>>
<br>>> <!-- Gibt an wieviele MapJobs gleichzeitig laufen duerfen-->
<br>>>
<br>>> <property>
<br>>>
<br>>> <name>mapred.tasktracker.map.tasks.maximum</name>
<br>>>
<br>>> <value>2</value>
<br>>>
<br>>> <description>
<br>>>
<br>>> define mapred.map tasks to be number of slave hosts
<br>>>
<br>>> </description>
<br>>>
<br>>> </property>
<br>>>
<br>>>
<br>>>
<br>>> <!-- Gibt an wieviele ReduceJobs gleichzeitig laufen duerfen-->
<br>>>
<br>>> <property>
<br>>>
<br>>> <name>mapred.tasktracker.reduce.tasks.maximum</name>
<br>>>
<br>>> <value>2</value>
<br>>>
<br>>> <description>
<br>>>
<br>>> define mapred.reduce tasks to be number of slave hosts
<br>>>
<br>>> </description>
<br>>>
<br>>> </property>
<br>>>
<br>>>
<br>>>
<br>>> <property>
<br>>>
<br>>> <name>mapred.child.java.opts</name>
<br>>>
<br>>> <value>-Xmx1500m</value>
<br>>>
<br>>> </property>
<br>>>
<br>>>
<br>>>
<br>>> <property>
<br>>>
<br>>> <name>mapred.jobtracker.restart.recover</name>
<br>>>
<br>>> <value>true</value>
<br>>>
<br>>> </property>
<br>>>
<br>>>
<br>>>
<br>>> <!-- Die naechsten Einstellungen geben an wo das HadoopFS seine Datein
<br>>> auf der Festplatte jeder Instanz speichert. -->
<br>>>
<br>>> <property>
<br>>>
<br>>> <name>dfs.name.dir</name>
<br>>>
<br>>> <value>/nutch/filesystem/name</value>
<br>>>
<br>>> </property>
<br>>>
<br>>>
<br>>>
<br>>> <property>
<br>>>
<br>>> <name>dfs.data.dir</name>
<br>>>
<br>>> <value>/nutch/filesystem/data</value>
<br>>>
<br>>> </property>
<br>>>
<br>>>
<br>>>
<br>>> <property>
<br>>>
<br>>> <name>mapred.system.dir</name>
<br>>>
<br>>> <value>/nutch/filesystem/mapreduce/system</value>
<br>>>
<br>>> </property>
<br>>>
<br>>>
<br>>>
<br>>> <property>
<br>>>
<br>>> <name>mapred.local.dir</name>
<br>>>
<br>>> <value>/nutch/filesystem/mapreduce/local</value>
<br>>>
<br>>> </property>
<br>>>
<br>>>
<br>>>
<br>>> <!-- Gibt an wieviele Replikate einer Datei im Dateisystem vorhanden
<br>>> sein muessen damit sie erreichbar ist. Am Anfang 1 -->
<br>>>
<br>>> <property>
<br>>>
<br>>> <name>dfs.replication</name>
<br>>>
<br>>> <value>2</value>
<br>>>
<br>>> </property>
<br>>>
<br>>>
<br>>>
<br>>> I hope someone can help me.
<br>>>
<br>>>
<br>>>
<br>>> Thanks
<br>>>
<br>>>
<br>>>
<br>>> Tom
<br>>>
<br>>>
<br>>
<br>>
<br>> --
<br>> -MilleBii-
<br>>
</div><br><br>--
<br>-MilleBii-
<br><p>From forum: <a href="http://old.nabble.com/Nutch---User-f375.html" embed="fixTarget[375]" target="_top" >Nutch - User</a></p>tag:old.nabble.com,2006:post-26644288RE: How to force recrawl of everything2009-12-04T07:36:09Z2009-12-04T07:36:09ZPeters, Vijaya<br>Running:
<br>bin/nutch readdb crawldb -url <url> I got the following exception.
<br>Also, how do I force a recrawl in Nutch 1.0?
<br><br><br>Exception in thread "main" java.lang.ArithmeticException: / by zero
<br> at
<br>org.apache.hadoop.mapred.lib.HashPartitioner.getPartition(HashPartiti
<br>oner.java:32)
<br> at
<br>org.apache.hadoop.mapred.MapFileOutputFormat.getEntry(MapFileOutputFo
<br>rmat.java:104)
<br> at
<br>org.apache.nutch.crawl.CrawlDbReader.get(CrawlDbReader.java:380)
<br> at
<br>org.apache.nutch.crawl.CrawlDbReader.readUrl(CrawlDbReader.java:386)
<br> at
<br>org.apache.nutch.crawl.CrawlDbReader.main(CrawlDbReader.java:511)
<br><br>Vijaya Peters
<br>SRA International, Inc.
<br>4350 Fair Lakes Court North
<br>Room 4004
<br>Fairfax, VA 22033
<br>Tel: 703-502-1184
<br><br>www.sra.com
<br>Named to FORTUNE's "100 Best Companies to Work For" list for 10
<br>consecutive years
<br>P Please consider the environment before printing this e-mail
<br>This electronic message transmission contains information from SRA
<br>International, Inc. which may be confidential, privileged or
<br>proprietary. The information is intended for the use of the individual
<br>or entity named above. If you are not the intended recipient, be aware
<br>that any disclosure, copying, distribution, or use of the contents of
<br>this information is strictly prohibited. If you have received this
<br>electronic information in error, please notify us immediately by
<br>telephone at 866-584-2143.
<br>-----Original Message-----
<br>From: reinhard schwab [mailto:<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26644288&i=0" target="_top" rel="nofollow">reinhard.schwab@...</a>]
<br>Sent: Friday, December 04, 2009 8:32 AM
<br>To: <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26644288&i=1" target="_top" rel="nofollow">nutch-user@...</a>
<br>Subject: Re: How to force recrawl of everything
<br><br>Peters, Vijaya schrieb:
<br>> I am using Nutch 1.0. I want to perform a 'clean' crawl.
<br>>
<br>>
<br>>
<br>> I see the force option in this patch: NUTCH-601v1.0.patch
<br>>
<br><<a href="https://issues.apache.org/jira/secure/attachment/12375717/NUTCH-601v1.0" target="_top" rel="nofollow">https://issues.apache.org/jira/secure/attachment/12375717/NUTCH-601v1.0</a><br>> .patch>
<br>>
<br>> Do I have to make those code changes, or does Nutch 1.0 have another
<br>way
<br>> to do this?
<br>>
<br>>
<br>>
<br>> Also, everytime I do another crawl, I see the same file being fetched
<br>> over and over again. Is it appending the same url over and over to the
<br>>
<br>which file?
<br>you can check the crawl date of this file with
<br><br>reinhard@thord:>bin/nutch readdb <crawldb> -url <url>
<br><br><div class='shrinkable-quote'><br>> fetch list?
<br>>
<br>>
<br>>
<br>> Thanks,
<br>>
<br>> - Vijaya
<br>>
<br>>
<br>>
<br>>
<br>>
<br>> Vijaya Peters
<br>> SRA International, Inc.
<br>> 4350 Fair Lakes Court North
<br>> Room 4004
<br>> Fairfax, VA 22033
<br>> Tel: 703-502-1184
<br>>
<br>> www.sra.com <<a href="http://www.sra.com/" target="_top" rel="nofollow">http://www.sra.com/</a>>
<br>> Named to FORTUNE's "100 Best Companies to Work For" list for 10
<br>> consecutive years
<br>>
<br>> P Please consider the environment before printing this e-mail
<br>>
<br>> This electronic message transmission contains information from SRA
<br>> International, Inc. which may be confidential, privileged or
<br>> proprietary. The information is intended for the use of the
</div>individual
<br>> or entity named above. If you are not the intended recipient, be
<br>aware
<br>> that any disclosure, copying, distribution, or use of the contents of
<br>> this information is strictly prohibited. If you have received this
<br>> electronic information in error, please notify us immediately by
<br>> telephone at 866-584-2143.
<br>>
<br>>
<br>>
<br>>
<br>>
<br><br><p>From forum: <a href="http://old.nabble.com/Nutch---User-f375.html" embed="fixTarget[375]" target="_top" >Nutch - User</a></p>tag:old.nabble.com,2006:post-26644263RE: Problems with a new Installation of Nutch2009-12-04T07:35:08Z2009-12-04T07:35:08ZTom LandvoigtHi,
<br><br>I don't have tomcat on this system because I don't want to use the websearch. But if it is necessary for hadoop what I don’t think I will install it.
<br><br>nutch@ip-10-224-113-210:/nutch/search> ./bin/hadoop fs -ls /
<br>Found 1 items
<br>-rw-r--r-- 2 nutch supergroup 0 2009-12-04 14:04 /url.txt
<br>nutch@ip-10-224-113-210:/nutch/search>
<br><br>I get the normal answer but the file is empty.
<br><br>-----Original Message-----
<br>From: MilleBii [mailto:<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26644263&i=0" target="_top" rel="nofollow">millebii@...</a>]
<br>Sent: Freitag, 4. Dezember 2009 15:06
<br>To: <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26644263&i=1" target="_top" rel="nofollow">nutch-user@...</a>
<br>Subject: Re: Problems with a new Installation of Nutch
<br><br>Did you check with the web interface ? It gives a lot of info you can
<br>even browse the file system.
<br><br>Try hadoop fs -ls to see what it gives you ?
<br><br>2009/12/4, Tom Landvoigt <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26644263&i=2" target="_top" rel="nofollow">tom.landvoigt@...</a>>:
<div class='shrinkable-quote'><br>> Hallo,
<br>>
<br>>
<br>>
<br>> I hope someone can help me.
<br>>
<br>>
<br>>
<br>> I installed nutch on 2 Amazon EC2 computers. Everything is fine but I
<br>> can't put data in the hdfs.
<br>>
<br>>
<br>>
<br>> I formatted the namenode and start the hdfs with start all.
<br>>
<br>>
<br>>
<br>> All java processes start properly, but when I want to make hadoop fs
<br>> -put something / I get these logs:
<br>>
<br>>
<br>>
<br>>
<br>>
<br>>
<br>>
<br>> nutch@bla:/nutch/search> ./bin/hadoop fs -put
<br>> /tmp/hadoop-nutch-tasktracker.pid blub
<br>>
<br>> put: Protocol not available
<br>>
<br>>
<br>>
<br>> DATA NODE LOG on the master:
<br>>
<br>> 2009-12-04 12:50:15,566 INFO http.HttpServer - Version Jetty/5.1.4
<br>>
<br>> 2009-12-04 12:50:15,582 INFO util.Credential - Checking Resource
<br>> aliases
<br>>
<br>> 2009-12-04 12:50:16,483 INFO util.Container - Started
<br>> org.mortbay.jetty.servlet.WebApplicationHandler@e45b5e
<br>>
<br>> 2009-12-04 12:50:16,614 INFO util.Container - Started
<br>> WebApplicationContext[/static,/static]
<br>>
<br>> 2009-12-04 12:50:16,882 INFO util.Container - Started
<br>> org.mortbay.jetty.servlet.WebApplicationHandler@1284fd4
<br>>
<br>> 2009-12-04 12:50:16,883 INFO util.Container - Started
<br>> WebApplicationContext[/logs,/logs]
<br>>
<br>> 2009-12-04 12:50:17,827 INFO util.Container - Started
<br>> org.mortbay.jetty.servlet.WebApplicationHandler@39c8c1
<br>>
<br>> 2009-12-04 12:50:17,849 INFO util.Container - Started
<br>> WebApplicationContext[/,/]
<br>>
<br>> 2009-12-04 12:50:18,485 INFO http.SocketListener - Started
<br>> SocketListener on 0.0.0.0:50075
<br>>
<br>> 2009-12-04 12:50:18,485 INFO util.Container - Started
<br>> org.mortbay.jetty.Server@36527f
<br>>
<br>> 2009-12-04 12:54:20,745 ERROR datanode.DataNode -
<br>> DatanodeRegistration(10.224.113.210:50010,
<br>> storageID=DS-1135263253-10.224.113.210-50010-1259926637370,
<br>> infoPort=50075, ipcPort=50020):DataXceiver
<br>>
<br>> java.io.EOFException
<br>>
<br>> at java.io.DataInputStream.readShort(DataInputStream.java:315)
<br>>
<br>> at
<br>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:
<br>> 79)
<br>>
<br>> at java.lang.Thread.run(Thread.java:636)
<br>>
<br>> 2009-12-04 12:54:20,746 ERROR datanode.DataNode -
<br>> DatanodeRegistration(10.224.113.210:50010,
<br>> storageID=DS-1135263253-10.224.113.210-50010-1259926637370,
<br>> infoPort=50075, ipcPort=50020):DataXceiver
<br>>
<br>> java.io.EOFException
<br>>
<br>> at java.io.DataInputStream.readShort(DataInputStream.java:315)
<br>>
<br>> at
<br>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:
<br>> 79)
<br>>
<br>> at java.lang.Thread.run(Thread.java:636)
<br>>
<br>> 2009-12-04 12:54:20,747 ERROR datanode.DataNode -
<br>> DatanodeRegistration(10.224.113.210:50010,
<br>> storageID=DS-1135263253-10.224.113.210-50010-1259926637370,
<br>> infoPort=50075, ipcPort=50020):DataXceiver
<br>>
<br>> java.io.EOFException
<br>>
<br>> at java.io.DataInputStream.readShort(DataInputStream.java:315)
<br>>
<br>> at
<br>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:
<br>> 79)
<br>>
<br>> at java.lang.Thread.run(Thread.java:636)
<br>>
<br>> 2009-12-04 12:54:20,747 ERROR datanode.DataNode -
<br>> DatanodeRegistration(10.224.113.210:50010,
<br>> storageID=DS-1135263253-10.224.113.210-50010-1259926637370,
<br>> infoPort=50075, ipcPort=50020):DataXceiver
<br>>
<br>> java.io.EOFException
<br>>
<br>> at java.io.DataInputStream.readShort(DataInputStream.java:315)
<br>>
<br>> at
<br>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:
<br>> 79)
<br>>
<br>> at java.lang.Thread.run(Thread.java:636)
<br>>
<br>>
<br>>
<br>> NAME NODE LOG
<br>>
<br>> 2009-12-04 12:50:11,539 INFO http.HttpServer - Version Jetty/5.1.4
<br>>
<br>> 2009-12-04 12:50:11,573 INFO util.Credential - Checking Resource
<br>> aliases
<br>>
<br>> 2009-12-04 12:50:12,488 INFO util.Container - Started
<br>> org.mortbay.jetty.servlet.WebApplicationHandler@19fe451
<br>>
<br>> 2009-12-04 12:50:12,565 INFO util.Container - Started
<br>> WebApplicationContext[/static,/static]
<br>>
<br>> 2009-12-04 12:50:12,891 INFO util.Container - Started
<br>> org.mortbay.jetty.servlet.WebApplicationHandler@1570945
<br>>
<br>> 2009-12-04 12:50:12,891 INFO util.Container - Started
<br>> WebApplicationContext[/logs,/logs]
<br>>
<br>> 2009-12-04 12:50:13,569 INFO util.Container - Started
<br>> org.mortbay.jetty.servlet.WebApplicationHandler@11410e5
<br>>
<br>> 2009-12-04 12:50:13,582 INFO util.Container - Started
<br>> WebApplicationContext[/,/]
<br>>
<br>> 2009-12-04 12:50:13,613 INFO http.SocketListener - Started
<br>> SocketListener on 0.0.0.0:50070
<br>>
<br>> 2009-12-04 12:50:13,613 INFO util.Container - Started
<br>> org.mortbay.jetty.Server@173ec72
<br>>
<br>>
<br>>
<br>> SECONDARY NAMENODE LOG
<br>>
<br>> 2009-12-04 12:50:19,163 INFO http.HttpServer - Version Jetty/5.1.4
<br>>
<br>> 2009-12-04 12:50:19,207 INFO util.Credential - Checking Resource
<br>> aliases
<br>>
<br>> 2009-12-04 12:50:20,365 INFO util.Container - Started
<br>> org.mortbay.jetty.servlet.WebApplicationHandler@174d93a
<br>>
<br>> 2009-12-04 12:50:20,454 INFO util.Container - Started
<br>> WebApplicationContext[/static,/static]
<br>>
<br>> 2009-12-04 12:50:21,396 INFO util.Container - Started
<br>> org.mortbay.jetty.servlet.WebApplicationHandler@31f2a7
<br>>
<br>> 2009-12-04 12:50:21,396 INFO util.Container - Started
<br>> WebApplicationContext[/logs,/logs]
<br>>
<br>> 2009-12-04 12:50:21,533 INFO servlet.XMLConfiguration - No
<br>> WEB-INF/web.xml in file:/mnt/nutch/nutch-1.0/webapps/secondary. Serving
<br>> files and default/dynamic servlets only
<br>>
<br>> 2009-12-04 12:50:22,206 INFO util.Container - Started
<br>> org.mortbay.jetty.servlet.WebApplicationHandler@383118
<br>>
<br>> 2009-12-04 12:50:22,785 INFO util.Container - Started
<br>> WebApplicationContext[/,/]
<br>>
<br>> 2009-12-04 12:50:22,787 INFO http.SocketListener - Started
<br>> SocketListener on 0.0.0.0:50090
<br>>
<br>> 2009-12-04 12:50:22,787 INFO util.Container - Started
<br>> org.mortbay.jetty.Server@297ffb
<br>>
<br>> 2009-12-04 12:50:22,787 WARN namenode.SecondaryNameNode - Checkpoint
<br>> Period :3600 secs (60 min)
<br>>
<br>> 2009-12-04 12:50:22,787 WARN namenode.SecondaryNameNode - Log Size
<br>> Trigger :67108864 bytes (65536 KB)
<br>>
<br>> 2009-12-04 12:55:23,908 WARN namenode.SecondaryNameNode - Checkpoint
<br>> done. New Image Size: 1056
<br>>
<br>>
<br>>
<br>> HADOOP LOG
<br>>
<br>> 2009-12-04 12:54:20,708 WARN hdfs.DFSClient - DataStreamer Exception:
<br>> java.io.IOException: Unable to create new block.
<br>>
<br>> at
<br>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(D
<br>> FSClient.java:2722)
<br>>
<br>> at
<br>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.j
<br>> ava:1996)
<br>>
<br>> at
<br>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCli
<br>> ent.java:2183)
<br>>
<br>>
<br>>
<br>> 2009-12-04 12:54:20,709 WARN hdfs.DFSClient - Error Recovery for block
<br>> blk_5506837520665828594_1002 bad datanode[0] nodes == null
<br>>
<br>> 2009-12-04 12:54:20,709 WARN hdfs.DFSClient - Could not get block
<br>> locations. Source file "/user/nutch/blub/hadoop-nutch-tasktracker.pid" -
<br>> Aborting...
<br>>
<br>>
<br>>
<br>> DATA NODE LOG on the slave
<br>>
<br>> 2009-12-04 12:49:49,433 INFO http.HttpServer - Version Jetty/5.1.4
<br>>
<br>> 2009-12-04 12:49:49,438 INFO util.Credential - Checking Resource
<br>> aliases
<br>>
<br>> 2009-12-04 12:49:50,288 INFO util.Container - Started
<br>> org.mortbay.jetty.servlet.WebApplicationHandler@e45b5e
<br>>
<br>> 2009-12-04 12:49:50,357 INFO util.Container - Started
<br>> WebApplicationContext[/static,/static]
<br>>
<br>> 2009-12-04 12:49:50,555 INFO util.Container - Started
<br>> org.mortbay.jetty.servlet.WebApplicationHandler@2016b0
<br>>
<br>> 2009-12-04 12:49:50,555 INFO util.Container - Started
<br>> WebApplicationContext[/logs,/logs]
<br>>
<br>> 2009-12-04 12:49:50,816 INFO util.Container - Started
<br>> org.mortbay.jetty.servlet.WebApplicationHandler@118278a
<br>>
<br>> 2009-12-04 12:49:50,820 INFO util.Container - Started
<br>> WebApplicationContext[/,/]
<br>>
<br>> 2009-12-04 12:49:50,849 INFO http.SocketListener - Started
<br>> SocketListener on 0.0.0.0:50075
<br>>
<br>> 2009-12-04 12:49:50,849 INFO util.Container - Started
<br>> org.mortbay.jetty.Server@b02928
<br>>
<br>>
<br>>
<br>> HADOOP SITE XML
<br>>
<br>> <property>
<br>>
<br>> <name>fs.default.name</name>
<br>>
<br>> <value>hdfs://(yes here is the right ip):9000</value>
<br>>
<br>> <description>
<br>>
<br>> The name of the default file system. Either the literal string
<br>>
<br>> "local" or a host:port for NDFS.
<br>>
<br>> </description>
<br>>
<br>> </property>
<br>>
<br>>
<br>>
<br>> <!-- Gibt an wo der JobTracker (koordiniert die (MapReduce-)Auftraege)
<br>> zu finden ist. -->
<br>>
<br>> <property>
<br>>
<br>> <name>mapred.job.tracker</name>
<br>>
<br>> <value>hdfs://(here to):9001</value>
<br>>
<br>> <description>
<br>>
<br>> The host and port that the MapReduce job tracker runs at. If
<br>>
<br>> "local", then jobs are run in-process as a single map and
<br>>
<br>> reduce task.
<br>>
<br>> </description>
<br>>
<br>> </property>
<br>>
<br>>
<br>>
<br>> <!-- Gibt an wieviele MapJobs gleichzeitig laufen duerfen-->
<br>>
<br>> <property>
<br>>
<br>> <name>mapred.tasktracker.map.tasks.maximum</name>
<br>>
<br>> <value>2</value>
<br>>
<br>> <description>
<br>>
<br>> define mapred.map tasks to be number of slave hosts
<br>>
<br>> </description>
<br>>
<br>> </property>
<br>>
<br>>
<br>>
<br>> <!-- Gibt an wieviele ReduceJobs gleichzeitig laufen duerfen-->
<br>>
<br>> <property>
<br>>
<br>> <name>mapred.tasktracker.reduce.tasks.maximum</name>
<br>>
<br>> <value>2</value>
<br>>
<br>> <description>
<br>>
<br>> define mapred.reduce tasks to be number of slave hosts
<br>>
<br>> </description>
<br>>
<br>> </property>
<br>>
<br>>
<br>>
<br>> <property>
<br>>
<br>> <name>mapred.child.java.opts</name>
<br>>
<br>> <value>-Xmx1500m</value>
<br>>
<br>> </property>
<br>>
<br>>
<br>>
<br>> <property>
<br>>
<br>> <name>mapred.jobtracker.restart.recover</name>
<br>>
<br>> <value>true</value>
<br>>
<br>> </property>
<br>>
<br>>
<br>>
<br>> <!-- Die naechsten Einstellungen geben an wo das HadoopFS seine Datein
<br>> auf der Festplatte jeder Instanz speichert. -->
<br>>
<br>> <property>
<br>>
<br>> <name>dfs.name.dir</name>
<br>>
<br>> <value>/nutch/filesystem/name</value>
<br>>
<br>> </property>
<br>>
<br>>
<br>>
<br>> <property>
<br>>
<br>> <name>dfs.data.dir</name>
<br>>
<br>> <value>/nutch/filesystem/data</value>
<br>>
<br>> </property>
<br>>
<br>>
<br>>
<br>> <property>
<br>>
<br>> <name>mapred.system.dir</name>
<br>>
<br>> <value>/nutch/filesystem/mapreduce/system</value>
<br>>
<br>> </property>
<br>>
<br>>
<br>>
<br>> <property>
<br>>
<br>> <name>mapred.local.dir</name>
<br>>
<br>> <value>/nutch/filesystem/mapreduce/local</value>
<br>>
<br>> </property>
<br>>
<br>>
<br>>
<br>> <!-- Gibt an wieviele Replikate einer Datei im Dateisystem vorhanden
<br>> sein muessen damit sie erreichbar ist. Am Anfang 1 -->
<br>>
<br>> <property>
<br>>
<br>> <name>dfs.replication</name>
<br>>
<br>> <value>2</value>
<br>>
<br>> </property>
<br>>
<br>>
<br>>
<br>> I hope someone can help me.
<br>>
<br>>
<br>>
<br>> Thanks
<br>>
<br>>
<br>>
<br>> Tom
<br>>
<br>>
</div><br><br>--
<br>-MilleBii-
<br><p>From forum: <a href="http://old.nabble.com/Nutch---User-f375.html" embed="fixTarget[375]" target="_top" >Nutch - User</a></p>tag:old.nabble.com,2006:post-26643852unsubscribe from nutch-user2009-12-04T07:07:02Z2009-12-04T07:07:02ZLukas, RayWell three is a charm.. I need to move these to a different email as
<br>well.. Please if you could.. Could we also remove this email address as
<br>well..
<br>Thanks
<br>ray
<br><br>-----Original Message-----
<br>From: M S Ram [mailto:<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26643852&i=0" target="_top" rel="nofollow">msram@...</a>]
<br>Sent: Friday, December 04, 2009 10:01 AM
<br>To: <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26643852&i=1" target="_top" rel="nofollow">nutch-user@...</a>
<br>Subject: Re: unsubscribe from nutch-user
<br><br>Same here. Please remove my ID also from the mailing list.
<br><br>Thanks,
<br>MSR
<br><br>rengan xu wrote:
<br>> To whom it may concern,
<br>>
<br>> Hello! Because I will use this E-mail for special purpose. I will use
<br>> another E-mail to subscribe in nutch-user. So I want to unsubscribe
<br>from
<br>> nutch-user.
<br>>
<br>> Thank you!
<br>>
<br>>
<br>>
<br><br><br><p>From forum: <a href="http://old.nabble.com/Nutch---User-f375.html" embed="fixTarget[375]" target="_top" >Nutch - User</a></p>tag:old.nabble.com,2006:post-26643844Re: unsubscribe from nutch-user2009-12-04T07:06:36Z2009-12-04T07:06:36Zprashant ullegaddi-2Take a look at it:
<br><br><a href="http://lucene.apache.org/nutch/mailing_lists.html" target="_top" rel="nofollow">http://lucene.apache.org/nutch/mailing_lists.html</a><br><br>or
<br>probably sending a blank mail to:
<br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26643844&i=0" target="_top" rel="nofollow">nutch-user-unsubscribe@...</a> also
<br>work.
<br><br>Thanks,
<br>Prashant.
<br><br>On Fri, Dec 4, 2009 at 8:30 PM, M S Ram <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26643844&i=1" target="_top" rel="nofollow">msram@...</a>> wrote:
<br><div class='shrinkable-quote'><br>> Same here. Please remove my ID also from the mailing list.
<br>>
<br>> Thanks,
<br>> MSR
<br>>
<br>> rengan xu wrote:
<br>>
<br>>> To whom it may concern,
<br>>>
<br>>> Hello! Because I will use this E-mail for special purpose. I will use
<br>>> another E-mail to subscribe in nutch-user. So I want to unsubscribe from
<br>>> nutch-user.
<br>>>
<br>>> Thank you!
<br>>>
<br>>>
<br>>>
<br>>>
<br>>
<br>>
<br>>
</div><br><br>--
<br>Thanks,
<br>Prashant Ullegaddi,
<br>Search and Information Extraction Lab,
<br>IIIT-Hyderabad, INDIA.
<br><p>From forum: <a href="http://old.nabble.com/Nutch---User-f375.html" embed="fixTarget[375]" target="_top" >Nutch - User</a></p>tag:old.nabble.com,2006:post-26643804Re: unsubscribe from nutch-user2009-12-04T07:00:51Z2009-12-04T07:00:51ZM S RamSame here. Please remove my ID also from the mailing list.
<br><br>Thanks,
<br>MSR
<br><br>rengan xu wrote:
<div class='shrinkable-quote'><br>> To whom it may concern,
<br>>
<br>> Hello! Because I will use this E-mail for special purpose. I will use
<br>> another E-mail to subscribe in nutch-user. So I want to unsubscribe from
<br>> nutch-user.
<br>>
<br>> Thank you!
<br>>
<br>>
<br>>
</div><br><br><p>From forum: <a href="http://old.nabble.com/Nutch---User-f375.html" embed="fixTarget[375]" target="_top" >Nutch - User</a></p>tag:old.nabble.com,2006:post-26643628unsubscribe from nutch-user2009-12-04T06:50:48Z2009-12-04T06:50:48Zrengan xuTo whom it may concern,
<br><br>Hello! Because I will use this E-mail for special purpose. I will use
<br>another E-mail to subscribe in nutch-user. So I want to unsubscribe from
<br>nutch-user.
<br><br>Thank you!
<br><br><br>--
<br>Regards,
<br>Xu, Rengan
<br><br>School of Computer Science and Information Engineering,
<br>Hefei University of Technology, Hefei 230009, China
<br><p>From forum: <a href="http://old.nabble.com/Nutch---User-f375.html" embed="fixTarget[375]" target="_top" >Nutch - User</a></p>tag:old.nabble.com,2006:post-26643195Re: Can nutch pause, stop and start where it left off?2009-12-04T06:19:16Z2009-12-04T06:19:16ZMilleBiiNutch behaves ...
<br>So by default it will not fetch more 1 url every 5s (setting
<br>changeable) to a given host (by name or ip depending on the nutch
<br>conf file).
<br>So actually you will find the opposite it is very slow for a single
<br>site... Speed comes when you fetch several sites in parallel.
<br><br><br>2009/12/4, Jesse Hires <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26643195&i=0" target="_top" rel="nofollow">jhires@...</a>>:
<div class='shrinkable-quote'><br>> use the -topN flag to only grab a small number of URLs.
<br>> Also I believe there is also a setting you can put in nutch-site.xml that
<br>> can be used to slow down how many URLs you grab over time.
<br>>
<br>> Jesse
<br>>
<br>> int GetRandomNumber()
<br>> {
<br>> return 4; // Chosen by fair roll of dice
<br>> // Guaranteed to be random
<br>> } // xkcd.com
<br>>
<br>>
<br>>
<br>> On Fri, Dec 4, 2009 at 4:10 AM, Mr Hadoop <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26643195&i=1" target="_top" rel="nofollow">mrhadoop@...</a>> wrote:
<br>>
<br>>> I am just staring to learn nutch. One question I wanted to know is that
<br>>> can
<br>>> nutch pause, stop and start indexing a site on a incremental daily basis?
<br>>> My concern with nutch is that nutch behaving like a hog and crawling
<br>>> everything with huge bandwidth consumption and pissing off the many site
<br>>> owners.
<br>>>
<br>>> Can some experts shed some light in this?
<br>>>
<br>>
</div><br><br>--
<br>-MilleBii-
<br><p>From forum: <a href="http://old.nabble.com/Nutch---User-f375.html" embed="fixTarget[375]" target="_top" >Nutch - User</a></p>tag:old.nabble.com,2006:post-26643017Re: Problems with a new Installation of Nutch2009-12-04T06:05:50Z2009-12-04T06:05:50ZMilleBiiDid you check with the web interface ? It gives a lot of info you can
<br>even browse the file system.
<br><br>Try hadoop fs -ls to see what it gives you ?
<br><br>2009/12/4, Tom Landvoigt <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26643017&i=0" target="_top" rel="nofollow">tom.landvoigt@...</a>>:
<div class='shrinkable-quote'><br>> Hallo,
<br>>
<br>>
<br>>
<br>> I hope someone can help me.
<br>>
<br>>
<br>>
<br>> I installed nutch on 2 Amazon EC2 computers. Everything is fine but I
<br>> can't put data in the hdfs.
<br>>
<br>>
<br>>
<br>> I formatted the namenode and start the hdfs with start all.
<br>>
<br>>
<br>>
<br>> All java processes start properly, but when I want to make hadoop fs
<br>> -put something / I get these logs:
<br>>
<br>>
<br>>
<br>>
<br>>
<br>>
<br>>
<br>> nutch@bla:/nutch/search> ./bin/hadoop fs -put
<br>> /tmp/hadoop-nutch-tasktracker.pid blub
<br>>
<br>> put: Protocol not available
<br>>
<br>>
<br>>
<br>> DATA NODE LOG on the master:
<br>>
<br>> 2009-12-04 12:50:15,566 INFO http.HttpServer - Version Jetty/5.1.4
<br>>
<br>> 2009-12-04 12:50:15,582 INFO util.Credential - Checking Resource
<br>> aliases
<br>>
<br>> 2009-12-04 12:50:16,483 INFO util.Container - Started
<br>> org.mortbay.jetty.servlet.WebApplicationHandler@e45b5e
<br>>
<br>> 2009-12-04 12:50:16,614 INFO util.Container - Started
<br>> WebApplicationContext[/static,/static]
<br>>
<br>> 2009-12-04 12:50:16,882 INFO util.Container - Started
<br>> org.mortbay.jetty.servlet.WebApplicationHandler@1284fd4
<br>>
<br>> 2009-12-04 12:50:16,883 INFO util.Container - Started
<br>> WebApplicationContext[/logs,/logs]
<br>>
<br>> 2009-12-04 12:50:17,827 INFO util.Container - Started
<br>> org.mortbay.jetty.servlet.WebApplicationHandler@39c8c1
<br>>
<br>> 2009-12-04 12:50:17,849 INFO util.Container - Started
<br>> WebApplicationContext[/,/]
<br>>
<br>> 2009-12-04 12:50:18,485 INFO http.SocketListener - Started
<br>> SocketListener on 0.0.0.0:50075
<br>>
<br>> 2009-12-04 12:50:18,485 INFO util.Container - Started
<br>> org.mortbay.jetty.Server@36527f
<br>>
<br>> 2009-12-04 12:54:20,745 ERROR datanode.DataNode -
<br>> DatanodeRegistration(10.224.113.210:50010,
<br>> storageID=DS-1135263253-10.224.113.210-50010-1259926637370,
<br>> infoPort=50075, ipcPort=50020):DataXceiver
<br>>
<br>> java.io.EOFException
<br>>
<br>> at java.io.DataInputStream.readShort(DataInputStream.java:315)
<br>>
<br>> at
<br>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:
<br>> 79)
<br>>
<br>> at java.lang.Thread.run(Thread.java:636)
<br>>
<br>> 2009-12-04 12:54:20,746 ERROR datanode.DataNode -
<br>> DatanodeRegistration(10.224.113.210:50010,
<br>> storageID=DS-1135263253-10.224.113.210-50010-1259926637370,
<br>> infoPort=50075, ipcPort=50020):DataXceiver
<br>>
<br>> java.io.EOFException
<br>>
<br>> at java.io.DataInputStream.readShort(DataInputStream.java:315)
<br>>
<br>> at
<br>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:
<br>> 79)
<br>>
<br>> at java.lang.Thread.run(Thread.java:636)
<br>>
<br>> 2009-12-04 12:54:20,747 ERROR datanode.DataNode -
<br>> DatanodeRegistration(10.224.113.210:50010,
<br>> storageID=DS-1135263253-10.224.113.210-50010-1259926637370,
<br>> infoPort=50075, ipcPort=50020):DataXceiver
<br>>
<br>> java.io.EOFException
<br>>
<br>> at java.io.DataInputStream.readShort(DataInputStream.java:315)
<br>>
<br>> at
<br>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:
<br>> 79)
<br>>
<br>> at java.lang.Thread.run(Thread.java:636)
<br>>
<br>> 2009-12-04 12:54:20,747 ERROR datanode.DataNode -
<br>> DatanodeRegistration(10.224.113.210:50010,
<br>> storageID=DS-1135263253-10.224.113.210-50010-1259926637370,
<br>> infoPort=50075, ipcPort=50020):DataXceiver
<br>>
<br>> java.io.EOFException
<br>>
<br>> at java.io.DataInputStream.readShort(DataInputStream.java:315)
<br>>
<br>> at
<br>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:
<br>> 79)
<br>>
<br>> at java.lang.Thread.run(Thread.java:636)
<br>>
<br>>
<br>>
<br>> NAME NODE LOG
<br>>
<br>> 2009-12-04 12:50:11,539 INFO http.HttpServer - Version Jetty/5.1.4
<br>>
<br>> 2009-12-04 12:50:11,573 INFO util.Credential - Checking Resource
<br>> aliases
<br>>
<br>> 2009-12-04 12:50:12,488 INFO util.Container - Started
<br>> org.mortbay.jetty.servlet.WebApplicationHandler@19fe451
<br>>
<br>> 2009-12-04 12:50:12,565 INFO util.Container - Started
<br>> WebApplicationContext[/static,/static]
<br>>
<br>> 2009-12-04 12:50:12,891 INFO util.Container - Started
<br>> org.mortbay.jetty.servlet.WebApplicationHandler@1570945
<br>>
<br>> 2009-12-04 12:50:12,891 INFO util.Container - Started
<br>> WebApplicationContext[/logs,/logs]
<br>>
<br>> 2009-12-04 12:50:13,569 INFO util.Container - Started
<br>> org.mortbay.jetty.servlet.WebApplicationHandler@11410e5
<br>>
<br>> 2009-12-04 12:50:13,582 INFO util.Container - Started
<br>> WebApplicationContext[/,/]
<br>>
<br>> 2009-12-04 12:50:13,613 INFO http.SocketListener - Started
<br>> SocketListener on 0.0.0.0:50070
<br>>
<br>> 2009-12-04 12:50:13,613 INFO util.Container - Started
<br>> org.mortbay.jetty.Server@173ec72
<br>>
<br>>
<br>>
<br>> SECONDARY NAMENODE LOG
<br>>
<br>> 2009-12-04 12:50:19,163 INFO http.HttpServer - Version Jetty/5.1.4
<br>>
<br>> 2009-12-04 12:50:19,207 INFO util.Credential - Checking Resource
<br>> aliases
<br>>
<br>> 2009-12-04 12:50:20,365 INFO util.Container - Started
<br>> org.mortbay.jetty.servlet.WebApplicationHandler@174d93a
<br>>
<br>> 2009-12-04 12:50:20,454 INFO util.Container - Started
<br>> WebApplicationContext[/static,/static]
<br>>
<br>> 2009-12-04 12:50:21,396 INFO util.Container - Started
<br>> org.mortbay.jetty.servlet.WebApplicationHandler@31f2a7
<br>>
<br>> 2009-12-04 12:50:21,396 INFO util.Container - Started
<br>> WebApplicationContext[/logs,/logs]
<br>>
<br>> 2009-12-04 12:50:21,533 INFO servlet.XMLConfiguration - No
<br>> WEB-INF/web.xml in file:/mnt/nutch/nutch-1.0/webapps/secondary. Serving
<br>> files and default/dynamic servlets only
<br>>
<br>> 2009-12-04 12:50:22,206 INFO util.Container - Started
<br>> org.mortbay.jetty.servlet.WebApplicationHandler@383118
<br>>
<br>> 2009-12-04 12:50:22,785 INFO util.Container - Started
<br>> WebApplicationContext[/,/]
<br>>
<br>> 2009-12-04 12:50:22,787 INFO http.SocketListener - Started
<br>> SocketListener on 0.0.0.0:50090
<br>>
<br>> 2009-12-04 12:50:22,787 INFO util.Container - Started
<br>> org.mortbay.jetty.Server@297ffb
<br>>
<br>> 2009-12-04 12:50:22,787 WARN namenode.SecondaryNameNode - Checkpoint
<br>> Period :3600 secs (60 min)
<br>>
<br>> 2009-12-04 12:50:22,787 WARN namenode.SecondaryNameNode - Log Size
<br>> Trigger :67108864 bytes (65536 KB)
<br>>
<br>> 2009-12-04 12:55:23,908 WARN namenode.SecondaryNameNode - Checkpoint
<br>> done. New Image Size: 1056
<br>>
<br>>
<br>>
<br>> HADOOP LOG
<br>>
<br>> 2009-12-04 12:54:20,708 WARN hdfs.DFSClient - DataStreamer Exception:
<br>> java.io.IOException: Unable to create new block.
<br>>
<br>> at
<br>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(D
<br>> FSClient.java:2722)
<br>>
<br>> at
<br>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.j
<br>> ava:1996)
<br>>
<br>> at
<br>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCli
<br>> ent.java:2183)
<br>>
<br>>
<br>>
<br>> 2009-12-04 12:54:20,709 WARN hdfs.DFSClient - Error Recovery for block
<br>> blk_5506837520665828594_1002 bad datanode[0] nodes == null
<br>>
<br>> 2009-12-04 12:54:20,709 WARN hdfs.DFSClient - Could not get block
<br>> locations. Source file "/user/nutch/blub/hadoop-nutch-tasktracker.pid" -
<br>> Aborting...
<br>>
<br>>
<br>>
<br>> DATA NODE LOG on the slave
<br>>
<br>> 2009-12-04 12:49:49,433 INFO http.HttpServer - Version Jetty/5.1.4
<br>>
<br>> 2009-12-04 12:49:49,438 INFO util.Credential - Checking Resource
<br>> aliases
<br>>
<br>> 2009-12-04 12:49:50,288 INFO util.Container - Started
<br>> org.mortbay.jetty.servlet.WebApplicationHandler@e45b5e
<br>>
<br>> 2009-12-04 12:49:50,357 INFO util.Container - Started
<br>> WebApplicationContext[/static,/static]
<br>>
<br>> 2009-12-04 12:49:50,555 INFO util.Container - Started
<br>> org.mortbay.jetty.servlet.WebApplicationHandler@2016b0
<br>>
<br>> 2009-12-04 12:49:50,555 INFO util.Container - Started
<br>> WebApplicationContext[/logs,/logs]
<br>>
<br>> 2009-12-04 12:49:50,816 INFO util.Container - Started
<br>> org.mortbay.jetty.servlet.WebApplicationHandler@118278a
<br>>
<br>> 2009-12-04 12:49:50,820 INFO util.Container - Started
<br>> WebApplicationContext[/,/]
<br>>
<br>> 2009-12-04 12:49:50,849 INFO http.SocketListener - Started
<br>> SocketListener on 0.0.0.0:50075
<br>>
<br>> 2009-12-04 12:49:50,849 INFO util.Container - Started
<br>> org.mortbay.jetty.Server@b02928
<br>>
<br>>
<br>>
<br>> HADOOP SITE XML
<br>>
<br>> <property>
<br>>
<br>> <name>fs.default.name</name>
<br>>
<br>> <value>hdfs://(yes here is the right ip):9000</value>
<br>>
<br>> <description>
<br>>
<br>> The name of the default file system. Either the literal string
<br>>
<br>> "local" or a host:port for NDFS.
<br>>
<br>> </description>
<br>>
<br>> </property>
<br>>
<br>>
<br>>
<br>> <!-- Gibt an wo der JobTracker (koordiniert die (MapReduce-)Auftraege)
<br>> zu finden ist. -->
<br>>
<br>> <property>
<br>>
<br>> <name>mapred.job.tracker</name>
<br>>
<br>> <value>hdfs://(here to):9001</value>
<br>>
<br>> <description>
<br>>
<br>> The host and port that the MapReduce job tracker runs at. If
<br>>
<br>> "local", then jobs are run in-process as a single map and
<br>>
<br>> reduce task.
<br>>
<br>> </description>
<br>>
<br>> </property>
<br>>
<br>>
<br>>
<br>> <!-- Gibt an wieviele MapJobs gleichzeitig laufen duerfen-->
<br>>
<br>> <property>
<br>>
<br>> <name>mapred.tasktracker.map.tasks.maximum</name>
<br>>
<br>> <value>2</value>
<br>>
<br>> <description>
<br>>
<br>> define mapred.map tasks to be number of slave hosts
<br>>
<br>> </description>
<br>>
<br>> </property>
<br>>
<br>>
<br>>
<br>> <!-- Gibt an wieviele ReduceJobs gleichzeitig laufen duerfen-->
<br>>
<br>> <property>
<br>>
<br>> <name>mapred.tasktracker.reduce.tasks.maximum</name>
<br>>
<br>> <value>2</value>
<br>>
<br>> <description>
<br>>
<br>> define mapred.reduce tasks to be number of slave hosts
<br>>
<br>> </description>
<br>>
<br>> </property>
<br>>
<br>>
<br>>
<br>> <property>
<br>>
<br>> <name>mapred.child.java.opts</name>
<br>>
<br>> <value>-Xmx1500m</value>
<br>>
<br>> </property>
<br>>
<br>>
<br>>
<br>> <property>
<br>>
<br>> <name>mapred.jobtracker.restart.recover</name>
<br>>
<br>> <value>true</value>
<br>>
<br>> </property>
<br>>
<br>>
<br>>
<br>> <!-- Die naechsten Einstellungen geben an wo das HadoopFS seine Datein
<br>> auf der Festplatte jeder Instanz speichert. -->
<br>>
<br>> <property>
<br>>
<br>> <name>dfs.name.dir</name>
<br>>
<br>> <value>/nutch/filesystem/name</value>
<br>>
<br>> </property>
<br>>
<br>>
<br>>
<br>> <property>
<br>>
<br>> <name>dfs.data.dir</name>
<br>>
<br>> <value>/nutch/filesystem/data</value>
<br>>
<br>> </property>
<br>>
<br>>
<br>>
<br>> <property>
<br>>
<br>> <name>mapred.system.dir</name>
<br>>
<br>> <value>/nutch/filesystem/mapreduce/system</value>
<br>>
<br>> </property>
<br>>
<br>>
<br>>
<br>> <property>
<br>>
<br>> <name>mapred.local.dir</name>
<br>>
<br>> <value>/nutch/filesystem/mapreduce/local</value>
<br>>
<br>> </property>
<br>>
<br>>
<br>>
<br>> <!-- Gibt an wieviele Replikate einer Datei im Dateisystem vorhanden
<br>> sein muessen damit sie erreichbar ist. Am Anfang 1 -->
<br>>
<br>> <property>
<br>>
<br>> <name>dfs.replication</name>
<br>>
<br>> <value>2</value>
<br>>
<br>> </property>
<br>>
<br>>
<br>>
<br>> I hope someone can help me.
<br>>
<br>>
<br>>
<br>> Thanks
<br>>
<br>>
<br>>
<br>> Tom
<br>>
<br>>
</div><br><br>--
<br>-MilleBii-
<br><p>From forum: <a href="http://old.nabble.com/Nutch---User-f375.html" embed="fixTarget[375]" target="_top" >Nutch - User</a></p>tag:old.nabble.com,2006:post-26642876Re: Can nutch pause, stop and start where it left off?2009-12-04T05:56:53Z2009-12-04T05:56:53ZJesse Hiresuse the -topN flag to only grab a small number of URLs.
<br>Also I believe there is also a setting you can put in nutch-site.xml that
<br>can be used to slow down how many URLs you grab over time.
<br><br>Jesse
<br><br>int GetRandomNumber()
<br>{
<br> return 4; // Chosen by fair roll of dice
<br> // Guaranteed to be random
<br>} // xkcd.com
<br><br><br><br>On Fri, Dec 4, 2009 at 4:10 AM, Mr Hadoop <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26642876&i=0" target="_top" rel="nofollow">mrhadoop@...</a>> wrote:
<br><br>> I am just staring to learn nutch. One question I wanted to know is that
<br>> can
<br>> nutch pause, stop and start indexing a site on a incremental daily basis?
<br>> My concern with nutch is that nutch behaving like a hog and crawling
<br>> everything with huge bandwidth consumption and pissing off the many site
<br>> owners.
<br>>
<br>> Can some experts shed some light in this?
<br>>
<br><p>From forum: <a href="http://old.nabble.com/Nutch---User-f375.html" embed="fixTarget[375]" target="_top" >Nutch - User</a></p>