Nabble - Nutch - Dev

Hudson build is back to normal: Nutch-trunk #986

2009-11-07T21:41:30Z

See <http://hudson.zones.apache.org/hudson/job/Nutch-trunk/986/>

Build failed in Hudson: Nutch-trunk #985

2009-11-06T20:03:16Z

See <http://hudson.zones.apache.org/hudson/job/Nutch-trunk/985/>

------------------------------------------
A timer trigger started this job
Building remotely on lucene.zones.apache.org (Solaris 10)
Checking out http://svn.apache.org/repos/asf/lucene/nutch/trunk
ERROR: Failed to check out http://svn.apache.org/repos/asf/lucene/nutch/trunk
org.tmatesoft.svn.core.SVNException: svn: timed out waiting for server
svn: OPTIONS request failed on '/repos/asf/lucene/nutch/trunk'
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:103)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:87)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:616)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:273)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:261)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:516)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:98)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1001)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getLatestRevision(DAVRepository.java:178)
at org.tmatesoft.svn.core.wc.SVNBasicClient.getRevisionNumber(SVNBasicClient.java:482)
at org.tmatesoft.svn.core.wc.SVNBasicClient.getLocations(SVNBasicClient.java:851)
at org.tmatesoft.svn.core.wc.SVNBasicClient.createRepository(SVNBasicClient.java:534)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:893)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:791)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:617)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:543)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2052)
at hudson.remoting.UserRequest.perform(UserRequest.java:69)
at hudson.remoting.UserRequest.perform(UserRequest.java:23)
at hudson.remoting.Request$2.run(Request.java:200)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:519)
at org.tmatesoft.svn.core.internal.util.SVNSocketConnection.run(SVNSocketConnection.java:57)
... 1 more
Archiving artifacts
Publishing Javadoc
Recording test results

New attachment added to page Presentations on Nutch Wiki

2009-11-06T10:42:08Z

Dear Wiki user,

You have subscribed to a wiki page "Presentations" for change notification. An attachment has been added to that page by AndrzejBialecki. Following detailed information is available:

Attachment name: apachecon09.pdf
Attachment size: 990738
Attachment link: http://wiki.apache.org/nutch/Presentations?action=AttachFile&do=get&target=apachecon09.pdf
Page link: http://wiki.apache.org/nutch/Presentations

[Nutch Wiki] Update of "Presentations" by AndrzejBialecki

2009-11-06T09:33:56Z

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Nutch Wiki" for change notification.

The "Presentations" page has been changed by AndrzejBialecki.
http://wiki.apache.org/nutch/Presentations?action=diff&rev1=10&rev2=11

--------------------------------------------------

+ Recent presentations:
+
+ * [[attachment:apachecon09.pdf|apachecon09.pdf]]: "Nutch, web-scale search engine toolkit", Andrzej Bialecki, 5 Nov 2009, Apache``Con`` 2009, Oakland.
+
+
Here are the slides from presentations about Nutch given by Doug Cutting:

* [[attachment:serverside1.pdf|serverside1.pdf]]: "Intranet Search with Nutch", 6 May 2004, The``Server``Side Java Symposium, Las Vegas.

New attachment added to page Presentations on Nutch Wiki

2009-11-06T09:29:18Z

Dear Wiki user,

You have subscribed to a wiki page "Presentations" for change notification. An attachment has been added to that page by AndrzejBialecki. Following detailed information is available:

Attachment name: apachecon09.pdf
Attachment size: 1313737
Attachment link: http://wiki.apache.org/nutch/Presentations?action=AttachFile&do=get&target=apachecon09.pdf
Page link: http://wiki.apache.org/nutch/Presentations

[jira] Created: (NUTCH-763) Separate configuration files from resources to be included in the job file

2009-11-05T10:34:32Z

Separate configuration files from resources to be included in the job file
--------------------------------------------------------------------------

Key: NUTCH-763
URL: https://issues.apache.org/jira/browse/NUTCH-763
Project: Nutch
Issue Type: Wish
Reporter: Julien Nioche
Priority: Minor
Fix For: 1.1

One of the things I found confusing when I was learning Nutch was the fact that the conf/ directory contains at the same time :
- configuration files for Hadoop / Nutch which are put in the jar files but not used there
- resource files (e.g. filtering rules) which MUST be up to date in the job file

I would separate the conf/ directory from say a resources/ directory which would contain the rule files and other things to put in the job file. Unless I am mistaken none of the configuration files need to be in the job file. I know it is a very minor point, but that would probably simplify things and make it easier for beginners to understand what has to be modified where.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Re: [Nutch Wiki] Update of "ApacheConUs2009MeetUp" by KenKrugler

2009-11-05T04:08:40Z

Bartosz Gadzimski wrote:
> Hello,
>
> Are there will be any materials after the meeting? Wiki pages, slides,
> video, podcasts? Would be grate!

I will be adding some stuff later - no slides, as Ken said it was very
informal. Though the slides that I present during my Nutch talk have a
section about the roadmap, and I will be adding these to the wiki, too.

--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com

Re: [Nutch Wiki] Update of "ApacheConUs2009MeetUp" by KenKrugler

2009-11-05T02:41:06Z

Hi Bartosz,

I've updated the wiki, and others who attended might add/edit as
necessary.

No video/podcast - it wasn't so high tech as that, just three of us in
a spare room with Thorsten on Skype.

We're still waiting for some input from the Heritrix team, I think,
before moving forward.

-- Ken

On Nov 5, 2009, at 12:01am, Bartosz Gadzimski wrote:

> Hello,
>
> Are there will be any materials after the meeting? Wiki pages,
> slides, video, podcasts? Would be grate!
>
> Thanks,
> Bartosz
>
> Apache Wiki pisze:
>> Dear Wiki user,
>>
>> You have subscribed to a wiki page or wiki category on "Nutch Wiki"
>> for change notification.
>>
>> The "ApacheConUs2009MeetUp" page has been changed by KenKrugler.
>> http://wiki.apache.org/nutch/ApacheConUs2009MeetUp?action=diff&rev1=5&rev2=6
>>
>> --------------------------------------------------
>>
>> - We were planning to have a "Web Crawler Developer" !MeetUp at
>> this year's [[http://www.us.apachecon.com/c/acus2009/|ApacheCon
>> US]] in Oakland.
>> + We had a "Web Crawler Developer" !MeetUp at this year's [[http://www.us.apachecon.com/c/acus2009/
>> |ApacheCon US]] in Oakland.
>> - Unfortunately the only time slot where people would be around
>> was Thursday night, which wound up conflicting with the Hadoop !
>> MeetUp.
>> + It wound up being an !UnMeetUp (!MeetDown?) on Wednesday,
>> November 4th from 11am - 1pm. - So we're going to have an !
>> UnMeetUp (!MeetDown?) on Wednesday, November 4th from 11am - 1pm.
>> Location is TBD, hopefully we can get some space at the event but
>> might be a lunch meeting :)
>> + == Attendees ==
>> + + * Andrzej Bialeki - Apache Nutch
>> + * Thorsten xxx - Apache Droids
>> + * Michael Stack - Formerly with Heritrix, now HBase
>> + * Ken Krugler - Bixo
>> + + == Topics ==
>> + + === Roadmaps ===
>> + + Nutch - become more component based.
>> + Droids - get more people involved.
>> + + === Sharable Components ===
>> + + * robots.txt parsing
>> + * URL normalization
>> + * URL filtering
>> + * Page cleansing
>> + * General purpose
>> + * Specialized
>> + * Sub-page parsing (portlets)
>> + * AJAX-ish page interactions
>> + * Document parsing (via Tika)
>> + * HttpClient (configuration)
>> + * Text similarity
>> + * Mime/charset/language detection
>> + + === Tika ===
>> + + * Needs help to become really usable
>> + * Would benefit from large test corpus
>> + * Could do comparison with Nutch parser
>> + * Needs option for direct DOM querying (screen scraping tasks)
>> + * Handles mime & charset detection now (some issues)
>> + * Could be extended to include language detection (wrap other
>> impl)
>> + + === URL Normalization ===
>> + + * Includes both domain (www.x.com == x.com), path, and query
>> portions of URL
>> + * Often site-specific rules
>> + * Option to derive rules using URLs to similar documents.
>> + + === AJAX-ish Page Interaction ===
>> + + * Not applicable for broad/general crawling
>> + * Can be very important for specific web sites
>> + * Use Selenium or headless Mozilla
>> + + === Component API Issues ===
>> + + * Want to avoid using an API that's tied too closely to any
>> implementation.
>> + * One option is to have simple (e.g. URL param) API that takes
>> meta-data.
>> + * Similar to Tika passing in of meta-data.
>> + + === Hosting Options ===
>> + + * As part of Nutch - but easy to get lost in Nutch codebase,
>> and can be associated too closely with Nutch.
>> + * As part of Droids - but Droids is both a framework (queue-
>> based) and set of components.
>> + * New sub-project under Lucene TLP - but overhead to set up/
>> maintain, and then confusion between it and Droids.
>> + * Google code - seems like a good short-term solution, to judge
>> level of interest and help shake out issues.
>> + + == Next Steps ==
>> + + * Get input from Gordon re Heritrix. Stack to follow up with
>> him. Ideally he'd add his comments to this page.
>> + * Get input from Thorsten on Google code option. If OK as
>> starting point, then Andrzej to set up.
>> + * Make decision about build system (and then move on to code
>> formatting debate :))
>> + * I'm going to propose ant + maven ant tasks for dependency
>> management. I'm using this with Bixo, and so far it's been pretty
>> good.
>> + * Start contributing code
>> + * Ken will put in robots.txt parser.
>> + + == Original Discussion Topic List ==
>> Below are some potential topics for discussion - feel free to
>> add/comment.
>>
>>
>

--------------------------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
e l a s t i c w e b m i n i n g

Re: [Nutch Wiki] Update of "ApacheConUs2009MeetUp" by KenKrugler

2009-11-05T00:01:40Z

Hello,

Are there will be any materials after the meeting? Wiki pages, slides,
video, podcasts? Would be grate!

Thanks,
Bartosz

Apache Wiki pisze:

> Dear Wiki user,
>
> You have subscribed to a wiki page or wiki category on "Nutch Wiki" for change notification.
>
> The "ApacheConUs2009MeetUp" page has been changed by KenKrugler.
> http://wiki.apache.org/nutch/ApacheConUs2009MeetUp?action=diff&rev1=5&rev2=6
>
> --------------------------------------------------
>
> - We were planning to have a "Web Crawler Developer" !MeetUp at this year's [[http://www.us.apachecon.com/c/acus2009/|ApacheCon US]] in Oakland.
> + We had a "Web Crawler Developer" !MeetUp at this year's [[http://www.us.apachecon.com/c/acus2009/|ApacheCon US]] in Oakland.
>
> - Unfortunately the only time slot where people would be around was Thursday night, which wound up conflicting with the Hadoop !MeetUp.
> + It wound up being an !UnMeetUp (!MeetDown?) on Wednesday, November 4th from 11am - 1pm.
>
> - So we're going to have an !UnMeetUp (!MeetDown?) on Wednesday, November 4th from 11am - 1pm. Location is TBD, hopefully we can get some space at the event but might be a lunch meeting :)
> + == Attendees ==
> +
> + * Andrzej Bialeki - Apache Nutch
> + * Thorsten xxx - Apache Droids
> + * Michael Stack - Formerly with Heritrix, now HBase
> + * Ken Krugler - Bixo
> +
> + == Topics ==
> +
> + === Roadmaps ===
> +
> + Nutch - become more component based.
> + Droids - get more people involved.
> +
> + === Sharable Components ===
> +
> + * robots.txt parsing
> + * URL normalization
> + * URL filtering
> + * Page cleansing
> + * General purpose
> + * Specialized
> + * Sub-page parsing (portlets)
> + * AJAX-ish page interactions
> + * Document parsing (via Tika)
> + * HttpClient (configuration)
> + * Text similarity
> + * Mime/charset/language detection
> +
> + === Tika ===
> +
> + * Needs help to become really usable
> + * Would benefit from large test corpus
> + * Could do comparison with Nutch parser
> + * Needs option for direct DOM querying (screen scraping tasks)
> + * Handles mime & charset detection now (some issues)
> + * Could be extended to include language detection (wrap other impl)
> +
> + === URL Normalization ===
> +
> + * Includes both domain (www.x.com == x.com), path, and query portions of URL
> + * Often site-specific rules
> + * Option to derive rules using URLs to similar documents.
> +
> + === AJAX-ish Page Interaction ===
> +
> + * Not applicable for broad/general crawling
> + * Can be very important for specific web sites
> + * Use Selenium or headless Mozilla
> +
> + === Component API Issues ===
> +
> + * Want to avoid using an API that's tied too closely to any implementation.
> + * One option is to have simple (e.g. URL param) API that takes meta-data.
> + * Similar to Tika passing in of meta-data.
> +
> + === Hosting Options ===
> +
> + * As part of Nutch - but easy to get lost in Nutch codebase, and can be associated too closely with Nutch.
> + * As part of Droids - but Droids is both a framework (queue-based) and set of components.
> + * New sub-project under Lucene TLP - but overhead to set up/maintain, and then confusion between it and Droids.
> + * Google code - seems like a good short-term solution, to judge level of interest and help shake out issues.
> +
> + == Next Steps ==
> +
> + * Get input from Gordon re Heritrix. Stack to follow up with him. Ideally he'd add his comments to this page.
> + * Get input from Thorsten on Google code option. If OK as starting point, then Andrzej to set up.
> + * Make decision about build system (and then move on to code formatting debate :))
> + * I'm going to propose ant + maven ant tasks for dependency management. I'm using this with Bixo, and so far it's been pretty good.
> + * Start contributing code
> + * Ken will put in robots.txt parser.
> +
> + == Original Discussion Topic List ==
>
> Below are some potential topics for discussion - feel free to add/comment.
>
>
>

MergeSegments - map reduce thread death

2009-11-04T17:29:14Z

Hi there,

seems i have some serious problems with hadoop during map-reduce for
MergeSegments.

i am out of ideas on this. Any suggestions will be quite welcome.

Here is my set up:

RAM: 4G
JVM HEAP: 2G
mapred.child.java.opts = 1024M
hadoop-0.19.1-core.jar
nutch-1.0
Xen VPS.

After running a recrawl a few times; i end up with one segment that is
relatively larger compared to the new ones last generated. here is my
segments structure when things blow up after a (5th) recrawl;

segment1 = 674Megs (after several recrawls)
segment2 = 580k (last recrawl)
segment3 = 568k (last recrawl)
segment4 = 584k (last recrawl)
..
segment8 = 560k (last recrawl)

when i run mergeSegments everything goes well until we get up to 90% of
the map-reduce and we get a thread death; here is a stack trace

2009-11-05 10:54:16,874 INFO [org.apache.hadoop.mapred.LocalJobRunner]
reduce > reduce
2009-11-05 10:54:29,794 INFO [org.apache.hadoop.mapred.LocalJobRunner]
reduce > reduce
2009-11-05 10:54:55,194 INFO [org.apache.hadoop.mapred.LocalJobRunner]
reduce > reduce
2009-11-05 10:57:25,844 WARN [org.apache.hadoop.mapred.LocalJobRunner]
job_local_0001
java.lang.ThreadDeath
at java.lang.Thread.stop(Thread.java:715)
at
org.apache.hadoop.mapred.LocalJobRunner.killJob(LocalJobRunner.java:310)
at
org.apache.hadoop.mapred.JobClient$NetworkedJob.killJob(JobClient.java:315)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1239)
at
org.apache.nutch.segment.SegmentMerger.merge(SegmentMerger.java:620)
at
org.apache.nutch.segment.SegmentMerger.main(SegmentMerger.java:665)

any suggestions please!!!!

thanks.

[Nutch Wiki] Update of "ApacheConUs2009MeetUp" by AndrzejBialecki

2009-11-04T14:12:11Z

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Nutch Wiki" for change notification.

The "ApacheConUs2009MeetUp" page has been changed by AndrzejBialecki.
http://wiki.apache.org/nutch/ApacheConUs2009MeetUp?action=diff&rev1=8&rev2=9

--------------------------------------------------

== Attendees ==

- * Andrzej Bialeki - Apache Nutch
+ * Andrzej Bialecki - Apache Nutch
* Thorsten Sherler - Apache Droids
* Michael Stack - Formerly with Heritrix, now HBase
* Ken Krugler - Bixo

[Nutch Wiki] Update of "ApacheConUs2009MeetUp" by KenKrugler

2009-11-04T14:04:41Z

[Nutch Wiki] Update of "ApacheConUs2009MeetUp" by KenKrugler

2009-11-04T14:03:23Z

[Nutch Wiki] Update of "ApacheConUs2009MeetUp" by KenKrugler

2009-11-04T14:02:35Z

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Nutch Wiki" for change notification.

The "ApacheConUs2009MeetUp" page has been changed by KenKrugler.
http://wiki.apache.org/nutch/ApacheConUs2009MeetUp?action=diff&rev1=5&rev2=6

--------------------------------------------------

- We were planning to have a "Web Crawler Developer" !MeetUp at this year's [[http://www.us.apachecon.com/c/acus2009/|ApacheCon US]] in Oakland.
+ We had a "Web Crawler Developer" !MeetUp at this year's [[http://www.us.apachecon.com/c/acus2009/|ApacheCon US]] in Oakland.

- Unfortunately the only time slot where people would be around was Thursday night, which wound up conflicting with the Hadoop !MeetUp.
+ It wound up being an !UnMeetUp (!MeetDown?) on Wednesday, November 4th from 11am - 1pm.

- So we're going to have an !UnMeetUp (!MeetDown?) on Wednesday, November 4th from 11am - 1pm. Location is TBD, hopefully we can get some space at the event but might be a lunch meeting :)
+ == Attendees ==
+
+ * Andrzej Bialeki - Apache Nutch
+ * Thorsten xxx - Apache Droids
+ * Michael Stack - Formerly with Heritrix, now HBase
+ * Ken Krugler - Bixo
+
+ == Topics ==
+
+ === Roadmaps ===
+
+ Nutch - become more component based.
+ Droids - get more people involved.
+
+ === Sharable Components ===
+
+ * robots.txt parsing
+ * URL normalization
+ * URL filtering
+ * Page cleansing
+ * General purpose
+ * Specialized
+ * Sub-page parsing (portlets)
+ * AJAX-ish page interactions
+ * Document parsing (via Tika)
+ * HttpClient (configuration)
+ * Text similarity
+ * Mime/charset/language detection
+
+ === Tika ===
+
+ * Needs help to become really usable
+ * Would benefit from large test corpus
+ * Could do comparison with Nutch parser
+ * Needs option for direct DOM querying (screen scraping tasks)
+ * Handles mime & charset detection now (some issues)
+ * Could be extended to include language detection (wrap other impl)
+
+ === URL Normalization ===
+
+ * Includes both domain (www.x.com == x.com), path, and query portions of URL
+ * Often site-specific rules
+ * Option to derive rules using URLs to similar documents.
+
+ === AJAX-ish Page Interaction ===
+
+ * Not applicable for broad/general crawling
+ * Can be very important for specific web sites
+ * Use Selenium or headless Mozilla
+
+ === Component API Issues ===
+
+ * Want to avoid using an API that's tied too closely to any implementation.
+ * One option is to have simple (e.g. URL param) API that takes meta-data.
+ * Similar to Tika passing in of meta-data.
+
+ === Hosting Options ===
+
+ * As part of Nutch - but easy to get lost in Nutch codebase, and can be associated too closely with Nutch.
+ * As part of Droids - but Droids is both a framework (queue-based) and set of components.
+ * New sub-project under Lucene TLP - but overhead to set up/maintain, and then confusion between it and Droids.
+ * Google code - seems like a good short-term solution, to judge level of interest and help shake out issues.
+
+ == Next Steps ==
+
+ * Get input from Gordon re Heritrix. Stack to follow up with him. Ideally he'd add his comments to this page.
+ * Get input from Thorsten on Google code option. If OK as starting point, then Andrzej to set up.
+ * Make decision about build system (and then move on to code formatting debate :))
+ * I'm going to propose ant + maven ant tasks for dependency management. I'm using this with Bixo, and so far it's been pretty good.
+ * Start contributing code
+ * Ken will put in robots.txt parser.
+
+ == Original Discussion Topic List ==

Below are some potential topics for discussion - feel free to add/comment.

Re: Free live video streaming of ApacheCon US 2009

2009-11-04T06:55:16Z

Thanks a lot.

This will be very helpful to me.

As I am not able to attend.

On Wed, Nov 4, 2009 at 8:25 AM, Michael McCandless <lucene@...> wrote:

Team,

For those Lucene fanatics not in Oakland this week for ApacheCon US,
don't miss the FREE live video streaming, starting today:

http://streaming.linux-magazin.de/en/program-apachecon-us-2009.htm

Note that there are many talks available, covering Apache Hadoop,
Apache HTTPD, Lucene, as well as the Apache Pioneer's Panel and
keynote presentations.

Lucene's track is this Friday (NOTE these times are UTC -- use
http://www.timeanddate.com to map to your time zone):

17:00 Implementing an Information Retrieval Framework for an
Organizational Repository, Sithu D Sudarsan

18:00 Apache Mahout - Going from raw data to information
Isabel Drost

19:15 MIME Magic with Apache Tika
Jukka Zitting

20:15 Keynote: How Open Source Developers Can (Still!) Save The World
Brian Behlendorf

22:00 Building Intelligent Search Applications with the Lucene
Ecosystem, Ted Dunning

23:00 Realtime Search
Jason Rutherglen

Happy viewing,

Mike

--
"Good Enough" is not good enough.
To give anything less than your best is to sacrifice the gift.
Quality First. Measure Twice. Cut Once.

Free live video streaming of ApacheCon US 2009

2009-11-04T05:25:25Z

Team,

For those Lucene fanatics not in Oakland this week for ApacheCon US,
don't miss the FREE live video streaming, starting today:

http://streaming.linux-magazin.de/en/program-apachecon-us-2009.htm

Note that there are many talks available, covering Apache Hadoop,
Apache HTTPD, Lucene, as well as the Apache Pioneer's Panel and
keynote presentations.

Lucene's track is this Friday (NOTE these times are UTC -- use
http://www.timeanddate.com to map to your time zone):

17:00 Implementing an Information Retrieval Framework for an
Organizational Repository, Sithu D Sudarsan

18:00 Apache Mahout - Going from raw data to information
Isabel Drost

19:15 MIME Magic with Apache Tika
Jukka Zitting

20:15 Keynote: How Open Source Developers Can (Still!) Save The World
Brian Behlendorf

22:00 Building Intelligent Search Applications with the Lucene
Ecosystem, Ted Dunning

23:00 Realtime Search
Jason Rutherglen

Happy viewing,

Mike

[jira] Updated: (NUTCH-762) Alternative Generator which can generate several segments in one parse of the crawlDB

2009-11-03T07:05:59Z

[ https://issues.apache.org/jira/browse/NUTCH-762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Julien Nioche updated NUTCH-762:
--------------------------------

Attachment: NUTCH-762-MultiGenerator.patch

Patch for the MultiGenerator

> Alternative Generator which can generate several segments in one parse of the crawlDB
> -------------------------------------------------------------------------------------
>
> Key: NUTCH-762
> URL: https://issues.apache.org/jira/browse/NUTCH-762
> Project: Nutch
> Issue Type: New Feature
> Components: generator
> Affects Versions: 1.0.0
> Reporter: Julien Nioche
> Attachments: NUTCH-762-MultiGenerator.patch
>
>
> When using Nutch on a large scale (e.g. billions of URLs), the operations related to the crawlDB (generate - update) tend to take the biggest part of the time. One solution is to limit such operations to a minimum by generating several fetchlists in one parse of the crawlDB then update the Db only once on several segments. The existing Generator allows several successive runs by generating a copy of the crawlDB and marking the URLs to be fetched. In practice this approach does not work well as we need to read the whole crawlDB as many time as we generate a segment.
> The patch attached contains an implementation of a MultiGenerator which can generate several fetchlists by reading the crawlDB only once. The MultiGenerator differs from the Generator in other aspects:
> * can filter the URLs by score
> * normalisation is optional
> * IP resolution is done ONLY on the entries which have been selected for fetching (during the partitioning). Running the IP resolution on the whole crawlDb is too slow to be usable on a large scale
> * can max the number of URLs per host or domain (but not by IP)
> * can choose to partition by host, domain or IP
> Typically the same unit (e.g. domain) would be used for maxing the URLs and for partitioning; however as we can't count the max number of URLs by IP another unit must be chosen while partitioning by IP.
> We found that using a filter on the score can dramatically improve the performance as this reduces the amount of data being sent to the reducers.
> The MultiGenerator is called via : nutch org.apache.nutch.crawl.MultiGenerator ...
> with the following options :
> MultiGenerator <crawldb> <segments_dir> [-force] [-topN N] [-numFetchers numFetchers] [-adddays numDays] [-noFilter] [-noNorm] [-maxNumSegments num]
> where most parameters are similar to the default Generator - apart from :
> -noNorm (explicit)
> -topN : max number of URLs per segment
> -maxNumSegments : the actual number of segments generated could be less than the max value select e.g. not enough URLs are available for fetching and fit in less segments
> Please give it a try and less me know what you think of it
> Julien Nioche
> http://www.digitalpebble.com
>

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (NUTCH-762) Alternative Generator which can generate several segments in one parse of the crawlDB

2009-11-03T07:03:59Z

Alternative Generator which can generate several segments in one parse of the crawlDB
-------------------------------------------------------------------------------------

Key: NUTCH-762
URL: https://issues.apache.org/jira/browse/NUTCH-762
Project: Nutch
Issue Type: New Feature
Components: generator
Affects Versions: 1.0.0
Reporter: Julien Nioche

When using Nutch on a large scale (e.g. billions of URLs), the operations related to the crawlDB (generate - update) tend to take the biggest part of the time. One solution is to limit such operations to a minimum by generating several fetchlists in one parse of the crawlDB then update the Db only once on several segments. The existing Generator allows several successive runs by generating a copy of the crawlDB and marking the URLs to be fetched. In practice this approach does not work well as we need to read the whole crawlDB as many time as we generate a segment.

The patch attached contains an implementation of a MultiGenerator which can generate several fetchlists by reading the crawlDB only once. The MultiGenerator differs from the Generator in other aspects:
* can filter the URLs by score
* normalisation is optional
* IP resolution is done ONLY on the entries which have been selected for fetching (during the partitioning). Running the IP resolution on the whole crawlDb is too slow to be usable on a large scale
* can max the number of URLs per host or domain (but not by IP)
* can choose to partition by host, domain or IP

Typically the same unit (e.g. domain) would be used for maxing the URLs and for partitioning; however as we can't count the max number of URLs by IP another unit must be chosen while partitioning by IP.
We found that using a filter on the score can dramatically improve the performance as this reduces the amount of data being sent to the reducers.

The MultiGenerator is called via : nutch org.apache.nutch.crawl.MultiGenerator ...
with the following options :
MultiGenerator <crawldb> <segments_dir> [-force] [-topN N] [-numFetchers numFetchers] [-adddays numDays] [-noFilter] [-noNorm] [-maxNumSegments num]

where most parameters are similar to the default Generator - apart from :
-noNorm (explicit)
-topN : max number of URLs per segment
-maxNumSegments : the actual number of segments generated could be less than the max value select e.g. not enough URLs are available for fetching and fit in less segments

Please give it a try and less me know what you think of it

Julien Nioche
http://www.digitalpebble.com

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (NUTCH-761) Avoid cloningCrawlDatum in CrawlDbReducer

2009-11-03T06:41:59Z

[ https://issues.apache.org/jira/browse/NUTCH-761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Julien Nioche updated NUTCH-761:
--------------------------------

Attachment: optiCrawlReducer.patch

> Avoid cloningCrawlDatum in CrawlDbReducer
> ------------------------------------------
>
> Key: NUTCH-761
> URL: https://issues.apache.org/jira/browse/NUTCH-761
> Project: Nutch
> Issue Type: Improvement
> Reporter: Julien Nioche
> Priority: Minor
> Attachments: optiCrawlReducer.patch
>
>
> In the huge majority of cases the CrawlDbReducer gets unique CrawlData in its reduce phase and these will be the entries coming from the crawlDB and not present in the segments.
> The patch attached optimizes the reduce step by avoid an unnecessary cloning of the CrawlDatum fields when there is only one CrawlDatum in the values. This has more impact has the crawlDB gets larger, we noticed an improvement of around 25-30% in the time spent in the reduce phase.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (NUTCH-761) Avoid cloningCrawlDatum in CrawlDbReducer

2009-11-03T06:39:59Z

Avoid cloningCrawlDatum in CrawlDbReducer
------------------------------------------

Key: NUTCH-761
URL: https://issues.apache.org/jira/browse/NUTCH-761
Project: Nutch
Issue Type: Improvement
Reporter: Julien Nioche
Priority: Minor

In the huge majority of cases the CrawlDbReducer gets unique CrawlData in its reduce phase and these will be the entries coming from the crawlDB and not present in the segments.
The patch attached optimizes the reduce step by avoid an unnecessary cloning of the CrawlDatum fields when there is only one CrawlDatum in the values. This has more impact has the crawlDB gets larger, we noticed an improvement of around 25-30% in the time spent in the reduce phase.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (NUTCH-585) [PARSE-HTML plugin] Block certain parts of HTML code from being indexed

2009-10-29T14:21:59Z

[ https://issues.apache.org/jira/browse/NUTCH-585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771625#action_12771625 ]

David Stuart commented on NUTCH-585:
------------------------------------

Hi Andrea,

I hope your week of demo's went well. I to would be interested in this code as I would like to look at extending to it be slightly more generic allowing for regular expression matches or an xpath like model (the plan is still formulating). From the web crawler view it would be a hard one to get right but we have about 26 sites that are will know to us that we wish to crawl and have common blocks that we wish to remove which a configurable version of your code may achieve.

Look forward to see your patch

Regards,

David Stuart

> [PARSE-HTML plugin] Block certain parts of HTML code from being indexed
> -----------------------------------------------------------------------
>
> Key: NUTCH-585
> URL: https://issues.apache.org/jira/browse/NUTCH-585
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Environment: All operating systems
> Reporter: Andrea Spinelli
> Priority: Minor
>
> We are using nutch to index our own web sites; we would like not to index certain parts of our pages, because we know they are not relevant (for instance, there are several links to change the background color) and generate spurious matches.
> We have modified the plugin so that it ignores HTML code between certain HTML comments, like
> 
> ... ignored part ...
> 
> We feel this might be useful to someone else, maybe factorizing the comment strings as constants in the configuration files (say parser.html.ignore.start and parser.html.ignore.stop in nutch-site.xml).
> We are almost ready to contribute our code snippet. Looking forward for any expression of interest - or for an explanation why waht we are doing is plain wrong!

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[Nutch Wiki] Update of "DownloadingNutch" by SteveKearns

2009-10-27T13:39:43Z

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Nutch Wiki" for change notification.

The "DownloadingNutch" page has been changed by SteveKearns.
http://wiki.apache.org/nutch/DownloadingNutch?action=diff&rev1=5&rev2=6

--------------------------------------------------

You have two choices in how to get Nutch:
- 1. You can download a release from http://lucene.apache.org/nutch/release/. This will give you a relatively stable release. At the moment the latest release is 0.9.
+ 1. You can download a release from http://lucene.apache.org/nutch/release/. This will give you a relatively stable release. At the moment the latest release is 1.0.
- 2. Or, you can check out the latest source code from subversion and build it with Ant. This gets you closer to the bleeding edge of development. The 0.9 should be relatively stable but the trunk (from which the [[http://lucene.apache.org/nutch/nightly.html|nightly builds]] are build) is under heavy development with bugs showing up and getting squashed fairly frequently.
+ 2. Or, you can check out the latest source code from subversion and build it with Ant. This gets you closer to the bleeding edge of development. The 1.0 release should be relatively stable but the trunk (from which the [[http://lucene.apache.org/nutch/nightly.html|nightly builds]] are build) is under heavy development with bugs showing up and getting squashed fairly frequently.

Note: As of 5/29/08 the Subversion trunk seems to be much better than the 0.9 release. If you have trouble with 0.9 your best bet is to try moving to trunk and see if the problems resolve themselves.

[Nutch Wiki] Update of "ApacheConUs2009MeetUp" by KenKrugler

2009-10-27T06:12:54Z

[jira] Commented: (NUTCH-760) Allow field mapping from nutch to solr index

2009-10-27T04:16:59Z

[ https://issues.apache.org/jira/browse/NUTCH-760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770464#action_12770464 ]

David Stuart commented on NUTCH-760:
------------------------------------

Hi Andrzej,

I have amended the patch to incorporate your suggestions
https://issues.apache.org/jira/browse/NUTCH-760

Regards,

Dave

> Allow field mapping from nutch to solr index
> --------------------------------------------
>
> Key: NUTCH-760
> URL: https://issues.apache.org/jira/browse/NUTCH-760
> Project: Nutch
> Issue Type: Improvement
> Components: indexer
> Reporter: David Stuart
> Attachments: solrindex_schema.patch, solrindex_schema.patch, solrindex_schema.patch, solrindex_schema.patch
>
>
> I am using nutch to crawl sites and have combined it
> with solr pushing the nutch index using the solrindex command. I have
> set it up as specified on the wiki using the copyField url to id in the
> schema. Whilst this works fine it is stuff's up my inputs from other
> sources in solr (e.g. using the solr data import handler) as they have
> both id's and url's. I have patch that implements a nutch xml schema
> defining what basic nutch fields map to in your solr push.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (NUTCH-760) Allow field mapping from nutch to solr index

2009-10-27T03:58:59Z

[ https://issues.apache.org/jira/browse/NUTCH-760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

David Stuart updated NUTCH-760:
-------------------------------

Attachment: solrindex_schema.patch

Have updated patch as per comment below
* the description of the property in nutch-default.xml could be more descriptive

* <schema> element has name and version attributes - do we really need these? It's not a Solr schema.xml anyway, so we don't have to pretend that we follow the same format.

* SolrSchemaReader uses static instance of NutchConfiguration - this is a big no-no, the whole point of using the property in nutch-default.xml is that you could set different values, and making this field static basically pins down the configuration to the version set on the first instantiation of the class ... Please do as other similar classes do - implement Configurable, or add Configuration to the constructor, and pass the current job configuration where appropriate.

* consequently, static references to SolrSchemaReader need to be un-staticized in other places.

* minor nits: code formatting should use 2 literal spaces indents. There are some accidental changes in NutchBean and SolrWriter.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

How to index files only with specific type

2009-10-26T02:11:58Z

Hi, I've create parser and indexer to specific file type(geo xml meta file - kml).
I am trying to crawl couple of sites, and index only files of this type.
I don't want to index html or anything else.
How can I achieve this?
Thanks.

[jira] Commented: (NUTCH-755) DomainURLFilter crashes on malformed URL

2009-10-26T00:39:59Z

[ https://issues.apache.org/jira/browse/NUTCH-755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769926#action_12769926 ]

Reinhard Schwab commented on NUTCH-755:
---------------------------------------

in the first case, the "url" is rejected or?
the filter method will return null.

catch (Exception e) {

// if an error happens, allow the url to pass
LOG.error("Could not apply filter on url: " + url + "\n"
+ org.apache.hadoop.util.StringUtils.stringifyException(e));
return null;
}

the comment in the code is wrong.
if the method returns null, the url does not pass.

the malformed check is done by java.net.URL constructor.
it accepts http:/comments.php

> DomainURLFilter crashes on malformed URL
> ----------------------------------------
>
> Key: NUTCH-755
> URL: https://issues.apache.org/jira/browse/NUTCH-755
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Affects Versions: 1.0.0
> Environment: Tomcat 6.0.14
> Java 1.6.0_14
> Linux
> Reporter: Mike Baranczak
>
> 2009-09-16 21:54:17,001 ERROR [Thread-156] DomainURLFilter - Could not apply filter on url: http:/comments.php
> java.lang.NullPointerException
> at org.apache.nutch.urlfilter.domain.DomainURLFilter.filter(DomainURLFilter.java:173)
> at org.apache.nutch.net.URLFilters.filter(URLFilters.java:88)
> at org.apache.nutch.parse.ParseOutputFormat$1.write(ParseOutputFormat.java:200)
> at org.apache.nutch.parse.ParseOutputFormat$1.write(ParseOutputFormat.java:113)
> at org.apache.nutch.fetcher.FetcherOutputFormat$1.write(FetcherOutputFormat.java:96)
> at org.apache.nutch.fetcher.FetcherOutputFormat$1.write(FetcherOutputFormat.java:70)
> at org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:410)
> at org.apache.hadoop.mapred.lib.IdentityReducer.reduce(IdentityReducer.java:39)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:170)
> Expected behavior would be to recognize the URL as malformed, and reject it.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[Nutch Wiki] Trivial Update of "首页" by yongping8204

2009-10-24T09:37:39Z

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Nutch Wiki" for change notification.

The "首页" page has been changed by yongping8204.
http://wiki.apache.org/nutch/%E9%A6%96%E9%A1%B5?action=diff&rev1=4&rev2=5

--------------------------------------------------

#format wiki
#language zh
#pragma section-numbers off
-
= 维基链接名维基 =
您也许可以从这些连接开始:
+
- * [[最新改动]]: 谁最近改动了什么
+ * [[最新改动]]: 谁最近改动了什么 (我在修改)
* [[维基沙盘演练]]: 您可以随意改动编辑，热身演练
* [[查找网页]]: 用多种方法搜索浏览这个站点
- * [[语法参考]]: 维基语法简便参考
+ * [[语法参考]]: 维基语法简便参考
* [[站点导航]]: 本站点内容概要
+
这个维基是有关什么的?

测试

+ == 如何使用这个站点 ==
+ 维基(wiki)是一种协同合作网站，任何人都可以参与网站的建立、编辑和维护并分享网站的内容：

- == 如何使用这个站点 ==
-
- 维基(wiki)是一种协同合作网站，任何人都可以参与网站的建立、编辑和维护并分享网站的内容：
* 点击每个网页页眉或页尾中的'''<<GetText(Edit)>>'''就可以随意编辑改动这个网页。
* 创建一个链接简单的不能再简单了：您可以使用连在一起的，每个单词第一个字母大写，但不用空格分隔的词组(比如WikiSandBox)，也可以用{{{["quoted words in brackets"]}}}。简体中文的链接可以使用后者，比如{{{["维基沙盘演练"]}}}。
* 每页的页眉中的搜索框可以用来将进行网页标题搜索或者进行全文检索。

Re: datanode.BlockAlreadyExistsException

2009-10-21T21:29:25Z

I am still getting the same errors.

I ran fsck (rebooted with a forced fsck at startup) No issues.
I increased the ulimit to 8192

I was using /etc/hosts for all name lookups (common across all machines and copied from same location). I have since modified hadoop-site.xml and the slaves file to use IP address only.

Using ifconfig, ping, and looking at /etc/sysconfig/network I've determined that all the machines are who they think they are.

Of note may be that I also get the following WARN in the logs after the BlockAlreadyExistsException. I am seeing the same on both datanodes (just swap the IP addresses)

2009-10-21 21:13:03,415 WARN  datanode.DataNode - DatanodeRegistration(192.168.1.7:50010, storageID=DS-1226842861-192.168.1.7-50010-1254609174303, infoPort=50075, ipcPort=50020):Failed to transfer blk_-2053461958845826983_3919 to 192.168.1.6:50010 got java.net.SocketException: Original Exception : java.io.IOException: Connection reset by peer

	at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
	at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:456)
	at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:557)
	at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:199)

	at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:313)
	at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:400)
	at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:1108)

	at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.IOException: Connection reset by peer

I am able to generate/fetch/updatedb/etc.... As near as I can tell, things seem to be working, but I really wouldn't know if I am missing anything anyway. No errors are being displayed on the command line. Every iteration seems to be growing the index, segments, linkdb accordingly.

Here is the hadoop-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>



<configuration>

<property>
    <name>fs.default.name</name>
    <value>hdfs://192.168.1.3:9000</value>
</property>

<property>
    <name>mapred.job.tracker</name>
    <value>hdfs://192.168.1.3:9001</value>
</property>

<property>
    <name>mapred.tasktracker.tasks.maximum</name>
    <value>1</value>
</property>

<property>
    <name>mapred.child.java.opts</name>
    <value>-Xmx512m</value>
</property>

<property>
    <name>dfs.name.dir</name>
    <value>/home/nutch/crawl/filesystem/name</value>
</property>

<property>
    <name>dfs.data.dir</name>
    <value>/home/nutch/crawl/filesystem/data</value>
</property>

<property>
    <name>mapred.system.dir</name>
    <value>/home/nutch/crawl/filesystem/mapreduce/system</value>
</property>

<property>
    <name>mapred.local.dir</name>
    <value>/home/nutch/crawl/filesystem/mapreduce/local</value>
</property>

<property>
    <name>dfs.replication</name>
    <value>1</value>
</property>

</configuration>

Jesse

int GetRandomNumber()
{
return 4; // Chosen by fair roll of dice
// Guaranteed to be random
} // xkcd.com

On Wed, Oct 21, 2009 at 6:01 PM, Jesse Hires <jhires@...> wrote:

Thanks for the pointers!
As soon as I have some results, I'll post them back and let you know if the problem is solved.

Jesse

int GetRandomNumber()
{
return 4; // Chosen by fair roll of dice
// Guaranteed to be random
} // xkcd.com

On Wed, Oct 21, 2009 at 4:46 AM, Andrzej Bialecki <ab@...> wrote:

Jesse Hires wrote:

I tried asking this over at the nutch-user alias, but I am seeing very little traction, so I thought I'd ask the developers. I realize this is most likely a configuration problem on my end, but I am very new to using nutch, so I am having a difficult time understanding where I need to look.

Does anyone have any insight into the following error I am seeing in the hadoop logs? Is this something I should be concerned with, or is it expected that this shows up in the logs from time to time? If it is not expected, where can I look for more information on what is going on?

It's not expected at all - this usually indicates some config error, or FS corruption, or it may be also caused by conflicting DNS (e.g. the same name resolving to different addresses on different nodes), or a problem with permissions (e.g. daemon started remotely uses uid/permissions/env that doesn't allow it to create/delete files in data dir). This may be also some weird corner case when processes run out of file descriptors - you should check ulimit -n and set it to a value higher than 4096.

Please also run fsck / and see what it says.

I can also provide config files if needed.

We need just the modifications in hadoop-site.xml, that's where the problem may be located.

--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com

Re: datanode.BlockAlreadyExistsException

2009-10-21T18:01:03Z

Thanks for the pointers!
As soon as I have some results, I'll post them back and let you know if the problem is solved.

Jesse

int GetRandomNumber()
{
return 4; // Chosen by fair roll of dice
// Guaranteed to be random
} // xkcd.com

On Wed, Oct 21, 2009 at 4:46 AM, Andrzej Bialecki <ab@...> wrote:

Jesse Hires wrote:

I tried asking this over at the nutch-user alias, but I am seeing very little traction, so I thought I'd ask the developers. I realize this is most likely a configuration problem on my end, but I am very new to using nutch, so I am having a difficult time understanding where I need to look.

Does anyone have any insight into the following error I am seeing in the hadoop logs? Is this something I should be concerned with, or is it expected that this shows up in the logs from time to time? If it is not expected, where can I look for more information on what is going on?

It's not expected at all - this usually indicates some config error, or FS corruption, or it may be also caused by conflicting DNS (e.g. the same name resolving to different addresses on different nodes), or a problem with permissions (e.g. daemon started remotely uses uid/permissions/env that doesn't allow it to create/delete files in data dir). This may be also some weird corner case when processes run out of file descriptors - you should check ulimit -n and set it to a value higher than 4096.

Please also run fsck / and see what it says.

I can also provide config files if needed.

We need just the modifications in hadoop-site.xml, that's where the problem may be located.

--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com

Re: datanode.BlockAlreadyExistsException

2009-10-21T04:46:27Z

Jesse Hires wrote:

> I tried asking this over at the nutch-user alias, but I am seeing very
> little traction, so I thought I'd ask the developers. I realize this is
> most likely a configuration problem on my end, but I am very new to
> using nutch, so I am having a difficult time understanding where I need
> to look.
>
> Does anyone have any insight into the following error I am seeing in the
> hadoop logs? Is this something I should be concerned with, or is it
> expected that this shows up in the logs from time to time? If it is not
> expected, where can I look for more information on what is going on?

It's not expected at all - this usually indicates some config error, or
FS corruption, or it may be also caused by conflicting DNS (e.g. the
same name resolving to different addresses on different nodes), or a
problem with permissions (e.g. daemon started remotely uses
uid/permissions/env that doesn't allow it to create/delete files in data
dir). This may be also some weird corner case when processes run out of
file descriptors - you should check ulimit -n and set it to a value
higher than 4096.

Please also run fsck / and see what it says.

> I can also provide config files if needed.

We need just the modifications in hadoop-site.xml, that's where the
problem may be located.

--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com

datanode.BlockAlreadyExistsException

2009-10-20T16:22:05Z

I tried asking this over at the nutch-user alias, but I am seeing very little traction, so I thought I'd ask the developers. I realize this is most likely a configuration problem on my end, but I am very new to using nutch, so I am having a difficult time understanding where I need to look.

Does anyone have any insight into the following error I am seeing in the hadoop logs? Is this something I should be concerned with, or is it expected that this shows up in the logs from time to time? If it is not expected, where can I look for more information on what is going on?

2009-10-16 17:02:43,061 ERROR datanode.DataNode - DatanodeRegistration(192.168.1.7:50010, storageID=DS-1226842861-192.168.1.7-50010-1254609174303, infoPort=50075, ipcPort=50020):DataXceiver


org.apache.hadoop.hdfs.server.datanode.BlockAlreadyExistsException: Block blk_909837363833332565_3277 is valid, and cannot be written to.
	at org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:975)


	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:97)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:259)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)


	at java.lang.Thread.run(Thread.java:636)

I am able to produce this just injecting the urls (2 of them), but it shows up on both datanodes, and happens whenever I run an operation that uses dfs.

I am running the latest sources from the trunk.
I've verified that only one instance of the following on the datanodes:

org.apache.hadoop.hdfs.server.datanode.DataNode
org.apache.hadoop.mapred.TaskTracker

I've also verified that only one instance of the following are running on the name node:

org.apache.hadoop.hdfs.server.namenode.NameNode
org.apache.hadoop.mapred.JobTracker

The hardware is as follows:
Two data nodes, both configured identical. Atom 330 proc, 2gigs ram, 320g SATA 3.0 hard drive, Fedora Core 10.
One name node, running some amd x86 proc, 2 gigs memory, 750g SATA, Fedora Core 10. (pieced together from spare parts)
All across a 100mb network.
Admittedly this is low end hardware, but I am doing this specifically as an exercise in using low power (as in electricity) hardware.

I can also provide config files if needed.

Jesse

int GetRandomNumber()
{
return 4; // Chosen by fair roll of dice
// Guaranteed to be random
} // xkcd.com

[jira] Commented: (NUTCH-760) Allow field mapping from nutch to solr index

2009-10-20T13:50:59Z

[ https://issues.apache.org/jira/browse/NUTCH-760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767934#action_12767934 ]

David Stuart commented on NUTCH-760:
------------------------------------

Thanks,

I will have another go. It quite a big task getting my head around all of the
ins and outs of nutch but its good to help to contribute to a great product

Regards,

Dave

> Allow field mapping from nutch to solr index
> --------------------------------------------
>
> Key: NUTCH-760
> URL: https://issues.apache.org/jira/browse/NUTCH-760
> Project: Nutch
> Issue Type: Improvement
> Components: indexer
> Reporter: David Stuart
> Attachments: solrindex_schema.patch, solrindex_schema.patch, solrindex_schema.patch
>
>
> I am using nutch to crawl sites and have combined it
> with solr pushing the nutch index using the solrindex command. I have
> set it up as specified on the wiki using the copyField url to id in the
> schema. Whilst this works fine it is stuff's up my inputs from other
> sources in solr (e.g. using the solr data import handler) as they have
> both id's and url's. I have patch that implements a nutch xml schema
> defining what basic nutch fields map to in your solr push.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (NUTCH-760) Allow field mapping from nutch to solr index

2009-10-20T12:42:59Z

[ https://issues.apache.org/jira/browse/NUTCH-760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767918#action_12767918 ]

Andrzej Bialecki commented on NUTCH-760:
-----------------------------------------

A few comments to the latest patch:

* the description of the property in nutch-default.xml could be more descriptive ;)

* <schema> element has name and version attributes - do we really need these? It's not a Solr schema.xml anyway, so we don't have to pretend that we follow the same format.

* SolrSchemaReader uses static instance of NutchConfiguration - this is a big no-no, the whole point of using the property in nutch-default.xml is that you could set different values, and making this field static basically pins down the configuration to the version set on the first instantiation of the class ... Please do as other similar classes do - implement Configurable, or add Configuration to the constructor, and pass the current job configuration where appropriate.

* consequently, static references to SolrSchemaReader need to be un-staticized in other places.

* minor nits: code formatting should use 2 literal spaces indents. There are some accidental changes in NutchBean and SolrWriter.

> Allow field mapping from nutch to solr index
> --------------------------------------------
>
> Key: NUTCH-760
> URL: https://issues.apache.org/jira/browse/NUTCH-760
> Project: Nutch
> Issue Type: Improvement
> Components: indexer
> Reporter: David Stuart
> Attachments: solrindex_schema.patch, solrindex_schema.patch, solrindex_schema.patch
>
>
> I am using nutch to crawl sites and have combined it
> with solr pushing the nutch index using the solrindex command. I have
> set it up as specified on the wiki using the copyField url to id in the
> schema. Whilst this works fine it is stuff's up my inputs from other
> sources in solr (e.g. using the solr data import handler) as they have
> both id's and url's. I have patch that implements a nutch xml schema
> defining what basic nutch fields map to in your solr push.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Re: solr index question

2009-10-20T11:48:15Z

Hi Andrzej,

updated patch submitted including SolrSearchBean modifications anything else needed? If not how do I get this into trunk?
https://issues.apache.org/jira/browse/NUTCH-760

Regards,

Dave