« Return to Thread: Plugins: when to perform web service requests, on fetch or on index?

Plugins: when to perform web service requests, on fetch or on index?

by caezar :: Rate this Message:

Reply to Author | View in Thread

Hi All,

I'm writing several nutch plugins, which will perform a requests to some webservices for pages being indexed and store retrieved data in index. The question is: on what stage of crawling it is better to perform these webservice requests: on fetching or on indexing (in HtmlParseFilter or in IndexingFilter), in terms of performance, of course?

Nutch version is 1.0, indexer is SolrIndexer.

Thanks.

 « Return to Thread: Plugins: when to perform web service requests, on fetch or on index?