Re: Per-host fetch-interval

Sandeep Tata wrote:

Hi,

I was wondering what would be the best way to configure per-host
re-crawl intervals. The default db.fetch.interval applies to all URLs,
but I'd like for some hosts to be recrawled more frequently. Is there
a JIRA ticket open on this? I haven't been able to find one

Fetch interval can be set on individual CrawlDatum-s in crawldb, at least technically speaking. In practice, there is no command-line tool to do this, and I don;t think there is a JIRA on this.

One idea would be to modify the Injector to accept a list of URL-s with matching metadata, and among others use a predefined metadata like fetchInterval. On the initial injection, all values in CrawlDatum would be set according to the metadata (or set to defaults). On subsequent injections, if a URL already exists in CrawlDb, its metadata would be reset to the values supplied in the injector file.

This should be easy to implement, and I think it would support your use case.

--
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com