All,
I am a masters student and want to crawl the whole web for my masters project.
While trying to generate, fetch, crawl the whole web using Nutch (I am following steps from
http://lucene.apache.org/nutch/tutorial8.html), I got confused among various nutch terms and usage:
1) What is the purpose and difference between
crawl_fetch and
crawldb ? If nutch stores all the info regarding urls in
crawldb, then what is the need for
crawl_fetch?
2) Moreover, what does fetch and generate do? Can anyone describe in detail? Is there any documentation for nutch commands like generate, fetch, etc?
Thanks & Regards,
Gaurang Patel
