Nutch 1.0 on the limits of the data

[Polsnet]

Nutch 1.0 largest number of data can support? (File size or number of records)

[Otis Gospodnetic-2]

Depends on hardware, of course!

 Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



----- Original Message ----

> From: Polsnet <polsnet@...>
> To: nutch-user@...
> Sent: Friday, July 3, 2009 12:03:30 AM
> Subject: Nutch 1.0 on the limits of the data
>
>
> Nutch 1.0 largest number of data can support? (File size or number of
> records)
> --
> View this message in context:
> http://www.nabble.com/Nutch-1.0-on-the-limits-of-the-data-tp24317298p24317298.html
> Sent from the Nutch - User mailing list archive at Nabble.com.

[Dennis Kubes-2]

Simple answer is billions, perhaps tens to hundreds of billions of
records, as it leverages Hadoop.  Yahoo is currently using Hadoop to
create its web index.  But as Otis pointed out, Hadoop is parallel
processing and as such is completely dependent on amount of hardware.

Dennis

Polsnet wrote:
> Nutch 1.0 largest number of data can support? (File size or number of
> records)