whole web crawl

[Gaurang Patel]

All-

I am novice to using Nutch. Can anyone tell me the estimated size in (I suppose, in TBs) that will be required to store the crawled results for whole web? I want to get estimate of the memory requirements for my project, that uses Nutch web crawler.



Regards,
Gaurang Patel

[kevin chen-6]

The estimated size of Google's index is 15 billion. So even for 1KB per
page, it will be 15 TBs. But I think the average page size is way more
than 1k.


On Sun, 2009-10-04 at 17:28 -0700, Gaurang Patel wrote:

> All-
>
> I am novice to using Nutch. Can anyone tell me the estimated size in
> (I suppose, in TBs) that will be required to store the crawled results
> for whole web? I want to get estimate of the memory requirements for
> my project, that uses Nutch web crawler.
>
>
>
> Regards,
> Gaurang Patel

[Gaurang Patel]

Hey Kevin,

You are right. It's around 30-40 TBs for google. But as far as Nutch is concerned, I think Jack Yu is right.

Following is what he said, in case you did not receive it:
0.1 billion pages for 1.5TB

Regards,
Gaurang

2009/10/4 kevin chen <kevinchen@...>

The estimated size of Google's index is 15 billion. So even for 1KB per
page, it will be 15 TBs. But I think the average page size is way more
than 1k.

On Sun, 2009-10-04 at 17:28 -0700, Gaurang Patel wrote:
> All-
>
> I am novice to using Nutch. Can anyone tell me the estimated size in
> (I suppose, in TBs) that will be required to store the crawled results
> for whole web? I want to get estimate of the memory requirements for
> my project, that uses Nutch web crawler.
>
>
>
> Regards,
> Gaurang Patel