« Return to Thread: Show db_gone in crawlDB

Re: Show db_gone in crawlDB

by Xiangjun(XJ) Wang :: Rate this Message:

Reply to Author | View in Thread

bin/nutch readdb <crawldb> -dump <out_dir> -format csv

schroedi wrote:

> Hi Nutch Guys,
>
> I used to show the crawldb stats. Now I want to show which urls are
> db_gone (it means an error 404 - or anything else)
> how may I showing the db_gone urls?
>
> bin/nutch readdb crawl/crawldb -stats
> CrawlDb statistics start: crawl/crawldb
> Statistics for CrawlDb: crawl/crawldb
> TOTAL urls:    2157
> retry 0:    2154
> retry 5:    3
> min score:    0.0
> avg score:    0.018363468
> max score:    3.01
> status 1 (db_unfetched):    1971
> status 2 (db_fetched):    158
> status 3 (db_gone):    13
> status 4 (db_redir_temp):    1
> status 5 (db_redir_perm):    14
> CrawlDb statistics: done
>
> thanks,
>
> Mario
>
>  

 « Return to Thread: Show db_gone in crawlDB