Nabble - Nutch - User

nutch search yields 0 results

2009-11-10T20:00:52Z

Hi all...

I was finally able to set up a multinode nutch cluster that seemed to work fine. When I set it up to do the example crawl of http://lucene.apache.org then the crawl seemed to finish successfully as indicated by the output on the console. When I copied the index files on to the local filesystem from the dfs and viewed the web application of nutch.war using tomcat on localhost:8080, all searches yield 0 results. The log files do not indicate any exceptions. I don't understand what is wrong. Please help. Thanks in advance.

nutch@athena:/nutch/search$ bin/nutch crawl urls -dir crawled -depth 2
crawl started in: crawled
rootUrlDir = urls
threads = 10
depth = 2
Injector: starting
Injector: crawlDb: crawled/crawldb
Injector: urlDir: urls
Injector: Converting injected urls to crawl db entries.
Injector: Merging injected urls into crawl db.
Injector: done
Generator: Selecting best-scoring urls due for fetch.
Generator: starting
Generator: segment: crawled/segments/20091110223659
Generator: filtering: true
Generator: Partitioning selected urls by host, for politeness.
Generator: done.
Fetcher: Your 'http.agent.name' value should be listed first in 'http.robots.agents' property.
Fetcher: starting
Fetcher: segment: crawled/segments/20091110223659
Fetcher: done
CrawlDb update: starting
CrawlDb update: db: crawled/crawldb
CrawlDb update: segments: [crawled/segments/20091110223659]
CrawlDb update: additions allowed: true
CrawlDb update: URL normalizing: true
CrawlDb update: URL filtering: true
CrawlDb update: Merging segment data into db.
CrawlDb update: done
Generator: Selecting best-scoring urls due for fetch.
Generator: starting
Generator: segment: crawled/segments/20091110223831
Generator: filtering: true
Generator: Partitioning selected urls by host, for politeness.
Generator: done.
Fetcher: Your 'http.agent.name' value should be listed first in 'http.robots.agents' property.
Fetcher: starting
Fetcher: segment: crawled/segments/20091110223831
Fetcher: done
CrawlDb update: starting
CrawlDb update: db: crawled/crawldb
CrawlDb update: segments: [crawled/segments/20091110223831]
CrawlDb update: additions allowed: true
CrawlDb update: URL normalizing: true
CrawlDb update: URL filtering: true
CrawlDb update: Merging segment data into db.
CrawlDb update: done
LinkDb: starting
LinkDb: linkdb: crawled/linkdb
LinkDb: URL normalize: true
LinkDb: URL filter: true
LinkDb: adding segment: hdfs://athena:9000/user/nutch/crawled/segments/20091110223659
LinkDb: adding segment: hdfs://athena:9000/user/nutch/crawled/segments/20091110223831
LinkDb: done
Indexer: starting
Indexer: done
Dedup: starting
Dedup: adding indexes in: crawled/indexes
Dedup: done
merging indexes to: crawled/index
Adding hdfs://athena:9000/user/nutch/crawled/indexes/part-00000
done merging
crawl finished: crawled

In addition, my crawl-urlfilter.txt has the following:

# accept hosts in MY.DOMAIN.NAME
+^http://([a-z0-9]*\.)*apache.org/

Also, the urllist.txt file has
http://lucene.apache.org

Nutch 0.20

2009-11-10T19:47:36Z

Hi everyone,

Does anybody know of a good way or is it possible to run nutch on
Hadoop 0.20.x?

thank you,

-John

dear

2009-11-10T17:32:58Z

Dear friend,<BR>How are you recently?<BR>I bought a laptop from a
website: <A href="http://www.yeurl.com">www.yeurl.com</A> last week.
<BR>I have got the product. Its quality is very good and the price
is<BR>competitive. They also sell phones, TV, psp, motor and so on. By
the<BR>way, they import product from Korea and sell new and
original<BR>products. They have good reputation and have many good
feedbacks. If<BR>you need these products, look at this website will be
a clever choice.<BR>I am sure you will get many surprise and
benefits.<BR>Greetings!

Apache Hadoop Get Together Berlin - December 2009

2009-11-10T16:35:28Z

As announced at ApacheCon US, the next Apache Hadoop Get Together Berlin is
scheduled for December 2009.

When: Wednesday December 16, 2009 at 5:00pm
Where: newthinking store, Tucholskystr. 48, Berlin

As always there will be slots of 20min each for talks on your Hadoop topic.
After each talk there will be a lot time to discuss. You can order drinks
directly at the bar in the newthinking store. If you like, you can order
pizza. We will go to Cafe Aufsturz after the event for some beer and
something to eat.

Talks scheduled so far:

Richard Hutton (nugg.ad): "Moving from five days to one hour." - This talk
explains how we made data processing scalable at nugg.ad. The company's core
business is online advertisement targeting. Our servers receive 10,000
requests per second resulting in data of 100GB per day.

As the classical data warehouse solution reached its limit, we moved to a
framework built on top of Hadoop to make analytics speedy,data mining
detailed and all of our lives easier. We will give an overview of our
solution involving file system structures, scheduling, messaging and
programming languages from the future.

Jörg Möllenkamp (Sun): "Hadoop on Sun"
Abstract: Hadoop is a well known technology inside of Sun. This talk want to
show some interesting use cases of Hadoop in conjunction with Sun
technologies. The first show case wants to demonstrate how Hadoop can used to
load massive multicore system with up to 256 threads in a single system to
the max. The second use case shows how several mechanisms integrated in
Solaris can ease the deployment and operation of Hadoop even in non-dedicated
environments. The last usecase will show the combination of the Sun Grid
Engine and Hadoop. Talk may contain command-line demonstrations ;).

Nikolaus Pohle (nurago): "M/R for MR - Online Market Research powered by
Apache Hadoop. Enable consultants to analyze online behavior for audience
segmentation, advertising effects and usage patterns."

We would like to invite you, the visitor to also tell your Hadoop story, if
you like, you can bring slides - there will be a beamer.

A big Thanks goes to the newthinking store for providing a room in the center
of Berlin for us. Another big thanks goes to StudiVZ for sponsoring videos of
the talks. Links to the videos will be posted here as well as on the StudiVZ
blog.

Please do indicate on the following upcoming event if you are planning to
attend to make planning (and booking tables at Aufsturz) easier:

http://upcoming.yahoo.com/event/4842528/

Looking forward to seeing you in Berlin,
Isabel

--
|\ _,,,---,,_ Web: <http://www.isabel-drost.de>
/,`.-'`' -. ;-;;,_
|,4- ) )-,_..;\ ( `'-'
'---''(_/--' `-'\_) (fL) IM: <xmpp://MaineC.@...>

signature.asc (204 bytes) Download Attachment

Re: How to make a Lucene-built index work with Nutch?

2009-11-10T16:30:30Z

hi,

not sure if this will work in your case;

but in a nutshell -

first create a nutch index by crawling some urls.

open both indexes ie
IndexReader r = IndexReader.open(nutch_index)
IndexReader r2 = IndexReader.open(your_index)

then write a new index;

IndexWriter writer = new IndexWriter(new_index,...)
writer.addIndexes(new IndexReader[]{r,r2})

replace your nutch index with new_index.

>
> I have built a custom index using Lucene, as the data source is not web
> pages
> but some custom text files. I stored several indexed fields in my index.
>
> Now my colleague requires me to make sure this index works with Nutch. So
> I
> set up Nutch on the server, followed the steps, crawled some pages and
> build
> an index, and use that index for search, everything is fine.
>
> Then I tried to replace the Nutch-built index to my Lucene-built one, by
> changing the <searcher.dir> value in conf/nutch-site.xml. It does not
> return
> anything. I know some extra work needs to be done to get it work, but
> don't
> know how.
>
> I've read some examples for building a plugin, but my problem seems a
> little
> different. I don't think I need to build a IndexFilter extension, but
> maybe
> only a QueryFilter or QueryParser, but not sure about any details.
>
> Newbie to Nutch, and thanks a lot for your help!
> --
> View this message in context:
> http://old.nabble.com/How-to-make-a-Lucene-built-index-work-with-Nutch--tp26286011p26286011.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
>

Re: How do I block/ban a specific domain name or a tld?

2009-11-10T14:43:36Z

opsec schrieb:
> I've added this to my conf/crawl-urlfilter.txt and conf/regex-urlfilter.txt
> yet when I start a crawl this domain is heavily spidered. I would like to
> remove it from my search results entirely and prevent it from being crawled
> in the future and possibly all *.int tlds, how can i accomplish this?
>
> -^http://([a-z0-9]*\.)*who.int/
>
why not

-^http://[^/]*\.int/

> Thanks for your time and any assistance,
>
> -Warren
>

How do I block/ban a specific domain name or a tld?

2009-11-10T12:26:52Z

I've added this to my conf/crawl-urlfilter.txt and conf/regex-urlfilter.txt yet when I start a crawl this domain is heavily spidered. I would like to remove it from my search results entirely and prevent it from being crawled in the future and possibly all *.int tlds, how can i accomplish this?

-^http://([a-z0-9]*\.)*who.int/

Thanks for your time and any assistance,

-Warren

How to make a Lucene-built index work with Nutch?

2009-11-10T08:11:47Z

I have built a custom index using Lucene, as the data source is not web pages but some custom text files. I stored several indexed fields in my index.

Now my colleague requires me to make sure this index works with Nutch. So I set up Nutch on the server, followed the steps, crawled some pages and build an index, and use that index for search, everything is fine.

Then I tried to replace the Nutch-built index to my Lucene-built one, by changing the <searcher.dir> value in conf/nutch-site.xml. It does not return anything. I know some extra work needs to be done to get it work, but don't know how.

I've read some examples for building a plugin, but my problem seems a little different. I don't think I need to build a IndexFilter extension, but maybe only a QueryFilter or QueryParser, but not sure about any details.

Newbie to Nutch, and thanks a lot for your help!

Re: changing/addding field in existing index

2009-11-09T20:15:06Z

that seems to work. thanks for that. it was a bit fiddly more than i
expected but got the index sorted.

found an issue with sorting as most fields cannot be sorted by; and
throwing a

java.lang.RuntimeException: Unknown sort value type!
at
org.apache.nutch.searcher.IndexSearcher.translateHits(IndexSearcher.java:159)
at
org.apache.nutch.searcher.IndexSearcher.search(IndexSearcher.java:98)
at
org.apache.nutch.searcher.LuceneSearchBean.search(LuceneSearchBean.java:84)
at org.apache.nutch.searcher.NutchBean.search(NutchBean.java:231)

On Mon, 2009-11-09 at 17:34 +0100, Andrzej Bialecki wrote:

> fadzi@... wrote:
> > hi all,
> >
> > i have an existing index - we have a custom field that needs to be added
> > or changed in every currently indexed document ;
> >
> > whats the best way to go about this without recreating the index again?
>
> There are ways to do it directly on the index, but this is complicated
> and involves hacking the low-level Lucene format. Alternatively, you
> could build a parallel index with just these fields, but synchronized
> internal docId-s, open both indexes with ParallelReader, and then create
> a new index using IndexWriter.addIndexes().
>
> I suggest recreating the index.
>

Cannot get slave nodes to run

2009-11-09T12:57:11Z

Hi All...

I have been trying to set up nutch on a cluster of 3 machines. I could get the crawling and searching process to run independently on all 3 machines but when I try to integrate them as a single cluster, then none of the slaves are shown in the listing of nodes on the Hadoop Machine List page (http://master:50030/machines.jsp).

There are no exceptions in any of the log files, and several log files are even empty. I am out of ideas. I will appreciate any help. Thanks in advance.

Here is my hadoop-site.xml (identical on all nodes)

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>



<configuration>

<property>
<name>hadoop.tmp.dir</name>
<value>/nutch/filesystem/tmp/</value>
</property>

<property>
<name>fs.default.name</name>
<value>hdfs://athena:9000</value>
</property>

<property>
<name>mapred.job.tracker</name>
<value>athena:9001</value>
</property>

<property>
<name>dfs.replication</name>
<value>2</value>
</property>

<property>
<name>dfs.name.dir</name>
<value>/nutch/filesystem/name</value>
</property>

<property>
<name>dfs.data.dir</name>
<value>/nutch/filesystem/data</value>
</property>

<property>
<name>mapred.system.dir</name>
<value>/nutch/filesystem/mapreduce/system</value>
</property>

<property>
<name>mapred.local.dir</name>
<value>/nutch/filesystem/mapreduce/local</value>
</property>

</configuration>

Re: changing/addding field in existing index

2009-11-09T08:34:50Z

fadzi@... wrote:
> hi all,
>
> i have an existing index - we have a custom field that needs to be added
> or changed in every currently indexed document ;
>
> whats the best way to go about this without recreating the index again?

There are ways to do it directly on the index, but this is complicated
and involves hacking the low-level Lucene format. Alternatively, you
could build a parallel index with just these fields, but synchronized
internal docId-s, open both indexes with ParallelReader, and then create
a new index using IndexWriter.addIndexes().

I suggest recreating the index.

--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com

Nutch near future - strategic directions

2009-11-09T08:24:06Z

Hi all,

The ApacheCon is over, our release 1.0 has been out already for some
time, so I think it's a good moment to discuss what are the next steps
in Nutch development.

Let me share with you the topics I identified and presented in the
ApacheCon slides, and some topics that are worth discussing based on
various conversations I had there, and the discussions we had on our
mailing list:

1. Avoid duplication of effort
------------------------------
Currently we spend significant effort on implementing functionality that
other projects are dedicated to. Instead of doing the same work, and
sometimes poorly, we should concentrate on delegating and reusing:

* Use Tika for content parsing: this will require some effort and
collaboration with the Tika project, to improve Tika's ability to handle
more complex formats well (e.g. hierarchical compound documents such as
archives, mailboxes, RSS), and to contribute any missing parsers (e.g.
parse-swf).

* Use Solr for indexing & search: it is hard to justify the effort of
developing and maintaining our own search server - Solr offers much more
functionality, configurability, performance and ease of integration than
our relatively primitive search server. Our integration with Solr needs
to be improved so that it's easier to setup and operate.

* Use database-like storage abstraction: this may seem like a serious
departure from the current architecture, but I don't mean that we should
switch to an SQL DB ... what this means is that we should provide an
option to use HBase, as well as the current plain MapFile-s (and perhaps
other types of DBs, such as Berkeley DB or SQL, if it makes sense) as
our storage. There is a very promising initial port of Nutch to HBase,
which is currently closely integrated with HBase API (which is both good
and bad) - it provides several improvements over our current storage, so
I think it's worth using as the new default, but let's see if we can
make it more abstract.

* Plugins: the initial OSGI port looks good, but I'm not sure yet at
this moment if the benefits of OSGI outweigh the cost of this change ...

* Shard management: this is currently an Achilles' heel of Nutch, where
users are left on their own ... If we switch to using HBase then at
least on the crawling side the shard management will become much easier.
This still leaves the problem of deploying new content to search
server(s). The candidate framework for this side of the shard management
is Katta + patches provided by Ted Dunning (see ???). If we switch to
using Solr we would have to also use the Katta / Solr integration, and
perhaps Solr/Hadoop integration as well. This is a complex mix of
half-ready components that needs to be well thought-through ...

* Crawler Commons: during our Crawler MeetUp all representatives agreed
that we should collect a few components that are nearly the same across
all projects and collaborate on their development, and use them as an
external dependency. The candidate components are:

- robots.txt parsing
- URL filtering and normalization
- page signature (fingerprint) implementations
- page template detection & removal (aka. main content extraction)
- possibly others, like URL redirection tracking, PageRank
calculation, crawler trap detection etc.

2. Make Nutch easier to use
---------------------------
This, as you may remember our earlier discussions, begs the question:
who is the target audience of Nutch?

In my opinion, the main users of Nutch are vertical search engines, and
this is the audience that we should cater to. There are many reasons for
this:

- Nutch is too complex and too heavy for those that need to crawl up to
a few thousand pages. Now that the Droids project exists it's probably
not worth the effort to attempt a complete re-design of Nutch so that it
fits the need of this group - Nutch is based on map-reduce, and it's not
likely we would want to change that, so this means there will always be
a significant overhead for small jobs. I'm not saying we should not make
Nutch easier to use, but for small crawls Nutch is an overkill. Also, in
many cases these users don't realize that they don't do any frontier
discovery and expansion, and what they really need is Solr.

- at the other end of the spectrum, there are very very few companies
that want to do a wide large web-scale crawling - this is costly, and
requires a solid business plan and serious funding. These users are
prepared anyway to spend significant effort on customizations and
problem-solving, or they may want to use only some parts of Nutch. Often
they are also not too eager to contribute back to the project - either
because of their proprietary nature or because their customizations are
not useful for general audience.

The remaining group is interested in medium-size, high quality crawling
(focused, with good spam & junk controls). Which is either an enterprise
search or a vertical search. We should make Nutch an attractive platform
for such users, and we should discuss what this entails. Also, if we
refactor Nutch in the way I described above, it will be easier for such
users to contribute back to Nutch and other related projects.

3. Provide a platform for solving the really interesting issues
---------------------------------------------------------------
Nutch has many bits and pieces that implement really smart algorithms
and heuristics to solve difficult issues that occur in crawling. The
problem is that they are often well hidden and poorly documented, and
their interaction with the rest of the system is far from obvious.
Sometimes this is related to premature performance optimizations, in
other cases this is just a poorly abstracted design. Examples would
include the OPIC scoring, meta-tags & metadata handling, deduplication,
redirection handling, etc.

Even though these components are usually implemented as plugins, this
lack of transparency and poor design makes it difficult to experiment
with Nutch. I believe that improving this area will result in many more
users contributing back to the project, both from business and from
academia.

And there are quite a few interesting challenges to solve:

* crawl scheduling, i.e. determining the order and composition of
fetchlists to maximize the crawling speed.

* spam & junk detection (I won't go into details on this, there are tons
of literature on the subject)

* crawler trap handling (e.g. the classic calendar page that generates
infinite number of pages).

* enterprise-specific ranking and scoring. This includes users' feedback
(explicit and implicit, e.g. click-throughs)

* pagelet-level crawling (e.g. portals, RSS feeds, discussion fora)

* near-duplicate detection, and closely related issue of extraction of
the main content from a templated page.

* URL aliasing (e.g. www.a.com == a.com == a.com/index.html ==
a.com/default.asp), and what happens with inlinks to such aliased pages.
Also related to this is the problem of temporary/permanent redirects and
complete mirrors.

Etc, etc ... I'm pretty sure there are many others. Let's make Nutch an
attractive platform to develop and experiment with such components.

-----------------
Briefly ;) that's what comes to my mind when I think about the future of
Nutch. I invite you all to share your thoughts and suggestions!

--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com

RE: Simple vertical search engine question

2009-11-09T08:11:13Z

Premium Google publishers (>20 mlns pageviews per month) may use more
features of AdSense such as explicit keywords in a query (to Google)

> -----Original Message-----
> From: Carlos Vera [mailto:carlodesilva2@...]
> Sent: November-09-09 10:53 AM
> To: nutch-user@...
> Subject: Simple vertical search engine question
>
> I have looked into few vertical search engines like indeed.com,
> simplyhired.com. Anyone know how vertical search engine like indeed.com
and
> simplyhired.com displays relevant google ads for the searched keywords on
> thier site?

Re: PRUNE : need some help on pruning syntax.

2009-11-09T08:06:54Z

one option is to extend the html parser and look for these things and
ignore them.

you might also want to look at this forum posting:

http://www.mail-archive.com/nutch-user@.../msg13969.html

On Mon, 2009-11-09 at 07:39 -0800, Annappa wrote:

> Hi,
>
> I am unsing Nutch-0.9 for crawing of sime web application which has a
> header part, menu part , left navigation and main contetn area.
>
> When i do a search on a perticular key word and if that appears in the main
> menu, then results are repeating as many times as pages are, bcz the menu
> will be included in all the pages. So i need to restrict my search not to
> search with the content of a perticular div
>
> ex : <div class="menu"> ................ </div>.
>
>
> Ho do i remove the content between a div from a search
>

Simple vertical search engine question

2009-11-09T07:52:32Z

I have looked into few vertical search engines like indeed.com,
simplyhired.com. Anyone know how vertical search engine like indeed.com and
simplyhired.com displays relevant google ads for the searched keywords on
thier site?

PRUNE : need some help on pruning syntax.

2009-11-09T07:39:07Z

Hi,

I am unsing Nutch-0.9 for crawing of sime web application which has a header part, menu part , left navigation and main contetn area.

When i do a search on a perticular key word and if that appears in the main menu, then results are repeating as many times as pages are, bcz the menu will be included in all the pages. So i need to restrict my search not to search with the content of a perticular div

ex : <div class="menu"> ................ </div>.

Ho do i remove the content between a div from a search

Re: Growing the index : Merging vs incremental

2009-11-08T20:10:01Z

hi,

we are in a sort of similar situation;

So would be really happy to hear any suggestions on this.

incremental crawling doesnt seem to really work for us because it seems
the same urls are being crawled over and over (on a daily basis!);

have you tried these settings or similar?

db.fetch.schedule.class = AdaptiveFetchSchedule
db.update.additions.allowed = true
db.ignore.internal.links = false
db.ignore.external.links = true (because we are intranet only)

>
>
> Currently we crawl every two days and create a new index and then merge
> with
> earlier index. For one it takes too long as mergesegs seems to take time
> proportional to the size of both indexes combined. Equally problematic
> issue is mergesegs fail a significant portion of the time. Probability
> becomes higher with size.Problems exist whether merge is done within
> Hadoop
> or outside.
>
> Two questions:
> (a) Has anybody been successful to do a Nutch merge predictably
> irrespective
> of the size. Any tips. We are trying to merge upto data for 200K url at a
> time.
>
> (b) How can we do incremental indexing, where we add data from latest
> crawl,
> but there is only one index that keeps growing. I saw lot of older posts
> regarding incremental indexing and no clear answers.
>
> Thanks in advance for your help.
>
> Shreekanth
>
> --
> View this message in context:
> http://old.nabble.com/Growing-the-index-%3A-Merging-vs-incremental-tp26228341p26228341.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
>

changing/addding field in existing index

2009-11-08T19:53:15Z

hi all,

i have an existing index - we have a custom field that needs to be added
or changed in every currently indexed document ;

whats the best way to go about this without recreating the index again?

currently some documents have the field some dont;

the ones that have it need to be updated - eventualy this field will be
used for sorting.

Thanks;

Re: MergeSegments - java.lang.OutOfMemoryError

2009-11-08T02:19:28Z

Hi guys,

Could you send a stack trace of the process? Have you tried using a profiler
to check where the memory was used?
Check http://hadoop.apache.org/common/docs/current/mapred_tutorial.html for
instructions on how to profile with Hadoop in (pseudo) distributed mode.

Julien
--
DigitalPebble Ltd
http://www.digitalpebble.com

2009/11/8 Fadzi Ushewokunze <fadzi@...>

> i have a similar issue; i havent been able to get to the bottom of it.
>
> On Sat, 2009-11-07 at 23:31 -0500, kevin chen wrote:
> > Hi, I have using a trunk version of nutch since Jul 2007. It's being
> > running fine since.
> >
> > Recently I am experimenting with nutch 1.0. Everything worked great and
> > better until I start to use MergeSegments. I was merging segments with
> > around 20k urls and it gave me OutOfMemoryError. I have tried to
> > increase the java heap max to 3G, I still got OutOfMemoryError. In
> > contrast, in my older version of nutch, same merge works with the
> > default java heap max setting of only 1G.
> >
> > Dose anybody have the same experience? Is there any work around this?
> >
> > Thanks
> > Kevin Chen
> >
>
>

Re: MergeSegments - java.lang.OutOfMemoryError

2009-11-08T01:23:53Z

i have a similar issue; i havent been able to get to the bottom of it.

On Sat, 2009-11-07 at 23:31 -0500, kevin chen wrote:

> Hi, I have using a trunk version of nutch since Jul 2007. It's being
> running fine since.
>
> Recently I am experimenting with nutch 1.0. Everything worked great and
> better until I start to use MergeSegments. I was merging segments with
> around 20k urls and it gave me OutOfMemoryError. I have tried to
> increase the java heap max to 3G, I still got OutOfMemoryError. In
> contrast, in my older version of nutch, same merge works with the
> default java heap max setting of only 1G.
>
> Dose anybody have the same experience? Is there any work around this?
>
> Thanks
> Kevin Chen
>

MergeSegments - java.lang.OutOfMemoryError

2009-11-07T20:31:21Z

Hi, I have using a trunk version of nutch since Jul 2007. It's being
running fine since.

Recently I am experimenting with nutch 1.0. Everything worked great and
better until I start to use MergeSegments. I was merging segments with
around 20k urls and it gave me OutOfMemoryError. I have tried to
increase the java heap max to 3G, I still got OutOfMemoryError. In
contrast, in my older version of nutch, same merge works with the
default java heap max setting of only 1G.

Dose anybody have the same experience? Is there any work around this?

Thanks
Kevin Chen

Re: What are the configuration parameters to fine tune Nutch performance

2009-11-07T17:18:21Z

The default tuning parameters are specified in nutch/conf/nutch-default.xml, and can be overridden in nutch/conf/nutch-site.xml. (Or in the crawl command line, but I believe that the 'best practice' is to configure settings in nutch-site.xml.)

My personal belief is that the two most valuable parameters for tuning the crawler are 'fetcher.threads.fetch' and 'fetcher.threads.per.host'. However, there are lots of other parameters for tuning, and you might find more value in some of the timeout parameters. (You might also want to look at tuning you JVM heap space, but I've never seen a real need to tweak it.)

As far as resuming a failed crawl, I don't know of any way to do so. I always discard and restart.

Re: can Nutch crawl XLS and XLSX file???

2009-11-07T16:43:27Z

Nutch can index MS Word, MS Powerpoint, MS Excel, and PDF files. In order for these types to be crawled, you need to have the plugins specified in the plugin.includes value of nutch/conf/nutch-site.xml (values are 'parse-(msexcel|mspowerpoint|msword|pdf)'.)

I was not sure if the new XLSX format was supported, so I looked at nutch/conf/tika-mimetypes.xml. From what I can tell, the XLSX files are not supported (as of the 11/6/2009 build of Nutch), only the following extensions are supported for are:

<mime-type type="application/vnd.ms-excel">
<magic priority="50">
<match value="Microsoft Excel 5.0 Worksheet" type="string"
offset="2080" />
</magic>
<glob pattern="*.xls" />
<glob pattern="*.xlc" />
<glob pattern="*.xll" />
<glob pattern="*.xlm" />
<glob pattern="*.xlw" />
<glob pattern="*.xla" />
<glob pattern="*.xlt" />
<glob pattern="*.xld" />
<alias type="application/msexcel" />
</mime-type>

...of course, I could be wrong.

Re: How to make nutch crawl within a sub category of an URL?

2009-11-07T14:44:05Z

If it were me, I'd try the following...

Use 'http://answers.yahoo.com/dir/index;_ylt=AqH5s00Y0dXDEjwmdUrxNabpy6IX;_ylv=3?link=list&sid=396545660' as a starting point URL, and set up the following filtering rules (crawl-urlfilter.txt):

+^http://answers.yahoo.com/dir/index;_ylt=AqH5s00Y0dXDEjwmdUrxNabpy6IX;_ylv=3?link=list&sid=396545660
+^http://answers.yahoo.com/question
-.

This should allow the 'Computers & Internet' page to be crawled, and also allow the associated questions to be crawled, but wouldn't traverse beyond that. In order to be sure, you would also want to limit your crawl depth to 2 or 3.

Re: No search results

2009-11-07T13:02:00Z

By any chance is your WAR file (used for the web server) from a slightly different version of Nutch than your crawler? I have seen that this results in the Nutch page showing up, but no results are listed. Another possibility is that you are not launching your search engine from the correct directory; I believe that you are supposed to launch it from the parent directory of you crawler results directory.

no results for local file crawls?

2009-11-07T10:16:41Z

Hello,

I'm trying to crawl the local filesystem. It appears that the crawl is successful, but later searches don't display the content. During the crawl, I see the following:

...
fetching file:///c:/test/test.txt
fetching http://www.cnn.com/
...

I know from this that it is finding the file (otherwise I would get a 404 error), and I know that the protocol-file plugin is configured (otherwise I would get a protocol not found error).My test file contains "Hello World!", but when I query on 'world', I'll I get is the CCN page in the results.

Anyone have any idea as to what I'm doing wrong? (I've tried this with 2 different 1.1 nightly builds; one from last week and a different one from September. Also, I'm running in a CygWin environment.)

Thanks,
John

Re: Hadoop wants to do whoami?

2009-11-07T05:02:34Z

On Fri, Nov 6, 2009 at 11:44 PM, Ken Krugler
<kkrugler_lists@...> wrote:

> Normally it works fine, but it will fail if you don't have swap space
> allocated because that's factored into the free space calc when the fork
> happens.
>
> What's the swap space setup for your VPS setup?

There's no swap space.

--
http://www.linkedin.com/in/paultomblin
http://careers.stackoverflow.com/ptomblin

Re: Distributed search, is there a better method?

2009-11-07T03:29:56Z

Hi,

Generating from a 100K crawlDB should be quite fast. Have you checked that
the IP resolution is turned off? Do you have any special URL filters that
could take a lot of time to process? Generating and merging tend to take
more and more time as the crawlDB grows but this should not be too much of
an issue at your scale.

Could you dump the stats of your crawlDB and tell us how long the generation
step takes?

> One problem I've run into so far is the amount of time the generate command
> increases with each iteration. The only item that really seems to grow out
> of control is the unfetched URLs, which is expected with such a small
> sample
> of web pages, but it doesn't make sense to me as to why it would take so
> long to generate a list of 1000 urls to fetch out of a list of 100k. Those
> are small numbers in terms of database and computing in general.
>
>
Julien
--
DigitalPebble Ltd
http://www.digitalpebble.com

Re: Hadoop wants to do whoami?

2009-11-07T02:43:09Z

if you are running under windows try to run the crawler under cygwin.

On Fri, 2009-11-06 at 20:21 -0500, Paul Tomblin wrote:

> I'm trying to move my crawler from a shared hosting environment (where
> it kept getting killed off for using too much memory) to a VPS. But
> on the new host, I'm getting the following exception:
>
> [ WARN] 01:15:17 (FileSystem.java:<init>:1440)
> uri=file:///
>
> javax.security.auth.login.LoginException: Login failed: Cannot run
> program "whoami": java.io.IOException: error=12, Cannot allocate
> memory
> at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:250)
> at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:275)
> at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:257)
> at org.apache.hadoop.security.UserGroupInformation.login(UserGroupInformation.java:67)
> at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1438)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1376)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:215)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:120)
> at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:319)
> at org.apache.hadoop.mapred.FileInputFormat.addInputPath(FileInputFormat.java:313)
> at org.apache.nutch.crawl.Injector.inject(Injector.java:152)
> at com.lucidityworks.nutch.crawler.Crawler.crawlIt(Crawler.java:407)
> at com.lucidityworks.nutch.crawler.Crawler.crawlSite(Crawler.java:381)
> at com.lucidityworks.nutch.crawler.Crawler.crawlCategory(Crawler.java:255)
> at com.lucidityworks.nutch.crawler.Crawler.crawl(Crawler.java:166)
> at com.lucidityworks.nutch.crawler.Crawler.main(Crawler.java:724)
>
> What is going on here?
>

Re: updatedb is talking long long time

2009-11-06T22:38:55Z

Hi

I have tried ur suggestion of lowering db.max.outlinks.per.page to a
smaller number. I could not reparse the segment as the segment was
already parsed... I tried modifying some other variables such as
java_heap memory and mapreduce_child_opts values... modifying these
values triggered some exceptions.

Therefore i have generated a new segment (considering maybe something
is wrong with the previous segment). and redoing the fetching process.
Once this is complete then i will try to do updatedb again and see if
that works...

Mathan
On Fri, Nov 6, 2009 at 5:40 AM, Julien Nioche
<lists.digitalpebble@...> wrote:

> Hello Kalaimathan,
>
> Any luck with your updateDB? I would be curious to know if the tricks I
> suggested worked.
>
> J.
> --
> DigitalPebble Ltd
> http://www.digitalpebble.com
>
> 2009/11/3 Julien Nioche <lists.digitalpebble@...>
>
>> OK. Try reparsing and set a lower value to *db.max.outlinks.per.page*. I
>> am pretty sure that you are running out of memory because of the inlinks
>> which are stored in RAM.
>> Applying the patch NUTCH-702 would also help.
>>
>> I have modified the CrawlDBReducer and added another parameter *db
>> .fetch.links.max* :
>>
>> * switch (datum.getStatus()) { // collect other info
>> case CrawlDatum.STATUS_LINKED:
>> if (maxLinks!=-1 && linked.size()>= maxLinks) break;
>> *
>> where maxLinks is a variable which I initialize from the configure() method
>>
>> * maxLinks = job.getInt("db.fetch.links.max", -1);*
>>
>> I have not tried *db.max.outlinks.per.page *at all but am pretty sure that
>> *db.fetch.links.max* works fine.
>>
>> There is also a parameter *db.max.inlinks* but it affects only the
>> LinkDBMerger
>>
>> Let us know if that fixes the problem
>>
>> Julien
>> --
>> DigitalPebble Ltd
>> http://www.digitalpebble.com
>>
>>
>> 2009/11/3 Kalaimathan Mahenthiran <mathan55@...>
>>
>>> I can see that its running out of ram because... before starting
>>> updatedb process i have approximately 7.7gb left on the system and as
>>> soon as this starts running for some time.. the ram comes to ~48
>>> bytes...
>>>
>>> definitely its clogging all the ram space...
>>>
>>> i specified the heap size to be 9 gb.. in the hadoop-site.xml like below
>>> <property>
>>> <name>mapred.child.java.opts</name>
>>> <value>-Xmx9096m -XX: -UseGCOverheadLimit</value>
>>> </propery>
>>>
>>> I have attached a screenshot of the jconsole view of the updatedb
>>> process...
>>> From jconsole i can see that cpu is not getting used at all.. its only
>>> being used 0.3~.5%.
>>>
>>> The system i'm using should not be a limitation because its an amd
>>> 64bit quad core processor with 8gbs of Ram and 1.5 Terabytes of hard
>>> disk space...
>>>
>>> Thanks again for all the help
>>>
>>> On Tue, Nov 3, 2009 at 4:15 AM, Julien Nioche
>>> <lists.digitalpebble@...> wrote:
>>> > OK. What heapsize did you specify for this job? Could it be that you are
>>> > running out of ram and GCing a lot? Still it should not take THAT long
>>> > Can you see some variations in the stacktraces or are they always
>>> pointing
>>> > at the same things?
>>> > The operations on the metadata take an awful lot of time, which I why I
>>> did
>>> > NUTCH-702, however that does not explain why processing a dataset this
>>> size
>>> > takes 20 days.
>>> >
>>> > J.
>>> >
>>> > 2009/11/3 Kalaimathan Mahenthiran <mathan55@...>
>>> >
>>> >> I have lot of space left on the /tmp . I don't have separate partition
>>> >> for /tmp... i have a folder called /tmp... There is lot of space
>>> >> left.. close to 1.3Terabytes...
>>> >>
>>> >> 1.4T 55G 1.3T 5% /
>>> >> tmpfs 3.8G 0 3.8G 0% /lib/init/rw
>>> >> varrun 3.8G 120K 3.8G 1% /var/run
>>> >> varlock 3.8G 0 3.8G 0% /var/lock
>>> >> udev 3.8G 152K 3.8G 1% /dev
>>> >> tmpfs 3.8G 0 3.8G 0% /dev/shm
>>> >> lrm 3.8G 2.5M 3.8G 1%
>>> >> /lib/modules/2.6.28-15-server/volatile
>>> >> /dev/sda5 228M 29M 187M 14% /boot
>>> >> /dev/sr0 388K 388K 0 100% /media/cdrom0
>>> >>
>>> >> I also noticed that /tmp/hadoop-root directory is 6.8 Gb...
>>> >>
>>> >> I have attached the jstack of the process that is doing the update....
>>> >> below
>>> >>
>>> >> 2009-11-02 19:11:54
>>> >> Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed
>>> mode):
>>> >>
>>> >> "Attach Listener" daemon prio=10 tid=0x0000000041bb1000 nid=0xd3b
>>> >> waiting on condition [0x0000000000000000]
>>> >> java.lang.Thread.State: RUNNABLE
>>> >>
>>> >> "Comm thread for attempt_local_0001_r_000000_0" daemon prio=10
>>> >> tid=0x00007f3ff4002800 nid=0x6b8f waiting on condition
>>> >> [0x00007f4000e97000]
>>> >> java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> >> at java.lang.Thread.sleep(Native Method)
>>> >> at org.apache.hadoop.mapred.Task$1.run(Task.java:403)
>>> >> at java.lang.Thread.run(Thread.java:619)
>>> >>
>>> >> "Thread-12" prio=10 tid=0x0000000041b37800 nid=0x25f3 runnable
>>> >> [0x00007f4000f98000]
>>> >> java.lang.Thread.State: RUNNABLE
>>> >> at java.lang.Byte.hashCode(Byte.java:394)
>>> >> at
>>> >> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:882)
>>> >> at
>>> >>
>>> org.apache.hadoop.io.AbstractMapWritable.addToMap(AbstractMapWritable.java:78)
>>> >> - locked <0x00007f47ef4d9310> (a
>>> org.apache.hadoop.io.MapWritable)
>>> >> at
>>> >>
>>> org.apache.hadoop.io.AbstractMapWritable.<init>(AbstractMapWritable.java:128)
>>> >> at org.apache.hadoop.io.MapWritable.<init>(MapWritable.java:42)
>>> >> at org.apache.hadoop.io.MapWritable.<init>(MapWritable.java:52)
>>> >> at org.apache.nutch.crawl.CrawlDatum.set(CrawlDatum.java:321)
>>> >> at
>>> >> org.apache.nutch.crawl.CrawlDbReducer.reduce(CrawlDbReducer.java:73)
>>> >> at
>>> >> org.apache.nutch.crawl.CrawlDbReducer.reduce(CrawlDbReducer.java:35)
>>> >> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
>>> >> at
>>> >>
>>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:170)
>>> >>
>>> >> "Low Memory Detector" daemon prio=10 tid=0x00007f3ffc004000 nid=0x25d0
>>> >> runnable [0x0000000000000000]
>>> >> java.lang.Thread.State: RUNNABLE
>>> >>
>>> >> "CompilerThread1" daemon prio=10 tid=0x00007f3ffc001000 nid=0x25cf
>>> >> waiting on condition [0x0000000000000000]
>>> >> java.lang.Thread.State: RUNNABLE
>>> >>
>>> >> "CompilerThread0" daemon prio=10 tid=0x00000000417be800 nid=0x25ce
>>> >> waiting on condition [0x0000000000000000]
>>> >> java.lang.Thread.State: RUNNABLE
>>> >>
>>> >> "Signal Dispatcher" daemon prio=10 tid=0x00000000417bc800 nid=0x25cd
>>> >> runnable [0x0000000000000000]
>>> >> java.lang.Thread.State: RUNNABLE
>>> >>
>>> >> "Finalizer" daemon prio=10 tid=0x000000004179e000 nid=0x25cc in
>>> >> Object.wait() [0x00007f40016f7000]
>>> >> java.lang.Thread.State: WAITING (on object monitor)
>>> >> at java.lang.Object.wait(Native Method)
>>> >> - waiting on <0x00007f400f63e6c0> (a
>>> >> java.lang.ref.ReferenceQueue$Lock)
>>> >> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>>> >> - locked <0x00007f400f63e6c0> (a
>>> java.lang.ref.ReferenceQueue$Lock)
>>> >> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>>> >> at
>>> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>>> >>
>>> >> "Reference Handler" daemon prio=10 tid=0x0000000041797000 nid=0x25cb
>>> >> in Object.wait() [0x00007f40017f8000]
>>> >> java.lang.Thread.State: WAITING (on object monitor)
>>> >> at java.lang.Object.wait(Native Method)
>>> >> - waiting on <0x00007f400f63e6f8> (a
>>> java.lang.ref.Reference$Lock)
>>> >> at java.lang.Object.wait(Object.java:485)
>>> >> at
>>> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>>> >> - locked <0x00007f400f63e6f8> (a java.lang.ref.Reference$Lock)
>>> >>
>>> >> "main" prio=10 tid=0x0000000041734000 nid=0x25c5 waiting on condition
>>> >> [0x00007f49d75c2000]
>>> >> java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> >> at java.lang.Thread.sleep(Native Method)
>>> >> at
>>> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1152)
>>> >> at org.apache.nutch.crawl.CrawlDb.update(CrawlDb.java:94)
>>> >> at org.apache.nutch.crawl.CrawlDb.run(CrawlDb.java:189)
>>> >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>> >> at org.apache.nutch.crawl.CrawlDb.main(CrawlDb.java:150)
>>> >>
>>> >> "VM Thread" prio=10 tid=0x0000000041790000 nid=0x25ca runnable
>>> >>
>>> >> "GC task thread#0 (ParallelGC)" prio=10 tid=0x000000004173e000
>>> >> nid=0x25c6 runnable
>>> >>
>>> >> "GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000041740000
>>> >> nid=0x25c7 runnable
>>> >>
>>> >> "GC task thread#2 (ParallelGC)" prio=10 tid=0x0000000041742000
>>> >> nid=0x25c8 runnable
>>> >>
>>> >> "GC task thread#3 (ParallelGC)" prio=10 tid=0x0000000041744000
>>> >> nid=0x25c9 runnable
>>> >>
>>> >> "VM Periodic Task Thread" prio=10 tid=0x00007f3ffc006800 nid=0x25d1
>>> >> waiting on condition
>>> >>
>>> >> JNI global references: 907
>>> >>
>>> >>
>>> >>
>>> >> Any help related to this would be really helpful...
>>> >>
>>> >> On Mon, Nov 2, 2009 at 3:56 PM, Julien Nioche
>>> >> <lists.digitalpebble@...> wrote:
>>> >> > Hi again
>>> >> >
>>> >> >
>>> >> >> i know the process is not stuck.. and the process is running because
>>> i
>>> >> >> turned on the hadoop logs and i can see logs being written to it...
>>> >> >> I'm not sure how to check if the task is completely stuck or not...
>>> >> >>
>>> >> >
>>> >> > run jps to identify the process id then *jstack id* several times to
>>> see
>>> >> if
>>> >> > it is blocked at the same place
>>> >> >
>>> >> > how much space do you have left on the partition where /tmp is
>>> mounted?
>>> >> >
>>> >> > J.
>>> >> >
>>> >> >
>>> >> >
>>> >> >> Below is the sample log as i'm sending this email.... Its been on
>>> the
>>> >> >> updatedb process for the last 19 days and the it has been generating
>>> >> >> debug logs similar to this........
>>> >> >>
>>> >> >> Has anyone else has this same issue before...
>>> >> >>
>>> >> >>
>>> >> >> 2009-11-02 13:34:21,112 DEBUG mapred.Counters - Creating group
>>> >> >> org.apache.hadoop.mapred.Task$FileSystemCounter with bundle
>>> >> >> 2009-11-02 13:34:21,112 DEBUG mapred.Counters - Adding LOCAL_READ
>>> >> >> 2009-11-02 13:34:21,112 DEBUG mapred.Counters - Adding LOCAL_WRITE
>>> >> >> 2009-11-02 13:34:21,112 DEBUG mapred.Counters - Creating group
>>> >> >> org.apache.hadoop.mapred.Task$Counter with bundle
>>> >> >> 2009-11-02 13:34:21,112 DEBUG mapred.Counters - Adding
>>> >> >> COMBINE_OUTPUT_RECORDS
>>> >> >> 2009-11-02 13:34:21,112 DEBUG mapred.Counters - Adding
>>> MAP_INPUT_RECORDS
>>> >> >> 2009-11-02 13:34:21,113 DEBUG mapred.Counters - Adding
>>> MAP_OUTPUT_BYTES
>>> >> >> 2009-11-02 13:34:21,113 DEBUG mapred.Counters - Adding
>>> MAP_INPUT_BYTES
>>> >> >> 2009-11-02 13:34:21,113 DEBUG mapred.Counters - Adding
>>> >> MAP_OUTPUT_RECORDS
>>> >> >> 2009-11-02 13:34:21,113 DEBUG mapred.Counters - Adding
>>> >> >> COMBINE_INPUT_RECORDS
>>> >> >> 2009-11-02 13:34:21,643 INFO mapred.JobClient - map 93% reduce 0%
>>> >> >> 2009-11-02 13:34:22,121 INFO mapred.MapTask - Spilling map output:
>>> >> >> record full = true
>>> >> >> 2009-11-02 13:34:22,121 INFO mapred.MapTask - bufstart = 10420198;
>>> >> >> bufend = 13893589; bufvoid = 99614720
>>> >> >> 2009-11-02 13:34:22,121 INFO mapred.MapTask - kvstart = 131070;
>>> kvend
>>> >> >> = 65533; length = 327680
>>> >> >> 2009-11-02 13:34:22,427 INFO mapred.MapTask - Finished spill 3
>>> >> >> 2009-11-02 13:34:23,301 INFO mapred.MapTask - Starting flush of map
>>> >> output
>>> >> >> 2009-11-02 13:34:23,384 INFO mapred.MapTask - Finished spill 4
>>> >> >> 2009-11-02 13:34:23,390 DEBUG mapred.MapTask -
>>> >> >> MapId=attempt_local_0001_m_000003_0 Reducer=0Spill =0(0,224, 228)
>>> >> >> 2009-11-02 13:34:23,390 DEBUG mapred.MapTask -
>>> >> >> MapId=attempt_local_0001_m_000003_0 Reducer=0Spill =1(0,242, 246)
>>> >> >> 2009-11-02 13:34:23,390 DEBUG mapred.MapTask -
>>> >> >> MapId=attempt_local_0001_m_000003_0 Reducer=0Spill =2(0,242, 246)
>>> >> >> 2009-11-02 13:34:23,390 DEBUG mapred.MapTask -
>>> >> >> MapId=attempt_local_0001_m_000003_0 Reducer=0Spill =3(0,242, 246)
>>> >> >> 2009-11-02 13:34:23,390 DEBUG mapred.MapTask -
>>> >> >> MapId=attempt_local_0001_m_000003_0 Reducer=0Spill =4(0,242, 246)
>>> >> >> 2009-11-02 13:34:23,390 INFO mapred.Merger - Merging 5 sorted
>>> segments
>>> >> >> 2009-11-02 13:34:23,392 INFO mapred.Merger - Down to the last
>>> >> >> merge-pass, with 5 segments left of total size: 1192 bytes
>>> >> >> 2009-11-02 13:34:23,393 INFO mapred.MapTask - Index: (0, 354, 358)
>>> >> >> 2009-11-02 13:34:23,394 INFO mapred.TaskRunner -
>>> >> >> Task:attempt_local_0001_m_000003_0 is done. And is in the process of
>>> >> >> commiting
>>> >> >> 2009-11-02 13:34:23,395 DEBUG mapred.TaskRunner -
>>> >> >> attempt_local_0001_m_000003_0 Progress/ping thread exiting since it
>>> >> >> got interrupted
>>> >> >> 2009-11-02 13:34:23,395 INFO mapred.LocalJobRunner -
>>> >> >>
>>> >> >>
>>> >>
>>> file:/opt/tsweb/nutch-1.0/newHyperseekCrawl/db/current/part-00000/data:100663296+33554432
>>> >> >> 2009-11-02 13:34:23,396 DEBUG mapred.Counters - Creating group
>>> >> >> org.apache.hadoop.mapred.Task$FileSystemCounter with bundle
>>> >> >> 2009-11-02 13:34:23,396 DEBUG mapred.Counters - Adding LOCAL_READ
>>> >> >> 2009-11-02 13:34:23,396 DEBUG mapred.Counters - Adding LOCAL_WRITE
>>> >> >> 2009-11-02 13:34:23,396 DEBUG mapred.Counters - Creating group
>>> >> >> org.apache.hadoop.mapred.Task$Counter with bundle
>>> >> >> 2009-11-02 13:34:23,396 DEBUG mapred.Counters - Adding
>>> >> >> COMBINE_OUTPUT_RECORDS
>>> >> >> 2009-11-02 13:34:23,396 DEBUG mapred.Counters - Adding
>>> MAP_INPUT_RECORDS
>>> >> >> 2009-11-02 13:34:23,396 DEBUG mapred.Counters - Adding
>>> MAP_OUTPUT_BYTES
>>> >> >> 2009-11-02 13:34:23,396 DEBUG mapred.Counters - Adding
>>> MAP_INPUT_BYTES
>>> >> >> 2009-11-02 13:34:23,396 DEBUG mapred.Counters - Adding
>>> >> MAP_OUTPUT_RECORDS
>>> >> >> 2009-11-02 13:34:23,396 DEBUG mapred.Counters - Adding
>>> >> >> COMBINE_INPUT_RECORDS
>>> >> >> 2009-11-02 13:34:23,397 INFO mapred.TaskRunner - Task
>>> >> >> 'attempt_local_0001_m_000003_0' done.
>>> >> >> 2009-11-02 13:34:23,397 DEBUG mapred.SortedRanges - currentIndex 0
>>> 0:0
>>> >> >> 2009-11-02 13:34:23,397 DEBUG conf.Configuration -
>>> >> >> java.io.IOException: config(config)
>>> >> >> at
>>> >> >> org.apache.hadoop.conf.Configuration.<init>(Configuration.java:192)
>>> >> >> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:139)
>>> >> >> at
>>> >> >>
>>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:132)
>>> >> >>
>>> >> >> 2009-11-02 13:34:23,398 DEBUG mapred.MapTask - Writing local split
>>> to
>>> >> >> /tmp/hadoop-root/mapred/local/localRunner/split.dta
>>> >> >> 2009-11-02 13:34:23,451 DEBUG mapred.TaskRunner -
>>> >> >> attempt_local_0001_m_000004_0 Progress/ping thread started
>>> >> >> 2009-11-02 13:34:23,452 INFO mapred.MapTask - numReduceTasks: 1
>>> >> >> 2009-11-02 13:34:23,453 INFO mapred.MapTask - io.sort.mb = 100
>>> >> >> 2009-11-02 13:34:23,644 INFO mapred.JobClient - map 100% reduce 0%
>>> >> >>
>>> >> >> Mathan
>>> >> >> On Mon, Nov 2, 2009 at 4:11 AM, Andrzej Bialecki <ab@...>
>>> wrote:
>>> >> >> > Kalaimathan Mahenthiran wrote:
>>> >> >> >>
>>> >> >> >> I forgot to add the detail...
>>> >> >> >>
>>> >> >> >> The segment i'm trying to do updatedb on has 1.3 millions urls
>>> >> fetched
>>> >> >> >> and 1.08 million urls parsed..
>>> >> >> >>
>>> >> >> >> Any help related to this would be appreciated...
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> On Sun, Nov 1, 2009 at 11:53 PM, Kalaimathan Mahenthiran
>>> >> >> >> <mathan55@...> wrote:
>>> >> >> >>>
>>> >> >> >>> hi everyone
>>> >> >> >>>
>>> >> >> >>> I'm using nutch 1.0. I have fetched successfully and currently
>>> on
>>> >> the
>>> >> >> >>> updatedb process. I'm doing updatedb and its taking so long. I
>>> don't
>>> >> >> >>> know why its taking this long. I have a new machine with quad
>>> core
>>> >> >> >>> processor and 8 gb of ram.
>>> >> >> >>>
>>> >> >> >>> I believe this system is really good in terms of processing
>>> power. I
>>> >> >> >>> don't think processing power is the problem here. I noticed that
>>> all
>>> >> >> >>> the ram is getting using up. close to 7.7gb by the updatedb
>>> process.
>>> >> >> >>> The computer is becoming is really slow.
>>> >> >> >>>
>>> >> >> >>> The updatedb process has been running for the last 19 days
>>> >> continually
>>> >> >> >>> with the message merging segment data into db.. Does anyone know
>>> why
>>> >> >> >>> its taking so long... Is there any configuration setting i can
>>> do to
>>> >> >> >>> increase the speed of the updatedb process...
>>> >> >> >
>>> >> >> > First, this process normally takes just a few minutes, depending
>>> on
>>> >> the
>>> >> >> > hardware, and not several days - so something is wrong.
>>> >> >> >
>>> >> >> > * do you run this in "local" or pseudo-distributed mode (i.e.
>>> running
>>> >> a
>>> >> >> real
>>> >> >> > jobtracker and tasktracker?) Try the pseudo-distributed mode,
>>> because
>>> >> >> then
>>> >> >> > you can monitor the progress in the web UI.
>>> >> >> >
>>> >> >> > * how many reduce tasks do you have? with large updates it helps
>>> if
>>> >> you
>>> >> >> run
>>> >> >> >> 1 reducer, to split the final sorting.
>>> >> >> >
>>> >> >> > * if the task appears to be completely stuck, please generate a
>>> thread
>>> >> >> dump
>>> >> >> > (kill -SIGQUIT) and see where it's stuck. This could be related to
>>> >> >> > urlfilter-regex or urlnormalizer-regex - you can identify if these
>>> are
>>> >> >> > problematic by removing them from the config and re-running the
>>> >> >> operation.
>>> >> >> >
>>> >> >> > * minor issue - when specifying the path names of segments and
>>> >> crawldb,
>>> >> >> do
>>> >> >> > NOT append the trailing slash - it's not harmful in this
>>> particular
>>> >> case,
>>> >> >> > but you could have a nasty surprise when doing e.g. copy / mv
>>> >> operations
>>> >> >> ...
>>> >> >> >
>>> >> >> > --
>>> >> >> > Best regards,
>>> >> >> > Andrzej Bialecki <><
>>> >> >> > ___. ___ ___ ___ _ _ __________________________________
>>> >> >> > [__ || __|__/|__||\/| Information Retrieval, Semantic Web
>>> >> >> > ___|||__|| \| || | Embedded Unix, System Integration
>>> >> >> > http://www.sigram.com Contact: info at sigram dot com
>>> >> >> >
>>> >> >> >
>>> >> >>
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> > DigitalPebble Ltd
>>> >> > http://www.digitalpebble.com
>>> >> >
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > DigitalPebble Ltd
>>> > http://www.digitalpebble.com
>>> >
>>>
>>
>>
>>
>

Re: Hadoop wants to do whoami?

2009-11-06T20:44:43Z

Hi Paul,

On Nov 6, 2009, at 5:39pm, Paul Tomblin wrote:

> Wait a second, doesn't Linux have a fork that does copy on write? I
> mean, Sun OS got that around 1990 or so (at least that's when I was
> told not to bother using vfork because fork now didn't use as much
> memory), surely Linux has caught up to 15 years ago.

Normally it works fine, but it will fail if you don't have swap space
allocated because that's factored into the free space calc when the
fork happens.

What's the swap space setup for your VPS setup?

-- Ken

>
> On Fri, Nov 6, 2009 at 8:29 PM, Ken Krugler <kkrugler_lists@...
> > wrote:
>> Hi Paul,
>>
>> Hadoop uses the whoami command to find out what user it's running as.
>>
>> When running a command line tool from Java, the process running the
>> JVM gets
>> forked.
>>
>> This in turn can trigger out of memory errors if you're running
>> without
>> much/any swap or your OS doesn't support memory overcommit.
>>
>> I ran into something similar when running some Java code in a VMWare
>> environment where the instances hadn't been set up with any swap
>> space.
>>
>> See http://issues.apache.org/jira/browse/HADOOP-5059 for more
>> details.
>>
>> -- Ken
>>
>>
>> On Nov 6, 2009, at 5:21pm, Paul Tomblin wrote:
>>
>>> I'm trying to move my crawler from a shared hosting environment
>>> (where
>>> it kept getting killed off for using too much memory) to a VPS. But
>>> on the new host, I'm getting the following exception:
>>>
>>> [ WARN] 01:15:17 (FileSystem.java:<init>:1440)
>>> uri=file:///
>>>
>>> javax.security.auth.login.LoginException: Login failed: Cannot run
>>> program "whoami": java.io.IOException: error=12, Cannot allocate
>>> memory
>>> at
>>> org
>>> .apache
>>> .hadoop
>>> .security
>>> .UnixUserGroupInformation.login(UnixUserGroupInformation.java:250)
>>> at
>>> org
>>> .apache
>>> .hadoop
>>> .security
>>> .UnixUserGroupInformation.login(UnixUserGroupInformation.java:275)
>>> at
>>> org
>>> .apache
>>> .hadoop
>>> .security
>>> .UnixUserGroupInformation.login(UnixUserGroupInformation.java:257)
>>> at
>>> org
>>> .apache
>>> .hadoop
>>> .security.UserGroupInformation.login(UserGroupInformation.java:67)
>>> at
>>> org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:
>>> 1438)
>>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:
>>> 1376)
>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:215)
>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:120)
>>> at
>>> org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:
>>> 319)
>>> at
>>> org
>>> .apache
>>> .hadoop.mapred.FileInputFormat.addInputPath(FileInputFormat.java:
>>> 313)
>>> at org.apache.nutch.crawl.Injector.inject(Injector.java:152)
>>> at
>>> com.lucidityworks.nutch.crawler.Crawler.crawlIt(Crawler.java:407)
>>> at
>>> com.lucidityworks.nutch.crawler.Crawler.crawlSite(Crawler.java:381)
>>> at
>>> com.lucidityworks.nutch.crawler.Crawler.crawlCategory(Crawler.java:
>>> 255)
>>> at
>>> com.lucidityworks.nutch.crawler.Crawler.crawl(Crawler.java:166)
>>> at com.lucidityworks.nutch.crawler.Crawler.main(Crawler.java:
>>> 724)
>>>
>>> What is going on here?
>>>
>>> --
>>> http://www.linkedin.com/in/paultomblin
>>> http://careers.stackoverflow.com/ptomblin
>>
>> --------------------------------------------
>> Ken Krugler
>> +1 530-210-6378
>> http://bixolabs.com
>> e l a s t i c w e b m i n i n g
>>
>>
>>
>>
>>
>
>
>
> --
> http://www.linkedin.com/in/paultomblin
> http://careers.stackoverflow.com/ptomblin

--------------------------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
e l a s t i c w e b m i n i n g

Distributed search, is there a better method?

2009-11-06T18:37:11Z

I was curious if anyone has a better method than what I am doing now for
distributed search.
Using one namenode and two datanodes, I am also using the namenode as the
tomcat server, and using the datanodes as the distributed search nodes.

First I generate a segment (-topN 1000)
Then fetch
Then updatedb
Then merge segements into one.
Then invertlinks

Then I do the following:
I get the stats of the crawldb, in order to get the number of URLs
I run mergesegs -slice (1/2 number of urls)

index segment 1 and copytolocal the new index to datanode 1
index segment 2 and copytolocal this new index to datanode 2

then restart nutch servers on the datanodes.

It seems to work fine, though I admit I've not gotten beyond about 30k urls
fetched with about 100k urls still unfetched.

I tried using the -slice option on the initial merge, but I found on
occasion there was no parse data in one segment, or I got an unexpected
number of segments. I'm guessing this is because updatedb needs to be run
before I can get an accurate number of URLs to do the math to get the same
number of segments as search servers.

One problem I've run into so far is the amount of time the generate command
increases with each iteration. The only item that really seems to grow out
of control is the unfetched URLs, which is expected with such a small sample
of web pages, but it doesn't make sense to me as to why it would take so
long to generate a list of 1000 urls to fetch out of a list of 100k. Those
are small numbers in terms of database and computing in general.

The next hangup I run into is the mergesegs and mergesegs -slice. Both of
these steps increase in amount of time by an extreme amount once reaching
about 100k URLs.

Is this expected or common?
Has anyone come up with a better way to go through the steps to get multiple
unique indexes to reside on the individual search server nodes?

This is purely academic for me, so there really is no time lost on my part
to change up my approach. I am also purposely using low power hardware.

Jesse

int GetRandomNumber()
{
return 4; // Chosen by fair roll of dice
// Guaranteed to be random
} // xkcd.com

Re: Hadoop wants to do whoami?

2009-11-06T17:39:07Z

Wait a second, doesn't Linux have a fork that does copy on write? I
mean, Sun OS got that around 1990 or so (at least that's when I was
told not to bother using vfork because fork now didn't use as much
memory), surely Linux has caught up to 15 years ago.

On Fri, Nov 6, 2009 at 8:29 PM, Ken Krugler <kkrugler_lists@...> wrote:

> Hi Paul,
>
> Hadoop uses the whoami command to find out what user it's running as.
>
> When running a command line tool from Java, the process running the JVM gets
> forked.
>
> This in turn can trigger out of memory errors if you're running without
> much/any swap or your OS doesn't support memory overcommit.
>
> I ran into something similar when running some Java code in a VMWare
> environment where the instances hadn't been set up with any swap space.
>
> See http://issues.apache.org/jira/browse/HADOOP-5059 for more details.
>
> -- Ken
>
>
> On Nov 6, 2009, at 5:21pm, Paul Tomblin wrote:
>
>> I'm trying to move my crawler from a shared hosting environment (where
>> it kept getting killed off for using too much memory) to a VPS. But
>> on the new host, I'm getting the following exception:
>>
>> [ WARN] 01:15:17 (FileSystem.java:<init>:1440)
>> uri=file:///
>>
>> javax.security.auth.login.LoginException: Login failed: Cannot run
>> program "whoami": java.io.IOException: error=12, Cannot allocate
>> memory
>> at
>> org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:250)
>> at
>> org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:275)
>> at
>> org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:257)
>> at
>> org.apache.hadoop.security.UserGroupInformation.login(UserGroupInformation.java:67)
>> at
>> org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1438)
>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1376)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:215)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:120)
>> at
>> org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:319)
>> at
>> org.apache.hadoop.mapred.FileInputFormat.addInputPath(FileInputFormat.java:313)
>> at org.apache.nutch.crawl.Injector.inject(Injector.java:152)
>> at com.lucidityworks.nutch.crawler.Crawler.crawlIt(Crawler.java:407)
>> at
>> com.lucidityworks.nutch.crawler.Crawler.crawlSite(Crawler.java:381)
>> at
>> com.lucidityworks.nutch.crawler.Crawler.crawlCategory(Crawler.java:255)
>> at com.lucidityworks.nutch.crawler.Crawler.crawl(Crawler.java:166)
>> at com.lucidityworks.nutch.crawler.Crawler.main(Crawler.java:724)
>>
>> What is going on here?
>>
>> --
>> http://www.linkedin.com/in/paultomblin
>> http://careers.stackoverflow.com/ptomblin
>
> --------------------------------------------
> Ken Krugler
> +1 530-210-6378
> http://bixolabs.com
> e l a s t i c w e b m i n i n g
>
>
>
>
>

--
http://www.linkedin.com/in/paultomblin
http://careers.stackoverflow.com/ptomblin

Re: Hadoop wants to do whoami?

2009-11-06T17:31:56Z

Most likely your allocated memory settings for java process is high.

Usually you get this error if
2 * Memory configured for the process > (total physical memory
available + swap space)

You can find more explanation about this error at

http://issues.apache.org/jira/browse/HADOOP-5059

HTH.

Neera

On Fri, Nov 6, 2009 at 5:21 PM, Paul Tomblin <ptomblin@...> wrote:

Re: Hadoop wants to do whoami?

2009-11-06T17:29:59Z

Hi Paul,

Hadoop uses the whoami command to find out what user it's running as.

When running a command line tool from Java, the process running the
JVM gets forked.

This in turn can trigger out of memory errors if you're running
without much/any swap or your OS doesn't support memory overcommit.

I ran into something similar when running some Java code in a VMWare
environment where the instances hadn't been set up with any swap space.

See http://issues.apache.org/jira/browse/HADOOP-5059 for more details.

-- Ken

On Nov 6, 2009, at 5:21pm, Paul Tomblin wrote:

> I'm trying to move my crawler from a shared hosting environment (where
> it kept getting killed off for using too much memory) to a VPS. But
> on the new host, I'm getting the following exception:
>
> [ WARN] 01:15:17 (FileSystem.java:<init>:1440)
> uri=file:///
>
> javax.security.auth.login.LoginException: Login failed: Cannot run
> program "whoami": java.io.IOException: error=12, Cannot allocate
> memory
> at
> org
> .apache
> .hadoop
> .security
> .UnixUserGroupInformation.login(UnixUserGroupInformation.java:250)
> at
> org
> .apache
> .hadoop
> .security
> .UnixUserGroupInformation.login(UnixUserGroupInformation.java:275)
> at
> org
> .apache
> .hadoop
> .security
> .UnixUserGroupInformation.login(UnixUserGroupInformation.java:257)
> at
> org
> .apache
> .hadoop
> .security.UserGroupInformation.login(UserGroupInformation.java:67)
> at org.apache.hadoop.fs.FileSystem$Cache
> $Key.<init>(FileSystem.java:1438)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:
> 1376)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:215)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:120)
> at
> org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:319)
> at
> org
> .apache
> .hadoop.mapred.FileInputFormat.addInputPath(FileInputFormat.java:313)
> at org.apache.nutch.crawl.Injector.inject(Injector.java:152)
> at
> com.lucidityworks.nutch.crawler.Crawler.crawlIt(Crawler.java:407)
> at
> com.lucidityworks.nutch.crawler.Crawler.crawlSite(Crawler.java:381)
> at
> com.lucidityworks.nutch.crawler.Crawler.crawlCategory(Crawler.java:
> 255)
> at com.lucidityworks.nutch.crawler.Crawler.crawl(Crawler.java:
> 166)
> at com.lucidityworks.nutch.crawler.Crawler.main(Crawler.java:
> 724)
>
> What is going on here?
>
> --
> http://www.linkedin.com/in/paultomblin
> http://careers.stackoverflow.com/ptomblin

--------------------------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
e l a s t i c w e b m i n i n g