Nabble - Nutch - Dev

Re: wrong wiki front page

2009-11-26T03:39:10Z

Issue and solutions described here : http://wiki.apache.org/httpd/HelpOnConfiguration#Default_front_page

2009/11/24 Alban Mouton <alban83@...>

Hello everybody,

I don't know if it is a known issue, but it's been like that since at least a couple of days so I figured I should tell someone. The root url for the nutch wiki http://wiki.apache.org/nutch/ doesn't redirect to http://wiki.apache.org/nutch/FrontPage ! It's annoying because that's the url given by google and the nutch website. It might be a language detection problem because I see this ugly and not very helpful page : http://wiki.apache.org/nutch/PageD%27Accueil (page d'accueil = home page in french).

Not much of a contribution for my first message here, but I hope to do more soon.

Alban Mouton

[jira] Resolved: (NUTCH-185) XMLParser is configurable xml parser plugin.

2009-11-25T19:16:39Z

[ https://issues.apache.org/jira/browse/NUTCH-185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris A. Mattmann resolved NUTCH-185.
-------------------------------------

Resolution: Won't Fix
Fix Version/s: 1.1

See comments related to NUTCH-767 in this issue's comments section. Once we address NUTCH-767, we get this functionality for free...

> XMLParser is configurable xml parser plugin.
> --------------------------------------------
>
> Key: NUTCH-185
> URL: https://issues.apache.org/jira/browse/NUTCH-185
> Project: Nutch
> Issue Type: New Feature
> Components: fetcher, indexer
> Affects Versions: 0.7.2, 0.8, 0.8.1
> Environment: OS Independent
> Reporter: Rida Benjelloun
> Assignee: Chris A. Mattmann
> Fix For: 1.1
>
> Attachments: parse-xml.patch, parse-xml.zip, parse-xml.zip
>
>
> Xml parser is configurable plugin. It use XPath and namespaces to do the mapping between the XML elements and Lucene fields.
> Informations :
> 1- Copy "xmlparser-conf.xml" to the nutch/conf dir
> 2- To index your custom XML file, you have to modify the "xmlparser-conf.xml".
> This parser uses namespaces and XPATH to parse XML content
> The config file do the mapping between the XML noeds (using XPATH) and lucene field.
> Example : <field name="dctitle" xpath="//dc:title" type="Text" boost="1.4" />
> 3- The xmlIndexerProperties encapsulate a set of fields associated to a namespace.
> If the namespace is found in the xml document, the fields represented by the namespace will be indexed.
> Example :
> <xmlIndexerProperties type="filePerDocument" namespace=" http://purl.org/dc/elements/1.1/">
> <field name="dctitle" xpath="//dc:title" type="Text" boost=" 1.4" />
> <field name="dccreator" xpath="//dc:creator" type="keyword" boost=" 1.0" />
> </xmlIndexerProperties>
> 4- It is possible to define a default namespace that will be applied when the parser
> didn't find any namespace in the document or when the namespace found in the xml document doesn't match with the namespace defined in the xmlIndexerProperties.
> Example :
> <xmlIndexerProperties type="filePerDocument" namespace="default">
> <field name="xmlcontent" xpath="//*" type="Unstored" boost="1.0" />
> </xmlIndexerProperties>

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (NUTCH-768) Upgrade Nutch 1.0 to use Hadoop 0.20

2009-11-25T13:34:39Z

[ https://issues.apache.org/jira/browse/NUTCH-768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dennis Kubes updated NUTCH-768:
-------------------------------

Attachment: NUTCH-768-1-20091125.patch

I thought I was going to be able to do this without code changes. No such luck.

There are many, many deprecations as a result of this upgrade. Anything that used the old Mapper and Reducer interfaces seems to have deprecated methods in it. The NutchBean class needed to implement the two RPC*Bean interfaces to handle changes in Hadoop RPC (that could have been a leftover from 1.0 changes but I don't think so). Also there are numerous changes to build scripts and the nutch bin script to support different hadoop jars.

There are also many new files for the conf directory as Hadoop has split out files and has new configuration files for new capabilities.

After all changes I was able to run everything in local and pseudo-distributed mode as well as test out local and distributed searching. Everything seems to work fine. After we make this upgrade I would recommend going back and updating all of the tool interfaces for the most recent APIs.

> Upgrade Nutch 1.0 to use Hadoop 0.20
> ------------------------------------
>
> Key: NUTCH-768
> URL: https://issues.apache.org/jira/browse/NUTCH-768
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 1.1
> Environment: All
> Reporter: Dennis Kubes
> Assignee: Dennis Kubes
> Fix For: 1.1
>
> Attachments: NUTCH-768-1-20091125.patch
>
>
> Upgrade Nutch 1.0 to use the Hadoop 0.20 release.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (NUTCH-772) Upgrade Nutch to use Lucene 2.9.1

2009-11-25T13:18:39Z

[ https://issues.apache.org/jira/browse/NUTCH-772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782624#action_12782624 ]

Andrzej Bialecki commented on NUTCH-772:
-----------------------------------------

Fixed in rev. 884277.

> Upgrade Nutch to use Lucene 2.9.1
> ---------------------------------
>
> Key: NUTCH-772
> URL: https://issues.apache.org/jira/browse/NUTCH-772
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 1.1
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Fix For: 1.1
>
> Attachments: lucene.patch
>
>
> Upgrade Nutch to the latest Lucene release.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Closed: (NUTCH-772) Upgrade Nutch to use Lucene 2.9.1

2009-11-25T13:18:39Z

[ https://issues.apache.org/jira/browse/NUTCH-772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrzej Bialecki closed NUTCH-772.
-----------------------------------

Resolution: Fixed
Fix Version/s: 1.1

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Closed: (NUTCH-760) Allow field mapping from nutch to solr index

2009-11-25T13:00:39Z

[ https://issues.apache.org/jira/browse/NUTCH-760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrzej Bialecki closed NUTCH-760.
-----------------------------------

Resolution: Fixed
Fix Version/s: 1.1

> Allow field mapping from nutch to solr index
> --------------------------------------------
>
> Key: NUTCH-760
> URL: https://issues.apache.org/jira/browse/NUTCH-760
> Project: Nutch
> Issue Type: Improvement
> Components: indexer
> Reporter: David Stuart
> Assignee: Andrzej Bialecki
> Fix For: 1.1
>
> Attachments: solrindex_schema.patch, solrindex_schema.patch, solrindex_schema.patch, solrindex_schema.patch
>
>
> I am using nutch to crawl sites and have combined it
> with solr pushing the nutch index using the solrindex command. I have
> set it up as specified on the wiki using the copyField url to id in the
> schema. Whilst this works fine it is stuff's up my inputs from other
> sources in solr (e.g. using the solr data import handler) as they have
> both id's and url's. I have patch that implements a nutch xml schema
> defining what basic nutch fields map to in your solr push.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (NUTCH-760) Allow field mapping from nutch to solr index

2009-11-25T13:00:39Z

[ https://issues.apache.org/jira/browse/NUTCH-760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782617#action_12782617 ]

Andrzej Bialecki commented on NUTCH-760:
-----------------------------------------

I reworked the patch to get rid of any left-overs of static Configuration, and changed the concept of "schema" (which was misleading) to "mapping" throughout the patch and class names.

This is now committed in rev. 884269 - thanks!

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Closed: (NUTCH-761) Avoid cloningCrawlDatum in CrawlDbReducer

2009-11-25T10:10:39Z

[ https://issues.apache.org/jira/browse/NUTCH-761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrzej Bialecki closed NUTCH-761.
-----------------------------------

Resolution: Fixed
Fix Version/s: 1.1
Assignee: Andrzej Bialecki

> Avoid cloningCrawlDatum in CrawlDbReducer
> ------------------------------------------
>
> Key: NUTCH-761
> URL: https://issues.apache.org/jira/browse/NUTCH-761
> Project: Nutch
> Issue Type: Improvement
> Reporter: Julien Nioche
> Assignee: Andrzej Bialecki
> Priority: Minor
> Fix For: 1.1
>
> Attachments: optiCrawlReducer.patch
>
>
> In the huge majority of cases the CrawlDbReducer gets unique CrawlData in its reduce phase and these will be the entries coming from the crawlDB and not present in the segments.
> The patch attached optimizes the reduce step by avoid an unnecessary cloning of the CrawlDatum fields when there is only one CrawlDatum in the values. This has more impact has the crawlDB gets larger, we noticed an improvement of around 25-30% in the time spent in the reduce phase.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (NUTCH-761) Avoid cloningCrawlDatum in CrawlDbReducer

2009-11-25T10:10:39Z

[ https://issues.apache.org/jira/browse/NUTCH-761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782537#action_12782537 ]

Andrzej Bialecki commented on NUTCH-761:
-----------------------------------------

I applied the patch with some changes - reverted the logic in the name of the boolean var, and applied the same method to other cases of non-multiple values. Committed in rev. 884224 - thanks!

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (NUTCH-762) Alternative Generator which can generate several segments in one parse of the crawlDB

2009-11-25T09:38:39Z

[ https://issues.apache.org/jira/browse/NUTCH-762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782524#action_12782524 ]

Andrzej Bialecki commented on NUTCH-762:
-----------------------------------------

This class offers a strict superset of the current Generator functionality. Maintaining both tools would be cumbersome and error-prone. I propose to replace Generator with MultiGenerator (under the current name Generator).

> Alternative Generator which can generate several segments in one parse of the crawlDB
> -------------------------------------------------------------------------------------
>
> Key: NUTCH-762
> URL: https://issues.apache.org/jira/browse/NUTCH-762
> Project: Nutch
> Issue Type: New Feature
> Components: generator
> Affects Versions: 1.0.0
> Reporter: Julien Nioche
> Attachments: NUTCH-762-MultiGenerator.patch
>
>
> When using Nutch on a large scale (e.g. billions of URLs), the operations related to the crawlDB (generate - update) tend to take the biggest part of the time. One solution is to limit such operations to a minimum by generating several fetchlists in one parse of the crawlDB then update the Db only once on several segments. The existing Generator allows several successive runs by generating a copy of the crawlDB and marking the URLs to be fetched. In practice this approach does not work well as we need to read the whole crawlDB as many time as we generate a segment.
> The patch attached contains an implementation of a MultiGenerator which can generate several fetchlists by reading the crawlDB only once. The MultiGenerator differs from the Generator in other aspects:
> * can filter the URLs by score
> * normalisation is optional
> * IP resolution is done ONLY on the entries which have been selected for fetching (during the partitioning). Running the IP resolution on the whole crawlDb is too slow to be usable on a large scale
> * can max the number of URLs per host or domain (but not by IP)
> * can choose to partition by host, domain or IP
> Typically the same unit (e.g. domain) would be used for maxing the URLs and for partitioning; however as we can't count the max number of URLs by IP another unit must be chosen while partitioning by IP.
> We found that using a filter on the score can dramatically improve the performance as this reduces the amount of data being sent to the reducers.
> The MultiGenerator is called via : nutch org.apache.nutch.crawl.MultiGenerator ...
> with the following options :
> MultiGenerator <crawldb> <segments_dir> [-force] [-topN N] [-numFetchers numFetchers] [-adddays numDays] [-noFilter] [-noNorm] [-maxNumSegments num]
> where most parameters are similar to the default Generator - apart from :
> -noNorm (explicit)
> -topN : max number of URLs per segment
> -maxNumSegments : the actual number of segments generated could be less than the max value select e.g. not enough URLs are available for fetching and fit in less segments
> Please give it a try and less me know what you think of it
> Julien Nioche
> http://www.digitalpebble.com
>

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (NUTCH-753) Prevent new Fetcher to retrieve the robots twice

2009-11-25T09:22:39Z

[ https://issues.apache.org/jira/browse/NUTCH-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782516#action_12782516 ]

Andrzej Bialecki commented on NUTCH-753:
-----------------------------------------

Fixed in rev. 884203 - thanks!

> Prevent new Fetcher to retrieve the robots twice
> ------------------------------------------------
>
> Key: NUTCH-753
> URL: https://issues.apache.org/jira/browse/NUTCH-753
> Project: Nutch
> Issue Type: Improvement
> Components: fetcher
> Affects Versions: 1.0.0
> Reporter: Julien Nioche
> Assignee: Andrzej Bialecki
> Fix For: 1.1
>
> Attachments: NUTCH-753.patch
>
>
> The new Fetcher which is now used by default handles the robots file directly instead of relying on the protocol. The options Protocol.CHECK_BLOCKING and Protocol.CHECK_ROBOTS are set to false to prevent fetching the robots.txt twice (in Fetcher + in protocol), which avoids calling robots.isAllowed. However in practice the robots file is still fetched as there is a call to robots.getCrawlDelay() a bit further which is not covered by the if (Protocol.CHECK_ROBOTS).

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Closed: (NUTCH-753) Prevent new Fetcher to retrieve the robots twice

2009-11-25T09:22:39Z

[ https://issues.apache.org/jira/browse/NUTCH-753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrzej Bialecki closed NUTCH-753.
-----------------------------------

Resolution: Fixed
Fix Version/s: 1.1
Assignee: Andrzej Bialecki

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Closed: (NUTCH-773) some minor bugs in AbstractFetchSchedule.java

2009-11-25T09:12:39Z

[ https://issues.apache.org/jira/browse/NUTCH-773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrzej Bialecki closed NUTCH-773.
-----------------------------------

Resolution: Fixed
Assignee: Andrzej Bialecki

> some minor bugs in AbstractFetchSchedule.java
> ---------------------------------------------
>
> Key: NUTCH-773
> URL: https://issues.apache.org/jira/browse/NUTCH-773
> Project: Nutch
> Issue Type: Bug
> Components: fetcher, generator
> Affects Versions: 1.0.0
> Reporter: Reinhard Schwab
> Assignee: Andrzej Bialecki
> Priority: Minor
> Fix For: 1.1
>
> Attachments: NUTCH-773.patch
>
>
> fixes some minor trivial bugs in AbstractFetchSchedule.java

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (NUTCH-773) some minor bugs in AbstractFetchSchedule.java

2009-11-25T09:12:39Z

[ https://issues.apache.org/jira/browse/NUTCH-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782509#action_12782509 ]

Andrzej Bialecki commented on NUTCH-773:
-----------------------------------------

That was a nasty bug - fixed in rev. 884198. Thanks!

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Re: svn commit: r884075 - /lucene/nutch/trunk/src/java/org/apache/nutch/indexer/solr/SolrIndexer.java

2009-11-25T08:01:44Z

Thanks
On 25 November 2009 at 16:58 Andrzej Bialecki <ab@...> wrote:

> david.stuart@... wrote:
> > While you are doing changes and commits in this area I have been
> > waiting for this patch https://issues.apache.org/jira/browse/NUTCH-760
> > of mine to be incorporated for a while now. Is it possible it get it in??
>
> It's on my agenda - I'll apply the patch either today or tomorrow, time
> permitting.
>
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>

[jira] Updated: (NUTCH-760) Allow field mapping from nutch to solr index

2009-11-25T07:59:39Z

[ https://issues.apache.org/jira/browse/NUTCH-760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrzej Bialecki updated NUTCH-760:
------------------------------------

Assignee: Andrzej Bialecki

> Allow field mapping from nutch to solr index
> --------------------------------------------
>
> Key: NUTCH-760
> URL: https://issues.apache.org/jira/browse/NUTCH-760
> Project: Nutch
> Issue Type: Improvement
> Components: indexer
> Reporter: David Stuart
> Assignee: Andrzej Bialecki
> Attachments: solrindex_schema.patch, solrindex_schema.patch, solrindex_schema.patch, solrindex_schema.patch
>
>
> I am using nutch to crawl sites and have combined it
> with solr pushing the nutch index using the solrindex command. I have
> set it up as specified on the wiki using the copyField url to id in the
> schema. Whilst this works fine it is stuff's up my inputs from other
> sources in solr (e.g. using the solr data import handler) as they have
> both id's and url's. I have patch that implements a nutch xml schema
> defining what basic nutch fields map to in your solr push.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Re: svn commit: r884075 - /lucene/nutch/trunk/src/java/org/apache/nutch/indexer/solr/SolrIndexer.java

2009-11-25T07:58:21Z

david.stuart@... wrote:
> While you are doing changes and commits in this area I have been
> waiting for this patch https://issues.apache.org/jira/browse/NUTCH-760
> of mine to be incorporated for a while now. Is it possible it get it in??

It's on my agenda - I'll apply the patch either today or tomorrow, time
permitting.

--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com

Re: svn commit: r884075 - /lucene/nutch/trunk/src/java/org/apache/nutch/indexer/solr/SolrIndexer.java

2009-11-25T07:14:00Z

While you are doing changes and commits in this area I have been waiting for this patch https://issues.apache.org/jira/browse/NUTCH-760 of mine to be incorporated for a while now. Is it possible it get it in??

Regards,

Dave

On 25 November 2009 at 14:36 Dennis Kubes <kubes@...> wrote:

> Oops. Sorry about that.
>
> ab@... wrote:
> > Author: ab
> > Date: Wed Nov 25 12:44:34 2009
> > New Revision: 884075
> >
> > URL: http://svn.apache.org/viewvc?rev=884075&view=rev
> > Log:
> > Change access from private to public - this fixes Crawl.java breakage.
> >
> > Modified:
> > lucene/nutch/trunk/src/java/org/apache/nutch/indexer/solr/SolrIndexer.java
> >
> > Modified: lucene/nutch/trunk/src/java/org/apache/nutch/indexer/solr/SolrIndexer.java
> > URL: http://svn.apache.org/viewvc/lucene/nutch/trunk/src/java/org/apache/nutch/indexer/solr/SolrIndexer.java?rev=884075&r1=884074&r2=884075&view=diff
> > ==============================================================================
> > --- lucene/nutch/trunk/src/java/org/apache/nutch/indexer/solr/SolrIndexer.java (original)
> > +++ lucene/nutch/trunk/src/java/org/apache/nutch/indexer/solr/SolrIndexer.java Wed Nov 25 12:44:34 2009
> > @@ -50,7 +50,7 @@
> > super(conf);
> > }
> >
> > - private void indexSolr(String solrUrl, Path crawlDb, Path linkDb,
> > + public void indexSolr(String solrUrl, Path crawlDb, Path linkDb,
> > List<Path> segments) throws IOException {
> > LOG.info("SolrIndexer: starting");
> >
> >
> >

[jira] Updated: (NUTCH-773) some minor bugs in AbstractFetchSchedule.java

2009-11-25T06:19:39Z

[ https://issues.apache.org/jira/browse/NUTCH-773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Reinhard Schwab updated NUTCH-773:
----------------------------------

Patch Info: [Patch Available]

> some minor bugs in AbstractFetchSchedule.java
> ---------------------------------------------
>
> Key: NUTCH-773
> URL: https://issues.apache.org/jira/browse/NUTCH-773
> Project: Nutch
> Issue Type: Bug
> Components: fetcher, generator
> Affects Versions: 1.0.0
> Reporter: Reinhard Schwab
> Priority: Minor
> Fix For: 1.1
>
> Attachments: NUTCH-773.patch
>
>
> fixes some minor trivial bugs in AbstractFetchSchedule.java

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (NUTCH-773) some minor bugs in AbstractFetchSchedule.java

2009-11-25T06:19:39Z

> some minor bugs in AbstractFetchSchedule.java
> ---------------------------------------------
>
> Key: NUTCH-773
> URL: https://issues.apache.org/jira/browse/NUTCH-773
> Project: Nutch
> Issue Type: Bug
> Components: fetcher, generator
> Affects Versions: 1.0.0
> Reporter: Reinhard Schwab
> Priority: Minor
> Fix For: 1.1
>
> Attachments: NUTCH-773.patch
>
>
> fixes some minor trivial bugs in AbstractFetchSchedule.java

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (NUTCH-773) some minor bugs in AbstractFetchSchedule.java

2009-11-25T06:15:39Z

some minor bugs in AbstractFetchSchedule.java
---------------------------------------------

Key: NUTCH-773
URL: https://issues.apache.org/jira/browse/NUTCH-773
Project: Nutch
Issue Type: Bug
Components: fetcher, generator
Affects Versions: 1.0.0
Reporter: Reinhard Schwab
Priority: Minor
Fix For: 1.1

fixes some minor trivial bugs in AbstractFetchSchedule.java

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Re: svn commit: r884075 - /lucene/nutch/trunk/src/java/org/apache/nutch/indexer/solr/SolrIndexer.java

2009-11-25T05:36:51Z

Oops. Sorry about that.

ab@... wrote:

> Author: ab
> Date: Wed Nov 25 12:44:34 2009
> New Revision: 884075
>
> URL: http://svn.apache.org/viewvc?rev=884075&view=rev
> Log:
> Change access from private to public - this fixes Crawl.java breakage.
>
> Modified:
> lucene/nutch/trunk/src/java/org/apache/nutch/indexer/solr/SolrIndexer.java
>
> Modified: lucene/nutch/trunk/src/java/org/apache/nutch/indexer/solr/SolrIndexer.java
> URL: http://svn.apache.org/viewvc/lucene/nutch/trunk/src/java/org/apache/nutch/indexer/solr/SolrIndexer.java?rev=884075&r1=884074&r2=884075&view=diff
> ==============================================================================
> --- lucene/nutch/trunk/src/java/org/apache/nutch/indexer/solr/SolrIndexer.java (original)
> +++ lucene/nutch/trunk/src/java/org/apache/nutch/indexer/solr/SolrIndexer.java Wed Nov 25 12:44:34 2009
> @@ -50,7 +50,7 @@
> super(conf);
> }
>
> - private void indexSolr(String solrUrl, Path crawlDb, Path linkDb,
> + public void indexSolr(String solrUrl, Path crawlDb, Path linkDb,
> List<Path> segments) throws IOException {
> LOG.info("SolrIndexer: starting");
>
>
>

[jira] Updated: (NUTCH-772) Upgrade Nutch to use Lucene 2.9.1

2009-11-25T05:00:39Z

[ https://issues.apache.org/jira/browse/NUTCH-772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrzej Bialecki updated NUTCH-772:
------------------------------------

Attachment: lucene.patch

Patch to commit shortly, if no objections.

> Upgrade Nutch to use Lucene 2.9.1
> ---------------------------------
>
> Key: NUTCH-772
> URL: https://issues.apache.org/jira/browse/NUTCH-772
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 1.1
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Attachments: lucene.patch
>
>
> Upgrade Nutch to the latest Lucene release.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (NUTCH-772) Upgrade Nutch to use Lucene 2.9.1

2009-11-25T04:36:39Z

Upgrade Nutch to use Lucene 2.9.1
---------------------------------

Key: NUTCH-772
URL: https://issues.apache.org/jira/browse/NUTCH-772
Project: Nutch
Issue Type: Improvement
Affects Versions: 1.1
Reporter: Andrzej Bialecki
Assignee: Andrzej Bialecki

Upgrade Nutch to the latest Lucene release.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Re: Plugin Developement Help

2009-11-24T14:00:24Z

Sorry keep pressing

But I dont quite understanding how the metadata is passed from the parse to the index if in my
public ParseResult filter...

Do this
        Parse parse = parseResult.get(content.getUrl());
        metadata = parse.getData().getParseMeta();
        metadata.add("filter_html_data", docTrans);

Then return
return parseResult;

Is the data passed by reference into parseResult? because when I try and retrieve it in
public NutchDocument filter...

by doing
      String html_filter_data = parse.getData().getMeta("html_filter_data");
    LOG.warn(html_filter_data);
    if (html_filter_data != null){
        LOG.warn("________________________Adding filter data_______________________");
        doc.add("html_filter_data", html_filter_data);
    }
I Never reach the add because the variable html_filter_data is empty

any ideas

Thanks for you help

On 24 November 2009 at 12:27 "david.stuart@..." <david.stuart@...> wrote:

> I thought I did but I thought before I did a bin/nutch index (or solrindex) it
> would be stored somewhere it does seems to be getting to the doc.add bit which
> makes me think the variable is empty
> {code}
>     public void addIndexBackendOptions(Configuration conf) {
>     LOG.warn("+_+_You called me _+_+");
>     LuceneWriter.addFieldOptions("html_filter_data", STORE.YES,
> INDEX.UNTOKENIZED, conf);
>     }
>
>     public NutchDocument filter(NutchDocument doc, Parse parse, Text url,
> CrawlDatum datum, Inlinks inlinks) throws IndexingException {
>     LOG.warn("________________________FILTER_______________________");
>     String html_filter_data = parse.getData().getMeta("html_filter_data");
>     if (html_filter_data != null){
>         LOG.warn("________________________Adding filter
> data_______________________");
>         doc.add("html_filter_data", html_filter_data);
>     }
>     return doc;
>     }
> {code}
> On 24 November 2009 at 12:05 Andrzej Bialecki <ab@...> wrote:
>
> > david.stuart@... wrote:
> > > Hi All,
> > >
> > > I think I am just about finished my plugin (nutch 1.0) which adds extra
> > > metadata to during parsing the problem I am having is it doesn't seem to
> > > be adding the data to the system (via luke or readseg). I looked at in
> > > the wiki but it seems to be for 0.9 and the syntax looks different.
> > >
> > > {code}
> > > public ParseResult filter(Content content, ParseResult parseResult,
> > > HTMLMetaTags metaTags, DocumentFragment doc) {
> > > Metadata metadata = new Metadata();
> > > // parse the content
> > > DocumentFragment root;
> > > String docTrans;
> > > try {
> > > byte[] contentInOctets = content.getContent();
> > > String input = new String(contentInOctets);
> > > XSLTSimpleTransform DocTransform = new XSLTSimpleTransform();
> > > docTrans = DocTransform.doTransform(input);
> > > Parse parse = parseResult.get(content.getUrl());
> > > metadata = parse.getData().getParseMeta();
> > > metadata.add("filter_html_data", docTrans);
> > >
> > > } catch (Exception e) {
> > > e.printStackTrace(LogUtil.getWarnStream(LOG));
> > > }
> > >
> > > return parseResult;
> > > }
> > > {code}
> >
> > Did you declare that you are adding this field in the
> > IndexingFilter.addIndexBackendOptions(..) ? See how other indexing
> > plugins do this.
> >
> >
> > --
> > Best regards,
> > Andrzej Bialecki <><
> > ___. ___ ___ ___ _ _ __________________________________
> > [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> > ___|||__|| \| || | Embedded Unix, System Integration
> > http://www.sigram.com Contact: info at sigram dot com
> >

[jira] Commented: (NUTCH-768) Upgrade Nutch 1.0 to use Hadoop 0.20

2009-11-24T13:27:39Z

[ https://issues.apache.org/jira/browse/NUTCH-768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782179#action_12782179 ]

Andrzej Bialecki commented on NUTCH-768:
-----------------------------------------

Are there any source code changes involved? If so, please upload a patch.

Did you check this in local, distributed or pseudo-distributed mode? In the past there have been errors related to local (or distributed) mode that wouldn't occur when running in other modes.

> Upgrade Nutch 1.0 to use Hadoop 0.20
> ------------------------------------
>
> Key: NUTCH-768
> URL: https://issues.apache.org/jira/browse/NUTCH-768
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 1.1
> Environment: All
> Reporter: Dennis Kubes
> Assignee: Dennis Kubes
> Fix For: 1.1
>
>
> Upgrade Nutch 1.0 to use the Hadoop 0.20 release.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (NUTCH-771) Add WebGraph classes to the bin/nutch script

2009-11-24T13:25:39Z

[ https://issues.apache.org/jira/browse/NUTCH-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782177#action_12782177 ]

Andrzej Bialecki commented on NUTCH-771:
-----------------------------------------

+1 to adding these to the script. The names are cryptic, though ... this would call for a clear documentation in the script itself, and in appropriate places on the wiki.

> Add WebGraph classes to the bin/nutch script
> --------------------------------------------
>
> Key: NUTCH-771
> URL: https://issues.apache.org/jira/browse/NUTCH-771
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 1.1
> Environment: All, shell script
> Reporter: Dennis Kubes
> Assignee: Dennis Kubes
> Fix For: 1.1
>
>
> Currently the webgraph jobs are called on the command line by calling main methods on their classes. I propose to upgrade the bin/nutch shell script to allow calling these jobs as well. This would include the webgraphdb, linkrank, scoreupdater, and nodedumper jobs.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (NUTCH-768) Upgrade Nutch 1.0 to use Hadoop 0.20

2009-11-24T13:17:39Z

[ https://issues.apache.org/jira/browse/NUTCH-768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782172#action_12782172 ]

Dennis Kubes commented on NUTCH-768:
------------------------------------

I have tested the upgrade with Hadoop 0.20. To upgrade this correctly we do need to upgrade Xerces both in the main lib jars and within the lib-xml plugin. I have upgraded to the most recent version of Xerces 2.9.x. Having run through multiple full crawl and index cycles both on the new and old indexing frameworks, including the webgraphdb, and the solr indexing process, I didn't find any errors within the process. If no one has any objections I will commit these changes within the next 24 hours.

> Upgrade Nutch 1.0 to use Hadoop 0.20
> ------------------------------------
>
> Key: NUTCH-768
> URL: https://issues.apache.org/jira/browse/NUTCH-768
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 1.1
> Environment: All
> Reporter: Dennis Kubes
> Assignee: Dennis Kubes
> Fix For: 1.1
>
>
> Upgrade Nutch 1.0 to use the Hadoop 0.20 release.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (NUTCH-771) Add WebGraph classes to the bin/nutch script

2009-11-24T12:47:39Z

Add WebGraph classes to the bin/nutch script
--------------------------------------------

Key: NUTCH-771
URL: https://issues.apache.org/jira/browse/NUTCH-771
Project: Nutch
Issue Type: Improvement
Affects Versions: 1.1
Environment: All, shell script
Reporter: Dennis Kubes
Assignee: Dennis Kubes
Fix For: 1.1

Currently the webgraph jobs are called on the command line by calling main methods on their classes. I propose to upgrade the bin/nutch shell script to allow calling these jobs as well. This would include the webgraphdb, linkrank, scoreupdater, and nodedumper jobs.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

wrong wiki front page

2009-11-24T08:46:21Z

Hello everybody,

I don't know if it is a known issue, but it's been like that since at least a couple of days so I figured I should tell someone. The root url for the nutch wiki http://wiki.apache.org/nutch/ doesn't redirect to http://wiki.apache.org/nutch/FrontPage ! It's annoying because that's the url given by google and the nutch website. It might be a language detection problem because I see this ugly and not very helpful page : http://wiki.apache.org/nutch/PageD%27Accueil (page d'accueil = home page in french).

Not much of a contribution for my first message here, but I hope to do more soon.

Alban Mouton

[Nutch Wiki] Update of "FrontPage" by DennisKubes

2009-11-24T08:00:33Z

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Nutch Wiki" for change notification.

The "FrontPage" page has been changed by DennisKubes.
http://wiki.apache.org/nutch/FrontPage?action=diff&rev1=122&rev2=123

--------------------------------------------------

* NonDefaultIntranetCrawlingOptions - Desirable options to add to your intranet crawling configuration.
* RunningNutchAndSolr - How to configure Nutch to crawl, but post to Solr for search/index
* NutchWithChineseAnalyzer - References to some Chinese articles explaining how to setup Nutch with 3rd party Chinese analyzers
+ * OptimizingCrawls - How to optimize your crawling/fetching speed with Nutch.

== Nutch Development ==
* [[Becoming_A_Nutch_Developer|Becoming a Nutch Developer]] - Start developing and contributing to Nutch.

[Nutch Wiki] Update of "OptimizingCrawls" by DennisKubes

2009-11-24T07:59:31Z

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Nutch Wiki" for change notification.

The "OptimizingCrawls" page has been changed by DennisKubes.
The comment on this change is: Page about optimizing crawling speed.
http://wiki.apache.org/nutch/OptimizingCrawls

--------------------------------------------------

New page:
'''Here are the things that could potentially slow down fetching'''

1) DNS setup

2) The number of crawlers you have, too many, too few.

3) Bandwidth limitations

4) Number of threads per host (politeness)

5) Uneven distribution of urls to fetch and politeness.

6) High crawl-delays from robots.txt (usually along with an uneven distribution of urls).

7) Many slow websites (again usually with an uneven distribution).

8) Downloading lots of content (PDFS, very large html pages, again possibly an uneven distribution).

9) Others

'''Now how do we fix them'''

1) Have a DNS setup on each local crawling machine, if multiple crawling machines and a single centralized DNS it can act like a DOS attack on the DNS server slowing the entire system. We always did a two layer setup hitting first to the local DNS cache then to a large DNS cache like OpenDNS or Verizon.

2) This would be number of map tasks * fetcher.threads.fetch. So 10 map tasks * 20 threads = 200 fetchers at once. Too many and you overload your system, too few and other factors and the machine sites idle. You will need to play around with this setting for your setup.

3) Bandwidth limitations. Use ntop, ganglia, and other monitoring tools to determine how much bandwidth you are using. Account for in and out bandwidth. A simple test, from a server inside the fetching network but not itself fetching, if it is very slow connecting to or downloading content when fetching is occurring, it is a good bet you are maxing out bandwidth. If you set http timeout as we describe later and are maxing your bandwidth, you will start seeing many http timeout errors.

4) Politeness along with uneven distribution of urls is probably the biggest limiting factor. If one thread is processing a single site and there are a lot of urls from that site to fetch all other threads will sit idle while that one thread finishes. Some solutions, use fetcher.server.delay to shorten the time between page fetches and use fetcher.threads.per.host to increase the number of threads fetching for a single site (this would still be in the same map task though and hence the same JVM ChildTask process). If increasing this > 0 you could also set fetcher.server.min.delay to some value > 0 for politeness to min and max bound the process.

5) Fetching a lot of pages from a single site or a lot of pages from a few sites will slow down fetching dramatically. For full web crawls you want an even distribution so all fetching threads can be active. Setting generate.max.per.host to a value > 0 will limit the number of pages from a single host/domain to fetch.

6) Crawl-delay can be used and is obeyed by nutch in robots.txt. Most sites don't use this setting but a few (some malicious do). I have seen crawl-delays as high as 2 days in seconds. The fetcher.max.crawl.delay variable will ignore pages with crawl delays > x. I usually set this to 10 seconds, default is 30. Even at 10 seconds if you have a lot of pages from a site from which you can only crawl 1 page every 10 seconds it is going to be slow. On the flip side, setting this to a low value will ignore and not fetch those pages.

7) Sometimes, manytimes websites are just slow. Setting a low value for http.timeout helps. The default is 10 seconds. If you don't care and want as many pages as fast as possible, set it lower. Some websites, digg for instance, will bandwidth limit you on their side only allowing x connections per given time frame. So even if you only have say 50 pages from a single site (which I still think is to many). It may be waiting 10 seconds on each page. The ftp.timeout can also be set if fetching ftp content. 8) Lots of content means slower fetching. If downloading PDFs and other non-html documents this is especially true. To avoid non-html content you can use the url filters. I prefer the prefix and suffix filters. The http.content.limit and ftp.content.limit can be used to limit the amount of content downloaded for a single document.

9) Other things that could be causing slow fetching:

* Max the number of open sockets/files on a machine. You will start seeing IO errors or can't open socket errors.
* Poor routing. Bad routers or home routers might not be able to handle the number of connections going through at once. An incorrect routing setup could also be causing problems but those are usually much more complex to diagnose. Use network trace and mapping tools if you think this is happening. Upstream routing can also be a problem from your network provider.
* Bad network cards. I have seen network cards flip once they reach a certain bandwidth point. This was more prevalent on, at the time, newer gigabit cards. Not usually my first thought but always a possibility. Use tcpdump and network monitoring tools on the single interface.

That is about it from my perspective. Feel free to add anything if anybody else thinks of other things.

Re: Plugin Developement Help

2009-11-24T07:32:43Z

Sorry its suppose to say "would be stored somewhere it DOESN'T seem to be getting to the doc.add bit which"

On 24 November 2009 at 12:27 "david.stuart@..." <david.stuart@...> wrote:

> I thought I did but I thought before I did a bin/nutch index (or solrindex) it
> would be stored somewhere it does seems to be getting to the doc.add bit which
> makes me think the variable is empty
> {code}
>     public void addIndexBackendOptions(Configuration conf) {
>     LOG.warn("+_+_You called me _+_+");
>     LuceneWriter.addFieldOptions("html_filter_data", STORE.YES,
> INDEX.UNTOKENIZED, conf);
>     }
>
>     public NutchDocument filter(NutchDocument doc, Parse parse, Text url,
> CrawlDatum datum, Inlinks inlinks) throws IndexingException {
>     LOG.warn("________________________FILTER_______________________");
>     String html_filter_data = parse.getData().getMeta("html_filter_data");
>     if (html_filter_data != null){
>         LOG.warn("________________________Adding filter
> data_______________________");
>         doc.add("html_filter_data", html_filter_data);
>     }
>     return doc;
>     }
> {code}
> On 24 November 2009 at 12:05 Andrzej Bialecki <ab@...> wrote:
>
> > david.stuart@... wrote:
> > > Hi All,
> > >
> > > I think I am just about finished my plugin (nutch 1.0) which adds extra
> > > metadata to during parsing the problem I am having is it doesn't seem to
> > > be adding the data to the system (via luke or readseg). I looked at in
> > > the wiki but it seems to be for 0.9 and the syntax looks different.
> > >
> > > {code}
> > > public ParseResult filter(Content content, ParseResult parseResult,
> > > HTMLMetaTags metaTags, DocumentFragment doc) {
> > > Metadata metadata = new Metadata();
> > > // parse the content
> > > DocumentFragment root;
> > > String docTrans;
> > > try {
> > > byte[] contentInOctets = content.getContent();
> > > String input = new String(contentInOctets);
> > > XSLTSimpleTransform DocTransform = new XSLTSimpleTransform();
> > > docTrans = DocTransform.doTransform(input);
> > > Parse parse = parseResult.get(content.getUrl());
> > > metadata = parse.getData().getParseMeta();
> > > metadata.add("filter_html_data", docTrans);
> > >
> > > } catch (Exception e) {
> > > e.printStackTrace(LogUtil.getWarnStream(LOG));
> > > }
> > >
> > > return parseResult;
> > > }
> > > {code}
> >
> > Did you declare that you are adding this field in the
> > IndexingFilter.addIndexBackendOptions(..) ? See how other indexing
> > plugins do this.
> >
> >
> > --
> > Best regards,
> > Andrzej Bialecki <><
> > ___. ___ ___ ___ _ _ __________________________________
> > [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> > ___|||__|| \| || | Embedded Unix, System Integration
> > http://www.sigram.com Contact: info at sigram dot com
> >

Re: Plugin Developement Help

2009-11-24T03:53:20Z

Sorry I meant doesn't get to doc.add

David

On 24 Nov 2009, at 11:27, "david.stuart@..." <david.stuart@...
> wrote:

> I thought I did but I thought before I did a bin/nutch index (or
> solrindex) it would be stored somewhere it does seems to be getting
> to the doc.add bit which makes me think the variable is empty
> {code}
> public void addIndexBackendOptions(Configuration conf) {
> LOG.warn("+_+_You called me _+_+");
> LuceneWriter.addFieldOptions("html_filter_data", STORE.YES,
> INDEX.UNTOKENIZED, conf);
> }
>
> public NutchDocument filter(NutchDocument doc, Parse parse, Text
> url, CrawlDatum datum, Inlinks inlinks) throws IndexingException {
> LOG.warn
> ("________________________FILTER_______________________");
> String html_filter_data = parse.getData().getMeta
> ("html_filter_data");
> if (html_filter_data != null){
> LOG.warn("________________________Adding filter
> data_______________________");
> doc.add("html_filter_data", html_filter_data);
> }
> return doc;
> }
> {code}
> On 24 November 2009 at 12:05 Andrzej Bialecki <ab@...> wrote:
>
> > david.stuart@... wrote:
> > > Hi All,
> > >
> > > I think I am just about finished my plugin (nutch 1.0) which
> adds extra
> > > metadata to during parsing the problem I am having is it doesn't
> seem to
> > > be adding the data to the system (via luke or readseg). I looked
> at in
> > > the wiki but it seems to be for 0.9 and the syntax looks
> different.
> > >
> > > {code}
> > > public ParseResult filter(Content content, ParseResult
> parseResult,
> > > HTMLMetaTags metaTags, DocumentFragment doc) {
> > > Metadata metadata = new Metadata();
> > > // parse the content
> > > DocumentFragment root;
> > > String docTrans;
> > > try {
> > > byte[] contentInOctets = content.getContent();
> > > String input = new String(contentInOctets);
> > > XSLTSimpleTransform DocTransform = new
> XSLTSimpleTransform();
> > > docTrans = DocTransform.doTransform(input);
> > > Parse parse = parseResult.get(content.getUrl());
> > > metadata = parse.getData().getParseMeta();
> > > metadata.add("filter_html_data", docTrans);
> > >
> > > } catch (Exception e) {
> > > e.printStackTrace(LogUtil.getWarnStream(LOG));
> > > }
> > >
> > > return parseResult;
> > > }
> > > {code}
> >
> > Did you declare that you are adding this field in the
> > IndexingFilter.addIndexBackendOptions(..) ? See how other indexing
> > plugins do this.
> >
> >
> > --
> > Best regards,
> > Andrzej Bialecki <><
> > ___. ___ ___ ___ _ _ __________________________________
> > [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> > ___|||__|| \| || | Embedded Unix, System Integration
> > http://www.sigram.com Contact: info at sigram dot com
> >

Re: Plugin Developement Help

2009-11-24T03:27:08Z

I thought I did but I thought before I did a bin/nutch index (or solrindex) it would be stored somewhere it does seems to be getting to the doc.add bit which makes me think the variable is empty
{code}
    public void addIndexBackendOptions(Configuration conf) {
    LOG.warn("+_+_You called me _+_+");
    LuceneWriter.addFieldOptions("html_filter_data", STORE.YES, INDEX.UNTOKENIZED, conf);
    }

    public NutchDocument filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) throws IndexingException {
    LOG.warn("________________________FILTER_______________________");
    String html_filter_data = parse.getData().getMeta("html_filter_data");
    if (html_filter_data != null){
        LOG.warn("________________________Adding filter data_______________________");
        doc.add("html_filter_data", html_filter_data);
    }
    return doc;
    }
{code}
On 24 November 2009 at 12:05 Andrzej Bialecki <ab@...> wrote:

> david.stuart@... wrote:
> > Hi All,
> >
> > I think I am just about finished my plugin (nutch 1.0) which adds extra
> > metadata to during parsing the problem I am having is it doesn't seem to
> > be adding the data to the system (via luke or readseg). I looked at in
> > the wiki but it seems to be for 0.9 and the syntax looks different.
> >
> > {code}
> > public ParseResult filter(Content content, ParseResult parseResult,
> > HTMLMetaTags metaTags, DocumentFragment doc) {
> > Metadata metadata = new Metadata();
> > // parse the content
> > DocumentFragment root;
> > String docTrans;
> > try {
> > byte[] contentInOctets = content.getContent();
> > String input = new String(contentInOctets);
> > XSLTSimpleTransform DocTransform = new XSLTSimpleTransform();
> > docTrans = DocTransform.doTransform(input);
> > Parse parse = parseResult.get(content.getUrl());
> > metadata = parse.getData().getParseMeta();
> > metadata.add("filter_html_data", docTrans);
> >
> > } catch (Exception e) {
> > e.printStackTrace(LogUtil.getWarnStream(LOG));
> > }
> >
> > return parseResult;
> > }
> > {code}
>
> Did you declare that you are adding this field in the
> IndexingFilter.addIndexBackendOptions(..) ? See how other indexing
> plugins do this.
>
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>