[jira] Created: (NUTCH-755) DomainURLFilter crashes on malformed URL

DomainURLFilter crashes on malformed URL
----------------------------------------

                 Key: NUTCH-755
                 URL: https://issues.apache.org/jira/browse/NUTCH-755
             Project: Nutch
          Issue Type: Bug
          Components: fetcher
    Affects Versions: 1.0.0
         Environment: Tomcat 6.0.14
Java 1.6.0_14
Linux
            Reporter: Mike Baranczak



2009-09-16 21:54:17,001 ERROR [Thread-156] DomainURLFilter - Could not apply filter on url: http:/comments.php
java.lang.NullPointerException
        at org.apache.nutch.urlfilter.domain.DomainURLFilter.filter(DomainURLFilter.java:173)
        at org.apache.nutch.net.URLFilters.filter(URLFilters.java:88)
        at org.apache.nutch.parse.ParseOutputFormat$1.write(ParseOutputFormat.java:200)
        at org.apache.nutch.parse.ParseOutputFormat$1.write(ParseOutputFormat.java:113)
        at org.apache.nutch.fetcher.FetcherOutputFormat$1.write(FetcherOutputFormat.java:96)
        at org.apache.nutch.fetcher.FetcherOutputFormat$1.write(FetcherOutputFormat.java:70)
        at org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:410)
        at org.apache.hadoop.mapred.lib.IdentityReducer.reduce(IdentityReducer.java:39)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:170)


Expected behavior would be to recognize the URL as malformed, and reject it.


--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/NUTCH-755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758785#action_12758785 ]

Mike Baranczak commented on NUTCH-755:
--------------------------------------

Update: this class also chokes on legal URLs that specify the port number, like:

http://localhost:9999/

> DomainURLFilter crashes on malformed URL
> ----------------------------------------
>
>                 Key: NUTCH-755
>                 URL: https://issues.apache.org/jira/browse/NUTCH-755
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>    Affects Versions: 1.0.0
>         Environment: Tomcat 6.0.14
> Java 1.6.0_14
> Linux
>            Reporter: Mike Baranczak
>
> 2009-09-16 21:54:17,001 ERROR [Thread-156] DomainURLFilter - Could not apply filter on url: http:/comments.php
> java.lang.NullPointerException
>         at org.apache.nutch.urlfilter.domain.DomainURLFilter.filter(DomainURLFilter.java:173)
>         at org.apache.nutch.net.URLFilters.filter(URLFilters.java:88)
>         at org.apache.nutch.parse.ParseOutputFormat$1.write(ParseOutputFormat.java:200)
>         at org.apache.nutch.parse.ParseOutputFormat$1.write(ParseOutputFormat.java:113)
>         at org.apache.nutch.fetcher.FetcherOutputFormat$1.write(FetcherOutputFormat.java:96)
>         at org.apache.nutch.fetcher.FetcherOutputFormat$1.write(FetcherOutputFormat.java:70)
>         at org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:410)
>         at org.apache.hadoop.mapred.lib.IdentityReducer.reduce(IdentityReducer.java:39)
>         at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
>         at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:170)
> Expected behavior would be to recognize the URL as malformed, and reject it.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/NUTCH-755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769926#action_12769926 ]

Reinhard Schwab commented on NUTCH-755:
---------------------------------------

in the first case, the "url" is rejected or?
the filter method will return null.

catch (Exception e) {

      // if an error happens, allow the url to pass
      LOG.error("Could not apply filter on url: " + url + "\n"
        + org.apache.hadoop.util.StringUtils.stringifyException(e));
      return null;
    }

the comment in the code is wrong.
if the method returns null, the url does not pass.

the malformed check is done by java.net.URL constructor.
it accepts http:/comments.php

> DomainURLFilter crashes on malformed URL
> ----------------------------------------
>
>                 Key: NUTCH-755
>                 URL: https://issues.apache.org/jira/browse/NUTCH-755
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>    Affects Versions: 1.0.0
>         Environment: Tomcat 6.0.14
> Java 1.6.0_14
> Linux
>            Reporter: Mike Baranczak
>
> 2009-09-16 21:54:17,001 ERROR [Thread-156] DomainURLFilter - Could not apply filter on url: http:/comments.php
> java.lang.NullPointerException
>         at org.apache.nutch.urlfilter.domain.DomainURLFilter.filter(DomainURLFilter.java:173)
>         at org.apache.nutch.net.URLFilters.filter(URLFilters.java:88)
>         at org.apache.nutch.parse.ParseOutputFormat$1.write(ParseOutputFormat.java:200)
>         at org.apache.nutch.parse.ParseOutputFormat$1.write(ParseOutputFormat.java:113)
>         at org.apache.nutch.fetcher.FetcherOutputFormat$1.write(FetcherOutputFormat.java:96)
>         at org.apache.nutch.fetcher.FetcherOutputFormat$1.write(FetcherOutputFormat.java:70)
>         at org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:410)
>         at org.apache.hadoop.mapred.lib.IdentityReducer.reduce(IdentityReducer.java:39)
>         at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
>         at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:170)
> Expected behavior would be to recognize the URL as malformed, and reject it.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/NUTCH-755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrzej Bialecki  closed NUTCH-755.
-----------------------------------

    Resolution: Cannot Reproduce
      Assignee: Andrzej Bialecki

> DomainURLFilter crashes on malformed URL
> ----------------------------------------
>
>                 Key: NUTCH-755
>                 URL: https://issues.apache.org/jira/browse/NUTCH-755
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>    Affects Versions: 1.0.0
>         Environment: Tomcat 6.0.14
> Java 1.6.0_14
> Linux
>            Reporter: Mike Baranczak
>            Assignee: Andrzej Bialecki
>
> 2009-09-16 21:54:17,001 ERROR [Thread-156] DomainURLFilter - Could not apply filter on url: http:/comments.php
> java.lang.NullPointerException
>         at org.apache.nutch.urlfilter.domain.DomainURLFilter.filter(DomainURLFilter.java:173)
>         at org.apache.nutch.net.URLFilters.filter(URLFilters.java:88)
>         at org.apache.nutch.parse.ParseOutputFormat$1.write(ParseOutputFormat.java:200)
>         at org.apache.nutch.parse.ParseOutputFormat$1.write(ParseOutputFormat.java:113)
>         at org.apache.nutch.fetcher.FetcherOutputFormat$1.write(FetcherOutputFormat.java:96)
>         at org.apache.nutch.fetcher.FetcherOutputFormat$1.write(FetcherOutputFormat.java:70)
>         at org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:410)
>         at org.apache.hadoop.mapred.lib.IdentityReducer.reduce(IdentityReducer.java:39)
>         at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
>         at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:170)
> Expected behavior would be to recognize the URL as malformed, and reject it.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/NUTCH-755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783299#action_12783299 ]

Andrzej Bialecki  commented on NUTCH-755:
-----------------------------------------

I could not verify that the filter indeed crashes - it simply prints the exception and then returns null, as you suggested.

> DomainURLFilter crashes on malformed URL
> ----------------------------------------
>
>                 Key: NUTCH-755
>                 URL: https://issues.apache.org/jira/browse/NUTCH-755
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>    Affects Versions: 1.0.0
>         Environment: Tomcat 6.0.14
> Java 1.6.0_14
> Linux
>            Reporter: Mike Baranczak
>            Assignee: Andrzej Bialecki
>
> 2009-09-16 21:54:17,001 ERROR [Thread-156] DomainURLFilter - Could not apply filter on url: http:/comments.php
> java.lang.NullPointerException
>         at org.apache.nutch.urlfilter.domain.DomainURLFilter.filter(DomainURLFilter.java:173)
>         at org.apache.nutch.net.URLFilters.filter(URLFilters.java:88)
>         at org.apache.nutch.parse.ParseOutputFormat$1.write(ParseOutputFormat.java:200)
>         at org.apache.nutch.parse.ParseOutputFormat$1.write(ParseOutputFormat.java:113)
>         at org.apache.nutch.fetcher.FetcherOutputFormat$1.write(FetcherOutputFormat.java:96)
>         at org.apache.nutch.fetcher.FetcherOutputFormat$1.write(FetcherOutputFormat.java:70)
>         at org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:410)
>         at org.apache.hadoop.mapred.lib.IdentityReducer.reduce(IdentityReducer.java:39)
>         at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
>         at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:170)
> Expected behavior would be to recognize the URL as malformed, and reject it.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.