Nabble - Apache Tika - Development

[jira] Created: (TIKA-338) Trying to use -encoding parameter alwyas results in an exception

2009-11-27T09:16:20Z

Trying to use -encoding parameter alwyas results in an exception
----------------------------------------------------------------

Key: TIKA-338
URL: https://issues.apache.org/jira/browse/TIKA-338
Project: Tika
Issue Type: Bug
Components: cli
Reporter: Peter Wolanin
Fix For: 0.6

There is a logical error in the CLI code - -encoding can never work and always results in an exception

$ java -jar tika-app/target/tika-app-0.6-SNAPSHOT.jar -encoding=UTF-8 -t test.txt

Exception in thread "main" java.io.UnsupportedEncodingException: ncoding=UTF-8
at sun.nio.cs.StreamEncoder.forOutputStreamWriter(StreamEncoder.java:42)

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (TIKA-324) Tika CLI mangles utf-8 content in text (-t) mode (on Mac OS X)

2009-11-27T09:02:20Z

[ https://issues.apache.org/jira/browse/TIKA-324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783134#action_12783134 ]

Peter Wolanin commented on TIKA-324:
------------------------------------

There is a logical bug in the committed code: -encoding= does not work, fails with exceptions like:

Exception in thread "main" java.io.UnsupportedEncodingException: ncoding=UTF-8

note "ncoding". Opening follow-up issue.

> Tika CLI mangles utf-8 content in text (-t) mode (on Mac OS X)
> --------------------------------------------------------------
>
> Key: TIKA-324
> URL: https://issues.apache.org/jira/browse/TIKA-324
> Project: Tika
> Issue Type: Bug
> Components: cli
> Affects Versions: 0.3, 0.4, 0.5
> Environment: Mac OS 10.5, java version "1.6.0_15"
> Reporter: Peter Wolanin
> Assignee: Jukka Zitting
> Priority: Critical
> Fix For: 0.6
>
> Attachments: test.txt, TIKA-324-0.5.patch, TIKA-324-macosx.patch, TIKA-324.patch, TIKA-324.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> When using the -t flag to tika, multi-byte content is destroyed in the output.
> Example:
> $ java -jar tika-app-0.4.jar -t ./test.txt
> I?t?rn?ti?n?liz?ti?n
> $ java -jar tika-app-0.4.jar -x ./test.txt
> <?xml version="1.0" encoding="UTF-8"?>
> <html xmlns="http://www.w3.org/1999/xhtml">
> <head>
> <title/>
> </head>
> <body>
> <p>Iñtërnâtiônàlizætiøn
> </p>
> </body>
> </html>
> see also: http://drupal.org/node/622508#comment-2267918

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (TIKA-324) Tika CLI mangles utf-8 content in text (-t) mode (on Mac OS X)

2009-11-27T07:12:20Z

[ https://issues.apache.org/jira/browse/TIKA-324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jukka Zitting resolved TIKA-324.
--------------------------------

Resolution: Fixed
Fix Version/s: 0.6
Assignee: Jukka Zitting

OK. I've committed the latest patch to trunk. The code now never uses the default platform encoding on Mac OS X, opting instead for UTF-8 as the default. People can still override the setting with an explicit --encoding argument.

For the CentOS case I recommend just setting the LANG environment variable correctly, as that's used also by other programs and there is no other easy way for Tika or Java to figure out which encoding should be used on that platform.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (TIKA-337) SWF parser

2009-11-27T02:23:39Z

[ https://issues.apache.org/jira/browse/TIKA-337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Julien Nioche updated TIKA-337:
-------------------------------

Attachment: TIKA-337.patch

patch for SWF parser

> SWF parser
> ----------
>
> Key: TIKA-337
> URL: https://issues.apache.org/jira/browse/TIKA-337
> Project: Tika
> Issue Type: New Feature
> Components: parser
> Reporter: Julien Nioche
> Attachments: TIKA-337.patch
>
>
> Here is an initial implementation of a SWF Parser which uses JavaSWF and has been adapted from A. Bialecki's implementation for Nutch.
> The main differences with the implementation for Nutch is that we use the latest version of JavaSWF and do not try to extract text from the actions or structured URLs. As usual URLs can be obtained from the text extracted using ParserPostProcessor.
> JavaSWF has changed quite a bit since the Nutch integration and I wanted to keep this initial port nice and simple. It should be possible to extract the URLs from the actions using JavaSWF's API, I think this is what they did in Heritrix.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (TIKA-337) SWF parser

2009-11-27T02:21:39Z

SWF parser
----------

Key: TIKA-337
URL: https://issues.apache.org/jira/browse/TIKA-337
Project: Tika
Issue Type: New Feature
Components: parser
Reporter: Julien Nioche

Here is an initial implementation of a SWF Parser which uses JavaSWF and has been adapted from A. Bialecki's implementation for Nutch.
The main differences with the implementation for Nutch is that we use the latest version of JavaSWF and do not try to extract text from the actions or structured URLs. As usual URLs can be obtained from the text extracted using ParserPostProcessor.
JavaSWF has changed quite a bit since the Nutch integration and I wanted to keep this initial port nice and simple. It should be possible to extract the URLs from the actions using JavaSWF's API, I think this is what they did in Heritrix.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (TIKA-336) More issues with RDF mime detection

2009-11-25T15:45:39Z

[ https://issues.apache.org/jira/browse/TIKA-336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris A. Mattmann resolved TIKA-336.
------------------------------------

Resolution: Fixed

- fixed in r884340

Yuan-Fang, please test out the latest Tika trunk. I've:

* updated the test-difficult-rdf2.xml file to remove the <?xml header
* updated the tika-mimetypes.xml to detect files that start with <!-- as xml files (as a default magic first check). Then, this forces xmlRoot detection to occur where the specific XML subclass is detected (which is what we want). There, application/rdf+xml is properly detected. Before, since there was no magic header for <!--, the initial magic result check was null and then the mimeTypes detector ended up returning text/plain.

In the future we may want to make:

* xmlRoot extraction occur on text/plain documents
* move the text/plain check to the beginning of the o.a.tika.mime.MimeTypes#getMimeType(byte[] data) function

> More issues with RDF mime detection
> -----------------------------------
>
> Key: TIKA-336
> URL: https://issues.apache.org/jira/browse/TIKA-336
> Project: Tika
> Issue Type: Bug
> Components: mime
> Affects Versions: 0.5
> Environment: several user environments as well as validated in Mattmann's environment.
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Fix For: 0.6
>
>
> See TIKA-309 for related discussion, but there seems to be further errors in RDF mime detection, on the OWL file located here:
> http://www.w3.org/2002/07/owl#

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (TIKA-336) More issues with RDF mime detection

2009-11-25T14:51:40Z

More issues with RDF mime detection
-----------------------------------

Key: TIKA-336
URL: https://issues.apache.org/jira/browse/TIKA-336
Project: Tika
Issue Type: Bug
Components: mime
Affects Versions: 0.5
Environment: several user environments as well as validated in Mattmann's environment.
Reporter: Chris A. Mattmann
Assignee: Chris A. Mattmann
Fix For: 0.6

See TIKA-309 for related discussion, but there seems to be further errors in RDF mime detection, on the OWL file located here:

http://www.w3.org/2002/07/owl#

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (TIKA-309) Mime type application/rdf+xml not correctly detected

2009-11-25T14:49:39Z

[ https://issues.apache.org/jira/browse/TIKA-309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782663#action_12782663 ]

Chris A. Mattmann commented on TIKA-309:
----------------------------------------

Yuang-Fang:

I've confirmed what you mentioned. When the XML header first-line is taken out of the test-difficult-rdf2.xml (as the remote URL exists), I get this:

[chipotle:~/src/tika/trunk] mattmann% mvn -Dtest=MimeDetectionTest clean test
[INFO] Scanning for projects...
[INFO] Reactor build order:
[INFO] Apache Tika parent
[INFO] Apache Tika core
[INFO] Apache Tika parsers
[INFO] Apache Tika application
[INFO] Apache Tika
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Tika parent
[INFO] task-segment: [clean, test]
[INFO] ------------------------------------------------------------------------
[INFO] [clean:clean]
[INFO] Setting property: classpath.resource.loader.class => 'org.codehaus.plexus.velocity.ContextClassLoaderResourceLoader'.
[INFO] Setting property: velocimacro.messages.on => 'false'.
[INFO] Setting property: resource.loader => 'classpath'.
[INFO] Setting property: resource.manager.logwhenfound => 'false'.
[INFO] [remote-resources:process {execution: default}]
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Tika core
[INFO] task-segment: [clean, test]
[INFO] ------------------------------------------------------------------------
[INFO] [clean:clean]
[INFO] [remote-resources:process {execution: default}]
[INFO] [resources:resources]
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 20 resources
[INFO] Copying 3 resources
[INFO] [compiler:compile]
[INFO] Compiling 86 source files to /Users/mattmann/src/tika/trunk/tika-core/target/classes
[INFO] [resources:testResources]
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 24 resources
[INFO] Copying 3 resources
[INFO] [compiler:testCompile]
[INFO] Compiling 19 source files to /Users/mattmann/src/tika/trunk/tika-core/target/test-classes
[INFO] [surefire:test]
[INFO] Surefire report directory: /Users/mattmann/src/tika/trunk/tika-core/target/surefire-reports

-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.tika.mime.MimeDetectionTest
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.568 sec <<< FAILURE!

Results :

Failed tests:
testDetection(org.apache.tika.mime.MimeDetectionTest)

Tests run: 2, Failures: 1, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] There are test failures.

Please refer to /Users/mattmann/src/tika/trunk/tika-core/target/surefire-reports for the individual test results.
[INFO] ------------------------------------------------------------------------
[INFO] For more information, run Maven with the -e switch
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 8 seconds
[INFO] Finished at: Wed Nov 25 14:45:52 PST 2009
[INFO] Final Memory: 15M/31M
[INFO] ------------------------------------------------------------------------
[chipotle:~/src/tika/trunk] mattmann%

[chipotle:~/src/tika/trunk] mattmann% more tika-core/target/surefire-reports/org.apache.tika.mime.MimeDetectionTest.txt
-------------------------------------------------------------------------------
Test set: org.apache.tika.mime.MimeDetectionTest
-------------------------------------------------------------------------------
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.573 sec <<< FAILURE!
testDetection(org.apache.tika.mime.MimeDetectionTest) Time elapsed: 0.44 sec <<< FAILURE!
junit.framework.ComparisonFailure: http://www.w3.org/2002/07/owl# is not properly detected. expected:<application/rdf+xml> but w
as:<text/plain>
at junit.framework.Assert.assertEquals(Assert.java:81)
at org.apache.tika.mime.MimeDetectionTest.testStream(MimeDetectionTest.java:87)
at org.apache.tika.mime.MimeDetectionTest.testUrl(MimeDetectionTest.java:71)
at org.apache.tika.mime.MimeDetectionTest.testDetection(MimeDetectionTest.java:54)

I'm looking into this right now...I'll file another issue for this..
I'm looking into this now:

> Mime type application/rdf+xml not correctly detected
> ----------------------------------------------------
>
> Key: TIKA-309
> URL: https://issues.apache.org/jira/browse/TIKA-309
> Project: Tika
> Issue Type: Bug
> Components: mime
> Affects Versions: 0.5
> Reporter: Yuan-Fang Li
> Assignee: Chris A. Mattmann
> Priority: Minor
> Fix For: 0.5
>
>
> Mime type detector using AutoDetectParser and Metadata returns "application/xml" for the URL http://www.w3.org/2002/07/owl#, where it should be "application/rdf+xml". The correct mime type is also suggested here: http://www.w3.org/TR/owl-ref/#MIMEType.
> P.S., Tika was downloaded from svn and built with Maven last week.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (TIKA-335) TXTParser should use incoming charset

2009-11-25T14:29:39Z

[ https://issues.apache.org/jira/browse/TIKA-335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ken Krugler updated TIKA-335:
-----------------------------

Attachment: TIKA-335.patch

This patch also cleans up some generics warnings (sorry about mixing the two, I was going to open a second issue but the two were co-mingled).

In order to make this work, I had to modify the charset detection code to actually use the hint - weird that ICU never actually implemented this.

Includes a test case for an ambiguous run of text that could be UTF-8 or 8859-1.

> TXTParser should use incoming charset
> -------------------------------------
>
> Key: TIKA-335
> URL: https://issues.apache.org/jira/browse/TIKA-335
> Project: Tika
> Issue Type: Improvement
> Affects Versions: 0.5
> Reporter: Ken Krugler
> Priority: Minor
> Attachments: TIKA-335.patch
>
>
> The incoming charset (if any) from metadata should be passed to CharsetDetector.setDeclaredEncoding().

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (TIKA-334) HtmlParser should use CharsetDetector whenever no charset is specified via meta http-equiv tag

2009-11-25T11:53:39Z

[ https://issues.apache.org/jira/browse/TIKA-334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ken Krugler updated TIKA-334:
-----------------------------

Attachment: TIKA-334.patch

> HtmlParser should use CharsetDetector whenever no charset is specified via meta http-equiv tag
> ----------------------------------------------------------------------------------------------
>
> Key: TIKA-334
> URL: https://issues.apache.org/jira/browse/TIKA-334
> Project: Tika
> Issue Type: Improvement
> Affects Versions: 0.5
> Reporter: Ken Krugler
> Attachments: TIKA-334.patch
>
>
> Currently the HtmlParser will just call TagSoup to parse, without specifying a charset, if no charset is passed in via metadata.
> TagSoup uses the platform encoding in this case, which is often going to be wrong.
> The right thing to do is to first check for a charset specified by a meta tag. If that doesn't exist, then create a CharsetDetector. If there's a charset in the incoming meta-data, use that to call setDeclaredEncoding().

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Missing href attribute handling

2009-11-25T11:12:36Z

Hi Jukka,

Previously the HtmlParser code (or rather, HtmlHandler) would ensure
that there was always an href attribute for an <a> tag - if it was
missing, it would get set to "".

The new code no longer does this, which caused some of my code to fail.

I see the new code looks like:

String href = atts.getValue("href");
if (href != null) {
xhtml.startElement("a", "href",
resolve(href.trim()));
} else {
String anchor = atts.getValue("name");
if (anchor != null) {
xhtml.startElement("a", "name", anchor.trim());
} else {
xhtml.startElement("a");
}
}

I'm assuming this is by design, though for XHTML the use of "name" is
deprecated, with "id" being its replacement.

-- Ken

--------------------------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
e l a s t i c w e b m i n i n g

[jira] Updated: (TIKA-335) TXTParser should use incoming charset

2009-11-25T10:50:39Z

[ https://issues.apache.org/jira/browse/TIKA-335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ken Krugler updated TIKA-335:
-----------------------------

Priority: Minor (was: Major)
Description:
The incoming charset (if any) from metadata should be passed to CharsetDetector.setDeclaredEncoding().

was:
In looking at how TXTParser uses CharsetDetector, I see the following issues:

1. The incoming charset (if any) from metadata should be passed to CharsetDetector.setDeclaredEncoding().
2. The first supported charset should be used, not the last. These are returned in confidence order, from best to worst.
3. The current code might also wind up setting a language from one result, and the charset from another.

So the biggest change is to bail out of the loop once a supported charset has been found.

Issue Type: Improvement (was: Bug)
Summary: TXTParser should use incoming charset (was: TXTParser use of CharsetDetector has several bugs)

> TXTParser should use incoming charset
> -------------------------------------
>
> Key: TIKA-335
> URL: https://issues.apache.org/jira/browse/TIKA-335
> Project: Tika
> Issue Type: Improvement
> Affects Versions: 0.5
> Reporter: Ken Krugler
> Priority: Minor
>
> The incoming charset (if any) from metadata should be passed to CharsetDetector.setDeclaredEncoding().

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (TIKA-335) TXTParser use of CharsetDetector has several bugs

2009-11-25T10:48:39Z

TXTParser use of CharsetDetector has several bugs
-------------------------------------------------

Key: TIKA-335
URL: https://issues.apache.org/jira/browse/TIKA-335
Project: Tika
Issue Type: Bug
Affects Versions: 0.5
Reporter: Ken Krugler

In looking at how TXTParser uses CharsetDetector, I see the following issues:

1. The incoming charset (if any) from metadata should be passed to CharsetDetector.setDeclaredEncoding().
2. The first supported charset should be used, not the last. These are returned in confidence order, from best to worst.
3. The current code might also wind up setting a language from one result, and the charset from another.

So the biggest change is to bail out of the loop once a supported charset has been found.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (TIKA-334) HtmlParser should use CharsetDetector whenever no charset is specified via meta http-equiv tag

2009-11-25T10:42:39Z

HtmlParser should use CharsetDetector whenever no charset is specified via meta http-equiv tag
----------------------------------------------------------------------------------------------

Key: TIKA-334
URL: https://issues.apache.org/jira/browse/TIKA-334
Project: Tika
Issue Type: Improvement
Affects Versions: 0.5
Reporter: Ken Krugler

Currently the HtmlParser will just call TagSoup to parse, without specifying a charset, if no charset is passed in via metadata.

TagSoup uses the platform encoding in this case, which is often going to be wrong.

The right thing to do is to first check for a charset specified by a meta tag. If that doesn't exist, then create a CharsetDetector. If there's a charset in the incoming meta-data, use that to call setDeclaredEncoding().

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (TIKA-332) Use http-equiv meta tag charset info when processing HTML documents

2009-11-25T10:40:39Z

[ https://issues.apache.org/jira/browse/TIKA-332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ken Krugler updated TIKA-332:
-----------------------------

Description:
Currently Tika doesn't use the charset info that's optionally present in HTML documents, via the <meta http-equiv="Content-type" content="text/html; charset=xxx"> tag.

If the mime-type is detected as being one that's handled by the HtmlParser, then the first 4-8K of text should be converted from bytes to us-ascii, and then scanned using a regex something like:

private static final Pattern HTTP_EQUIV_CHARSET_PATTERN = Pattern.compile("<meta\\s+http-equiv\\s*=\\s*['\"]\\s*Content-Type['\"]\\s+content\\s*=\\s*['\"][^;]+;\\s*charset\\s*=\\s*([^'\"]+)\"");

If a charset is detected, this should take precedence over a charset in the HTTP response headers, and (obviously) used to convert the bytes to text before the actual parsing of the document begins.

In a test I did of 100 random HTML pages, roughly 15% contained charset info in the meta tag that wound up being different from the detected or HTTP response header charset, so this is a pretty important improvement to make. Without it, Tika isn't that useful for processing HTML pages.

Though the other problem is that the HtmlParser code doesn't use the CharsetDetector, which is another reason for lots of incorrect text. I'll file a separate issue about that.

was:
Currently Tika doesn't use the charset info that's optionally present in HTML documents, via the <meta http-equiv="Content-type" content="text/html; charset=xxx"> tag.

If the mime-type is detected as being one that's handled by the HtmlParser, then the first 4-8K of text should be converted from bytes to us-ascii, and then scanned using a regex something like:

private static final Pattern HTTP_EQUIV_CHARSET_PATTERN = Pattern.compile("<meta\\s+http-equiv\\s*=\\s*['\"]\\s*Content-Type['\"]\\s+content\\s*=\\s*['\"][^;]+;\\s*charset\\s*=\\s*([^'\"]+)\"");

If a charset is detected, this should take precedence over a charset in the HTTP response headers, and (obviously) used to convert the bytes to text before the actual parsing of the document begins.

In a test I did of 100 random HTML pages, roughly 15% contained charset info in the meta tag that wound up being different from the detected or HTTP response header charset, so this is a pretty important improvement to make. Without it, Tika isn't that useful for processing HTML pages.

I believe one of the reasons why ICU4J doesn't do a good job in detecting the charset for HTML pages is that the first 2K+ of HTML text is often all us-ascii markup, versus real content. I'll file a separate issue about how to improve charset detection for HTML pages.

> Use http-equiv meta tag charset info when processing HTML documents
> -------------------------------------------------------------------
>
> Key: TIKA-332
> URL: https://issues.apache.org/jira/browse/TIKA-332
> Project: Tika
> Issue Type: Improvement
> Affects Versions: 0.5
> Reporter: Ken Krugler
> Priority: Critical
>
> Currently Tika doesn't use the charset info that's optionally present in HTML documents, via the <meta http-equiv="Content-type" content="text/html; charset=xxx"> tag.
> If the mime-type is detected as being one that's handled by the HtmlParser, then the first 4-8K of text should be converted from bytes to us-ascii, and then scanned using a regex something like:
> private static final Pattern HTTP_EQUIV_CHARSET_PATTERN = Pattern.compile("<meta\\s+http-equiv\\s*=\\s*['\"]\\s*Content-Type['\"]\\s+content\\s*=\\s*['\"][^;]+;\\s*charset\\s*=\\s*([^'\"]+)\"");
> If a charset is detected, this should take precedence over a charset in the HTTP response headers, and (obviously) used to convert the bytes to text before the actual parsing of the document begins.
> In a test I did of 100 random HTML pages, roughly 15% contained charset info in the meta tag that wound up being different from the detected or HTTP response header charset, so this is a pretty important improvement to make. Without it, Tika isn't that useful for processing HTML pages.
> Though the other problem is that the HtmlParser code doesn't use the CharsetDetector, which is another reason for lots of incorrect text. I'll file a separate issue about that.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (TIKA-332) Use http-equiv meta tag charset info when processing HTML documents

2009-11-25T10:38:39Z

[ https://issues.apache.org/jira/browse/TIKA-332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782550#action_12782550 ]

Ken Krugler commented on TIKA-332:
----------------------------------

It turns out the HtmlParser code doesn't even use the CharsetDetector support - this is only being used by the TXTParser, as far as I can tell (and incorrectly at that).

> Use http-equiv meta tag charset info when processing HTML documents
> -------------------------------------------------------------------
>
> Key: TIKA-332
> URL: https://issues.apache.org/jira/browse/TIKA-332
> Project: Tika
> Issue Type: Improvement
> Affects Versions: 0.5
> Reporter: Ken Krugler
> Priority: Critical
>
> Currently Tika doesn't use the charset info that's optionally present in HTML documents, via the <meta http-equiv="Content-type" content="text/html; charset=xxx"> tag.
> If the mime-type is detected as being one that's handled by the HtmlParser, then the first 4-8K of text should be converted from bytes to us-ascii, and then scanned using a regex something like:
> private static final Pattern HTTP_EQUIV_CHARSET_PATTERN = Pattern.compile("<meta\\s+http-equiv\\s*=\\s*['\"]\\s*Content-Type['\"]\\s+content\\s*=\\s*['\"][^;]+;\\s*charset\\s*=\\s*([^'\"]+)\"");
> If a charset is detected, this should take precedence over a charset in the HTTP response headers, and (obviously) used to convert the bytes to text before the actual parsing of the document begins.
> In a test I did of 100 random HTML pages, roughly 15% contained charset info in the meta tag that wound up being different from the detected or HTTP response header charset, so this is a pretty important improvement to make. Without it, Tika isn't that useful for processing HTML pages.
> I believe one of the reasons why ICU4J doesn't do a good job in detecting the charset for HTML pages is that the first 2K+ of HTML text is often all us-ascii markup, versus real content. I'll file a separate issue about how to improve charset detection for HTML pages.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Closed: (TIKA-333) Improve accuracy of charset detection for HTML pages

2009-11-25T10:38:39Z

[ https://issues.apache.org/jira/browse/TIKA-333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ken Krugler closed TIKA-333.
----------------------------

Resolution: Not A Problem

In actually walking the parse code, I see that the real problem is that the HtmlParser code doesn't use the CharsetDetector. If no charset is passed in, then it just calls TagSoup, which by default uses the platform encoding. See [http://home.ccil.org/~cowan/XML/tagsoup/].

So I'll open another issue for the HtmlParser.

> Improve accuracy of charset detection for HTML pages
> ----------------------------------------------------
>
> Key: TIKA-333
> URL: https://issues.apache.org/jira/browse/TIKA-333
> Project: Tika
> Issue Type: Improvement
> Affects Versions: 0.5
> Reporter: Ken Krugler
> Priority: Minor
>
> Charset detection for HTML pages doesn't work all that well currently, due to the amount of text that's HTML markup at the beginning of the document.
> A simple solution would be to skip over the first 2K (assuming the document is long enough) before passing bytes to ICU4J.
> A more complex solution would be to scan for title and body tags, and pass bytes found in each.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (TIKA-333) Improve accuracy of charset detection for HTML pages

2009-11-25T09:54:42Z

Improve accuracy of charset detection for HTML pages
----------------------------------------------------

Key: TIKA-333
URL: https://issues.apache.org/jira/browse/TIKA-333
Project: Tika
Issue Type: Improvement
Affects Versions: 0.5
Reporter: Ken Krugler
Priority: Minor

Charset detection for HTML pages doesn't work all that well currently, due to the amount of text that's HTML markup at the beginning of the document.

A simple solution would be to skip over the first 2K (assuming the document is long enough) before passing bytes to ICU4J.

A more complex solution would be to scan for title and body tags, and pass bytes found in each.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (TIKA-332) Use http-equiv meta tag charset info when processing HTML documents

2009-11-25T09:50:39Z

Use http-equiv meta tag charset info when processing HTML documents
-------------------------------------------------------------------

Key: TIKA-332
URL: https://issues.apache.org/jira/browse/TIKA-332
Project: Tika
Issue Type: Improvement
Affects Versions: 0.5
Reporter: Ken Krugler
Priority: Critical

Currently Tika doesn't use the charset info that's optionally present in HTML documents, via the <meta http-equiv="Content-type" content="text/html; charset=xxx"> tag.

If the mime-type is detected as being one that's handled by the HtmlParser, then the first 4-8K of text should be converted from bytes to us-ascii, and then scanned using a regex something like:

private static final Pattern HTTP_EQUIV_CHARSET_PATTERN = Pattern.compile("<meta\\s+http-equiv\\s*=\\s*['\"]\\s*Content-Type['\"]\\s+content\\s*=\\s*['\"][^;]+;\\s*charset\\s*=\\s*([^'\"]+)\"");

If a charset is detected, this should take precedence over a charset in the HTTP response headers, and (obviously) used to convert the bytes to text before the actual parsing of the document begins.

In a test I did of 100 random HTML pages, roughly 15% contained charset info in the meta tag that wound up being different from the detected or HTTP response header charset, so this is a pretty important improvement to make. Without it, Tika isn't that useful for processing HTML pages.

I believe one of the reasons why ICU4J doesn't do a good job in detecting the charset for HTML pages is that the first 2K+ of HTML text is often all us-ascii markup, versus real content. I'll file a separate issue about how to improve charset detection for HTML pages.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (TIKA-331) Windings font recognition in Tika parsing + spacing issue

2009-11-24T11:13:39Z

[ https://issues.apache.org/jira/browse/TIKA-331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782100#action_12782100 ]

Ken Krugler commented on TIKA-331:
----------------------------------

I believe this is an issue for the PDF parser (PDFBox) that Tika "wraps".

Please check https://issues.apache.org/jira/browse/PDFBOX to see if this is already filed, and if not, refile it there.

> Windings font recognition in Tika parsing + spacing issue
> ---------------------------------------------------------
>
> Key: TIKA-331
> URL: https://issues.apache.org/jira/browse/TIKA-331
> Project: Tika
> Issue Type: Wish
> Components: parser
> Affects Versions: 0.4
> Environment: Windows XP / Java JDK 1.6.0_15
> Reporter: MRIT64
> Attachments: Parsing_Result1.txt, Parsing_Result2.txt, test1.pdf, test2.pdf
>
>
> I have PDF files that include some characters in Windings font.
> Tika parser replaces them with some Unicode characters that have nothing to do with the original, and, in some cases, replaces them with alphabetic characters (that is normal regarding these characters codes).
> Would it be possible to improve the parsing and remplace these characters with more accurate Unicode characters ?
> (see http://www.alanwood.net/demos/wingdings.html for possible correspondences).
> I will attach examples files when this issue will be created (would it be possible to attach files directly when creating issues ?)

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (TIKA-331) Windings font recognition in Tika parsing + spacing issue

2009-11-24T11:11:39Z

[ https://issues.apache.org/jira/browse/TIKA-331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782097#action_12782097 ]

MRIT64 commented on TIKA-331:
-----------------------------

Spacing issue
--------------------

Look at lines 10 and 11 in test2.pdf.
Look at lines 11 and 12 in Tika parsing result (Parsing_result2.txt) :

ðLocalisation des zones de livraison et de stockage
ðLocalisation des zones dangereuses

There is no space between ð and Localisation (ð is the translation of Winding's "Rightwards white arrow" by Tika).

If you copy and paste lines 10 and 11 in test2.pdf into a Notepad Window, you get :

ð Localisation des zones de livraison et de stockage
ð Localisation des zones dangereuses

...with a space between ð and Localisation.

In my case, the missing space after Tika parsing result in considering "ðLocalisation" as a word in following processes.

Regards

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (TIKA-331) Windings font recognition in Tika parsing + spacing issue

2009-11-24T11:00:39Z

[ https://issues.apache.org/jira/browse/TIKA-331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

MRIT64 updated TIKA-331:
------------------------

Attachment: Parsing_Result2.txt
test2.pdf

Another example with the same WORD source file converted into PDF with another tool, and the Tika parsing result. Windings characters are translated into different Unicode characters than with the previous version.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (TIKA-331) Windings font recognition in Tika parsing + spacing issue

2009-11-24T10:58:40Z

[ https://issues.apache.org/jira/browse/TIKA-331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

MRIT64 updated TIKA-331:
------------------------

Attachment: Parsing_Result1.txt
test1.pdf

test1.pdf is a PDF file including Windings characters. Some are commonly used by people, others less fequently.

Parsing_result1.txt is the text file produced by Tika.

> Windings font recognition in Tika parsing + spacing issue
> ---------------------------------------------------------
>
> Key: TIKA-331
> URL: https://issues.apache.org/jira/browse/TIKA-331
> Project: Tika
> Issue Type: Wish
> Components: parser
> Affects Versions: 0.4
> Environment: Windows XP / Java JDK 1.6.0_15
> Reporter: MRIT64
> Attachments: Parsing_Result1.txt, test1.pdf
>
>
> I have PDF files that include some characters in Windings font.
> Tika parser replaces them with some Unicode characters that have nothing to do with the original, and, in some cases, replaces them with alphabetic characters (that is normal regarding these characters codes).
> Would it be possible to improve the parsing and remplace these characters with more accurate Unicode characters ?
> (see http://www.alanwood.net/demos/wingdings.html for possible correspondences).
> I will attach examples files when this issue will be created (would it be possible to attach files directly when creating issues ?)

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (TIKA-331) Windings font recognition in Tika parsing + spacing issue

2009-11-24T10:42:39Z

Windings font recognition in Tika parsing + spacing issue
---------------------------------------------------------

Key: TIKA-331
URL: https://issues.apache.org/jira/browse/TIKA-331
Project: Tika
Issue Type: Wish
Components: parser
Affects Versions: 0.4
Environment: Windows XP / Java JDK 1.6.0_15
Reporter: MRIT64

I have PDF files that include some characters in Windings font.
Tika parser replaces them with some Unicode characters that have nothing to do with the original, and, in some cases, replaces them with alphabetic characters (that is normal regarding these characters codes).
Would it be possible to improve the parsing and remplace these characters with more accurate Unicode characters ?
(see http://www.alanwood.net/demos/wingdings.html for possible correspondences).

I will attach examples files when this issue will be created (would it be possible to attach files directly when creating issues ?)

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Re: [ANNOUNCE] Apache Tika 0.5 Released

2009-11-24T07:45:49Z

Hi Chris,

about an hour ago it showed the old version now it shows the 0.5 release with the correct http://www.apache.org/dist/lucene/tika/CHANGES-0.5.txt file...

> > http://lucene.apache.org/tika/project-summary.html
> > show 0.5-SNAPSHOT instead 0.5
>
> Strangely enough, for me this shows 0.4:
>
> Build Information
> Field Value
> GroupId org.apache.tika
> ArtifactId tika-site
> Version 0.4
> Type pom

Yeah...that's what i would like to ask for? 0.4 ..i know that the site is a different module (POM) but shouldn't it be in sync with the rest of the package.

And now i found an other point which seemed to be not correct or may be a little bit confusing...

http://lucene.apache.org/tika/source-repository.html

The URL for the repository is given with:
http://svn.apache.org/repos/asf/maven/pom/tags/apache-4/tika-parent/tika-site
(Click on the Web-Access gives an error message as expected)

Hm...in my opinion it should be more or less like the following:
http://svn.apache.org/repos/asf/lucene/tika/tags/0.5/

Kind regards
Karl Heinz Marbaise
--
MfG
Karl Heinz Marbaise
--
SoftwareEntwicklung Beratung Schulung Tel.: +49 (0) 2405 / 415 893
Dipl.Ing.(FH) Karl Heinz Marbaise ICQ#: 135949029
Hauptstrasse 177 USt.IdNr: DE191347579
52146 Würselen http://www.soebes.de

Re: [ANNOUNCE] Apache Tika 0.5 Released

2009-11-24T07:06:00Z

Hi Karl,

>
> the given URL (http://repo1.maven.org/maven2/org/apache/tika/0.5/) is not
> ok...cause it should be...
>
> URL: http://repo1.maven.org/maven2/org/apache/tika/
>
> URL/tika-app/0.5
> URL/tika-parent/0.5
> URL/tika-core/0.5
> URL/tika-parsers/0.5
>

Yep, sorry about that < I had a typo on my pasted URL from a prior release
announcement (the package structure for Tika has since changed).

>
> an other short information is that at the moment the Web-Site is not
> up-to-date...
>
> http://lucene.apache.org/tika/download.html
> shows 0.4 instead....

The download page shows correctly for me:

Apache Tika 0.5 is now available. See the CHANGES.txt
<http://www.apache.org/dist/lucene/tika/CHANGES-0.5.txt> file for more
information on the list of updates in this initial release.
* apache-tika-0.5-src.zip
<http://www.apache.org/dyn/closer.cgi/lucene/tika/apache-tika-0.5-src.zip>
(PGP <http://www.apache.org/dist/lucene/tika/apache-tika-0.5-src.zip.asc> )
Apache Tika releases are available under the Apache License, Version 2.0
<http://www.apache.org/licenses/LICENSE-2.0> . See the NOTICE.txt file
contained in each release artifact for applicable copyright attribution
notices.

>
>
>
>
> http://lucene.apache.org/tika/project-summary.html
> show 0.5-SNAPSHOT instead 0.5

Strangely enough, for me this shows 0.4:

Build Information
Field Value
GroupId org.apache.tika
ArtifactId tika-site
Version 0.4
Type pom

I'm not sure what the problem is on this one -- these pages should be
generated automatically every night by Jukka's site generation script after
there are changes to the site portion of Tika SVN.

>
> May be it took some time to update the tika web-site so I'm a little bit to
> early...;-)

Nope, you are fine, thanks for the pointers, glad you are looking out!

Cheers,
Chris

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: Chris.Mattmann@...
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Re: [ANNOUNCE] Apache Tika 0.5 Released

2009-11-24T02:50:20Z

Hi,

On Tue, Nov 24, 2009 at 11:02 AM, Karl Heinz Marbaise <khmarbaise@...> wrote:
> an other short information is that at the moment the Web-Site is not up-to-date...
>
> http://lucene.apache.org/tika/download.html
> shows 0.4 instead....
> http://lucene.apache.org/tika/project-summary.html
> show 0.5-SNAPSHOT instead 0.5

Bugger, this must be a result of the recent problems with the Hudson
server. I guess it was restored from an older backup and thus our site
deployment scripts ended up reverting the site back to a previous
state. :-(

I've just regenerated and -deployed the site.

BR,

Jukka Zitting

Re: [ANNOUNCE] Apache Tika 0.5 Released

2009-11-24T02:02:41Z

Hi there,

the given URL (http://repo1.maven.org/maven2/org/apache/tika/0.5/) is not ok...cause it should be...

URL: http://repo1.maven.org/maven2/org/apache/tika/

URL/tika-app/0.5
URL/tika-parent/0.5
URL/tika-core/0.5
URL/tika-parsers/0.5

an other short information is that at the moment the Web-Site is not up-to-date...

http://lucene.apache.org/tika/download.html
shows 0.4 instead....
http://lucene.apache.org/tika/project-summary.html
show 0.5-SNAPSHOT instead 0.5

May be it took some time to update the tika web-site so I'm a little bit to early...;-)
Kind regards
Karl Heinz Marbaise
--
MfG
Karl Heinz Marbaise
--
SoftwareEntwicklung Beratung Schulung Tel.: +49 (0) 2405 / 415 893
Dipl.Ing.(FH) Karl Heinz Marbaise ICQ#: 135949029
Hauptstrasse 177 USt.IdNr: DE191347579
52146 Würselen http://www.soebes.de

Re: [ANNOUNCE] Apache Tika 0.5 Released

2009-11-23T07:39:50Z

Hey Steen,

It's already available on: http://repo1.maven.org/maven2/org/apache/tika/0.5/ and should be in the other repo shortly...

Cheers,
Chris

On 11/23/09 1:15 AM, "Steen Manniche" <stm@...> wrote:

Den Sun, Nov 22, 2009 at 07:50:47AM -0800 skrev Mattmann, Chris A (388J):

> (...apologies for the cross posting...)
>
> The Apache Lucene project is pleased to announce the release of Apache Tika
> 0.5. The release contents have been pushed out to the main Apache release
> site and the m2 ibiblio sync, so the releases should be available as soon as
> the mirrors get the syncs.
>
> Apache Tika, a subproject of Apache Lucene, is a toolkit for detecting and
> extracting metadata and structured text content from various documents using
> existing parser libraries.
>
> Apache Tika 0.5 contains a number of improvements and bug fixes. Details can
> be found in the changes file:
>
> http://www.apache.org/dist/lucene/tika/CHANGES-0.5.txt
>
> Apache Tika is available in source form from the following download page:
> http://www.apache.org/dyn/closer.cgi/lucene/tika/apache-tika-0.5-src.zip
>
> Apache Tika is also available in binary form or for use using Maven 2 from
> the Central Maven Repositories:
> http://repo1.maven.org/maven2/org/apache/tika/0.5/
> http://mirrors.ibiblio.org/pub/mirrors/maven2/org/apache/tika/0.5/

The above link and any of the source access links on the page
http://lucene.apache.org/tika/source-repository.html are broken at the
moment. Where should I point my wget at? Or should I just wait a
while?

Best regards and thanks for the effort,
Steen Manniche

>
> In the initial 48 hours, the release may not be available on all mirrors.
> When downloading from a mirror site, please remember to verify the downloads
> using signatures found on the Apache site:
> http://www.apache.org/dist/lucene/tika/KEYS-0.5.txt
>
> For more information on Apache Tika, visit the project home page:
> http://lucene.apache.org/tika
>
> -- Chris Mattmann (on behalf of the Apache Lucene community)
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: Chris.Mattmann@...
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: Chris.Mattmann@...
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[jira] Resolved: (TIKA-330) Better HWP (Hangul Word Processor) detection pattern

2009-11-23T03:29:39Z

[ https://issues.apache.org/jira/browse/TIKA-330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jukka Zitting resolved TIKA-330.
--------------------------------

Resolution: Fixed
Fix Version/s: 0.6

Fixed in revision 883306.

> Better HWP (Hangul Word Processor) detection pattern
> ----------------------------------------------------
>
> Key: TIKA-330
> URL: https://issues.apache.org/jira/browse/TIKA-330
> Project: Tika
> Issue Type: Improvement
> Components: mime
> Reporter: Jukka Zitting
> Assignee: Jukka Zitting
> Priority: Minor
> Fix For: 0.6
>
>
> The current magic byte pattern we have for the HWP (Hangul Word Processor, application/x-hwp) file format matches also the test-outlook.msg test file we have. I looked for a better detection pattern and found one from OpenOffice.org.
> The hwpfilter/source/hwpfile.cpp file suggests that all HWP files start with the signature string "HWP Document File V", so I'll change the detection pattern accordingly.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (TIKA-330) Better HWP (Hangul Word Processor) detection pattern

2009-11-23T03:27:39Z

Better HWP (Hangul Word Processor) detection pattern
----------------------------------------------------

Key: TIKA-330
URL: https://issues.apache.org/jira/browse/TIKA-330
Project: Tika
Issue Type: Improvement
Components: mime
Reporter: Jukka Zitting
Assignee: Jukka Zitting
Priority: Minor

The current magic byte pattern we have for the HWP (Hangul Word Processor, application/x-hwp) file format matches also the test-outlook.msg test file we have. I looked for a better detection pattern and found one from OpenOffice.org.

The hwpfilter/source/hwpfile.cpp file suggests that all HWP files start with the signature string "HWP Document File V", so I'll change the detection pattern accordingly.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Re: [ANNOUNCE] Apache Tika 0.5 Released

2009-11-23T01:15:39Z

Den Sun, Nov 22, 2009 at 07:50:47AM -0800 skrev Mattmann, Chris A (388J):

[jira] Commented: (TIKA-309) Mime type application/rdf+xml not correctly detected

2009-11-22T21:38:39Z

[ https://issues.apache.org/jira/browse/TIKA-309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12781311#action_12781311 ]

Yuan-Fang Li commented on TIKA-309:
-----------------------------------

Hi Chris, Jukka,

Yes, the Tika tests are passing for me. However, my test for one of the ontologies ("http://www.w3.org/2002/07/owl#") is still failing, and here is why.

In test tika-core/src/test/java/org/apache/tika/mime/MimeDetectionTest.java, the method testUrl(String expected, String url, String file) is actually testing the content in the file named "file" with the url being a clue for the detection. My test, however, opens an input stream on the actual url and use that to detect the mime type. For the above URL, tika is testing against the file named "test-difficult-rdf2.xml". The only difference I can see between this file and the actual content of the URl is the one line at the top: "<?xml version='1.0' encoding='ISO-8859-1'?>". This line is present in the tika test file but not in the URL.

So. if you remove/comment out that line from "test-difficult-rdf2.xml" and run the following maven command to run the test: mvn -Dtest=MimeDetectionTest test, it will fail. Or, you could use the following test case to test against the real URL.

@Test
public void testRDFStreamMimeType() throws IOException {
URL url = new URL("http://www.w3.org/2002/07/owl#");
final InputStream stream = new BufferedInputStream(url.openStream());
try {
MimeTypes mimeTypes = TikaConfig.getDefaultConfig().getMimeRepository();
Metadata metadata = new Metadata();
String mime = mimeTypes.detect(stream, metadata).toString();
assertEquals("application/rdf+xml", mime);
} finally {
stream.close();
}
}

Cheers
Yuan-Fang

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Hudson build is back to normal: Tika-trunk #231

2009-11-22T18:16:57Z

See <http://hudson.zones.apache.org/hudson/job/Tika-trunk/231/>

Build failed in Hudson: Tika-trunk #230

2009-11-22T16:17:10Z

See <http://hudson.zones.apache.org/hudson/job/Tika-trunk/230/>

------------------------------------------
Started by user jukka
Building remotely on minerva.apache.org (Ubuntu)
Updating http://svn.apache.org/repos/asf/lucene/tika/trunk
At revision 883196
no change for http://svn.apache.org/repos/asf/lucene/tika/trunk since the previous build
Parsing POMs
Exception in thread "main" java.lang.NoClassDefFoundError: hudson/maven/agent/Main
ERROR: Failed to launch Maven. Exit code = 1