Nabble - Apache Tika - Development

[jira] Commented: (TIKA-94) Speech recognition

2009-11-09T13:28:32Z

[ https://issues.apache.org/jira/browse/TIKA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775155#action_12775155 ]

David Woollard commented on TIKA-94:
------------------------------------

I've used Sphinx for a couple of projects, to great success, but its probably not what you are looking for here... fundamentally, there a two problems with general speech recognition. The first is training. Most speech recognition systems are trained in some way. Sphinx is "pre-trained" with a number of different english-language accents (mostly American), or you can retrain it yourself. The second problem is a grammar. In order to recognize a good collection of words in the english language, you need to provide a very large grammar (60,000ish words in the english language comes out to be a 200Mb+ file... probably many orders of magnitude larger than you would want in a dependency).

Long and short of it is that sphinx is great out of the box if you speak with a Mid-western American accent (as I happen to) and you are detailing with a command and control situation where you can produce a small, tailored grammar.

This is something that could be done, but i would advocate that it be left outside the core parsers.

> Speech recognition
> ------------------
>
> Key: TIKA-94
> URL: https://issues.apache.org/jira/browse/TIKA-94
> Project: Tika
> Issue Type: New Feature
> Components: parser
> Reporter: Jukka Zitting
> Priority: Minor
>
> Like OCR for image files (TIKA-93), we could try using speech recognition to extract text content (where available) from audio (and video!) files.
> The CMU Sphinx engine (http://cmusphinx.sourceforge.net/) looks promising and comes with a friendly license.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (TIKA-318) Upgrade nekohtml dependency from 1.9.9 to 1.9.13

2009-11-07T10:59:32Z

[ https://issues.apache.org/jira/browse/TIKA-318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774658#action_12774658 ]

Benson Margulies commented on TIKA-318:
---------------------------------------

Jukka took out nekkohtml in favor of TagSoup. You could close this.

> Upgrade nekohtml dependency from 1.9.9 to 1.9.13
> ------------------------------------------------
>
> Key: TIKA-318
> URL: https://issues.apache.org/jira/browse/TIKA-318
> Project: Tika
> Issue Type: Task
> Components: packaging
> Affects Versions: 0.4
> Reporter: Attila Király
> Priority: Minor
>
> It seems tika is still using an old version of nekohtml. It could be upgraded to 1.9.13.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (TIKA-316) Parsing Visio diagrams with tika-app causes TikaException (Found a chunk with a negative length)

2009-11-07T10:03:32Z

[ https://issues.apache.org/jira/browse/TIKA-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris A. Mattmann updated TIKA-316:
-----------------------------------

Component/s: cli

- set fix component

> Parsing Visio diagrams with tika-app causes TikaException (Found a chunk with a negative length)
> ------------------------------------------------------------------------------------------------
>
> Key: TIKA-316
> URL: https://issues.apache.org/jira/browse/TIKA-316
> Project: Tika
> Issue Type: Bug
> Components: cli
> Affects Versions: 0.4, 0.5
> Environment: Windows Server 2003 SP2, JRE 1.6.0_16, tika-app, Visio 2003
> Reporter: Mike Hays
> Attachments: repro-TIKA-316.vsd
>
>
> tika-app (0.4 and 0.5 nightly) return the following when attempting to parse a Visio 2003 file (other versions may be affected):
> Exception in thread "main" org.apache.tika.exception.TikaException: Unexpected RuntimeException from org.apache.tika.parser.microsoft.OfficeParser@145e044
> at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:123)
> at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:103)
> at org.apache.tika.cli.TikaCLI.process(TikaCLI.java:176)
> at org.apache.tika.cli.TikaCLI.main(TikaCLI.java:63)
> Caused by: java.lang.IllegalArgumentException: Found a chunk with a negative length, which isn't allowed
> at org.apache.poi.hdgf.chunks.ChunkFactory.createChunk(ChunkFactory.java:120)
> at org.apache.poi.hdgf.streams.ChunkStream.findChunks(ChunkStream.java:59)
> at org.apache.poi.hdgf.streams.PointerContainingStream.findChildren(PointerContainingStream.java:93)
> at org.apache.poi.hdgf.streams.PointerContainingStream.findChildren(PointerContainingStream.java:100)
> at org.apache.poi.hdgf.streams.PointerContainingStream.findChildren(PointerContainingStream.java:100)
> at org.apache.poi.hdgf.HDGFDiagram.<init>(HDGFDiagram.java:95)
> at org.apache.poi.hdgf.extractor.VisioTextExtractor.<init>(VisioTextExtractor.java:52)
> at org.apache.poi.hdgf.extractor.VisioTextExtractor.<init>(VisioTextExtractor.java:49)
> at org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:118)
> at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:121)
> ... 3 more

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (TIKA-318) Upgrade nekohtml dependency from 1.9.9 to 1.9.13

2009-11-07T10:03:32Z

[ https://issues.apache.org/jira/browse/TIKA-318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris A. Mattmann updated TIKA-318:
-----------------------------------

Component/s: packaging

- set fix component

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (TIKA-298) CompositeParser.getParser() should use mimetype hierarchy when falling back

2009-11-07T10:03:32Z

[ https://issues.apache.org/jira/browse/TIKA-298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris A. Mattmann updated TIKA-298:
-----------------------------------

Component/s: parser

- set fix component

> CompositeParser.getParser() should use mimetype hierarchy when falling back
> ---------------------------------------------------------------------------
>
> Key: TIKA-298
> URL: https://issues.apache.org/jira/browse/TIKA-298
> Project: Tika
> Issue Type: Improvement
> Components: parser
> Affects Versions: 0.4
> Reporter: Ken Krugler
>
> CompositeParser.getParser() doesn't use supertypes when falling back - if it can't get a parser for the exact mimetype, then it goes
> straight to the fallback parser.
> So, for example, if the file mimetype is application/<whatever>+xml, and no parser exists for it, then you get the default "do nothing" parser versus the XML parser.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (TIKA-315) Tika appears to skip over an entire section of a Microsoft Word Document

2009-11-07T10:03:32Z

[ https://issues.apache.org/jira/browse/TIKA-315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris A. Mattmann updated TIKA-315:
-----------------------------------

Component/s: parser

- set fix component

> Tika appears to skip over an entire section of a Microsoft Word Document
> ------------------------------------------------------------------------
>
> Key: TIKA-315
> URL: https://issues.apache.org/jira/browse/TIKA-315
> Project: Tika
> Issue Type: Bug
> Components: parser
> Affects Versions: 0.4
> Environment: Microsoft Windows Vista 32 bit; Apache tika 0.4 release; used the -gui and the command line -t option.
> Reporter: Sanjeev Rao
> Attachments: Biolink07_fromTika0.4.txt
>
>
> I saved this MS word file http://mingo.info-science.uiowa.edu/padmini/Papers/Biolink07.doc to my 32 bit Vista desktop. Then I tried both the command line -t option (output attached) and the GUI option. In either case, a large section of the word got lost.. From the bottom, you can scroll up and look for reference #4 which exists but references 1 - 3 don't.. You will also find that a page or so of the original document is missing! It is not clear what about this document caused this behavior.
> I am trying to use tika to convert content from the web.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (TIKA-318) Upgrade nekohtml dependency from 1.9.9 to 1.9.13

2009-11-07T09:31:32Z

Upgrade nekohtml dependency from 1.9.9 to 1.9.13
------------------------------------------------

Key: TIKA-318
URL: https://issues.apache.org/jira/browse/TIKA-318
Project: Tika
Issue Type: Task
Affects Versions: 0.4
Reporter: Attila Király
Priority: Minor

It seems tika is still using an old version of nekohtml. It could be upgraded to 1.9.13.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (TIKA-317) Annotation-based Tika configuration

2009-11-06T20:59:32Z

[ https://issues.apache.org/jira/browse/TIKA-317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jukka Zitting updated TIKA-317:
-------------------------------

Fix Version/s: (was: 0.5)

Postponing to after 0.5

> Annotation-based Tika configuration
> -----------------------------------
>
> Key: TIKA-317
> URL: https://issues.apache.org/jira/browse/TIKA-317
> Project: Tika
> Issue Type: Improvement
> Components: parser
> Reporter: Jukka Zitting
> Assignee: Jukka Zitting
> Priority: Minor
>
> I'd like to simplify Tika configuration and make it easier to customize by pushing the information in tika-config.xml to Parser annotations and Java SPI service files.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (TIKA-209) Language detection is weak.

2009-11-06T20:35:41Z

[ https://issues.apache.org/jira/browse/TIKA-209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jukka Zitting resolved TIKA-209.
--------------------------------

Resolution: Fixed

I have refactored and simplified the language identifier code to better meet the needs of Tika. Most notably I fixed the ngram length to three characters to reduce the size of the language profile files and to make the ngram classes simpler.

AutoDetectParser now automatically attempts to detect the document language and sets the Metadata.LANGUAGE property if a reasonably certain language profile match is found.

> Language detection is weak.
> ---------------------------
>
> Key: TIKA-209
> URL: https://issues.apache.org/jira/browse/TIKA-209
> Project: Tika
> Issue Type: Bug
> Components: languageidentifier
> Affects Versions: 0.3
> Reporter: Robert Newson
> Fix For: 0.5
>
>
> in org.apache.tika.utils.Utils the getUTF8Reader method assigns a language determination without checking the confidence rating from ICU's CharsetDetector.
> Please add a configurable level (0-100);
> if (language != null && match.getConfidence() > THRESHOLD) {
> metadata.set(Metadata.CONTENT_LANGUAGE, match.getLanguage());
> metadata.set(Metadata.LANGUAGE, match.getLanguage());
> }
> Obviously using charset to imply language is generally weak but it would be sufficient if the confidence threshold was controlled. Today, the text "hello" is tagged as French, for example.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (TIKA-275) Parse context

2009-11-06T20:31:43Z

[ https://issues.apache.org/jira/browse/TIKA-275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jukka Zitting resolved TIKA-275.
--------------------------------

Resolution: Fixed

This is now done. I'll add the following change log entry as soon as svn.apache.org is back online:

* A new parse context argument was added to the Parser.parse() method.
This context map can be used to pass things like a delegate parser or
other settings to the parsing process. The previous parse() method
signature has been deprecated and will be removed in Tika 1.0. (TIKA-275)

> Parse context
> -------------
>
> Key: TIKA-275
> URL: https://issues.apache.org/jira/browse/TIKA-275
> Project: Tika
> Issue Type: New Feature
> Components: parser
> Reporter: Jukka Zitting
> Assignee: Jukka Zitting
> Fix For: 0.5
>
>
> As discussed on dev@ [1], I'd like to add an extra parse context argument to the Parser.parse() method.
> [1] http://markmail.org/message/yfgngwhbm6ra42ty

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (TIKA-309) Mime type application/rdf+xml not correctly detected

2009-11-06T20:27:41Z

[ https://issues.apache.org/jira/browse/TIKA-309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774551#action_12774551 ]

Chris A. Mattmann commented on TIKA-309:
----------------------------------------

Hey Guys, I think we just need another line in the tika-mimetypes.xml file for this. I'll take a crack at it, if there are no objections. Thanks!

> Mime type application/rdf+xml not correctly detected
> ----------------------------------------------------
>
> Key: TIKA-309
> URL: https://issues.apache.org/jira/browse/TIKA-309
> Project: Tika
> Issue Type: Bug
> Components: mime
> Affects Versions: 0.5
> Reporter: Yuan-Fang Li
> Assignee: Chris A. Mattmann
> Priority: Minor
> Fix For: 0.5
>
>
> Mime type detector using AutoDetectParser and Metadata returns "application/xml" for the URL http://www.w3.org/2002/07/owl#, where it should be "application/rdf+xml". The correct mime type is also suggested here: http://www.w3.org/TR/owl-ref/#MIMEType.
> P.S., Tika was downloaded from svn and built with Maven last week.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Assigned: (TIKA-309) Mime type application/rdf+xml not correctly detected

2009-11-06T20:27:41Z

[ https://issues.apache.org/jira/browse/TIKA-309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris A. Mattmann reassigned TIKA-309:
--------------------------------------

Assignee: Chris A. Mattmann (was: Jukka Zitting)

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

0.5 release

2009-11-06T20:25:37Z

Hey Guys,

I'll be working over the next week to prepare the 0.5 RC. I'm looking
forward to using the new process that Jukka put together that involves
repository.apache.org.

If there are any objections to 0.5 RC going up in the next week or so,
please speak up, otherwise, I'll continue pushing towards that as planned.

Thanks!

Cheers,
Chris

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: Chris.Mattmann@...
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[jira] Resolved: (TIKA-314) Initial support for JPEG EXIF metadata extraction

2009-11-06T20:23:41Z

[ https://issues.apache.org/jira/browse/TIKA-314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris A. Mattmann resolved TIKA-314.
------------------------------------

Resolution: Fixed

I think this fix is fine as is -- as Jukka suggested, we may move to a diff package, but if so, we can open a new issue for it at that time. I'm marking this as fixed. Thanks, Jukka, and thanks Maxim for the patch!

> Initial support for JPEG EXIF metadata extraction
> -------------------------------------------------
>
> Key: TIKA-314
> URL: https://issues.apache.org/jira/browse/TIKA-314
> Project: Tika
> Issue Type: Improvement
> Components: parser
> Reporter: Maxim Valyanskiy
> Assignee: Jukka Zitting
> Fix For: 0.5
>
> Attachments: initial-support-for-jpeg-exif-extraction.patch, testJPEG_EXIF.jpg
>
>
> This patch adds initial support for JPEG EXIF metadata extraction

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Assigned: (TIKA-314) Initial support for JPEG EXIF metadata extraction

2009-11-06T20:21:41Z

[ https://issues.apache.org/jira/browse/TIKA-314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris A. Mattmann reassigned TIKA-314:
--------------------------------------

Assignee: Jukka Zitting (was: Chris A. Mattmann)

Jukka already fixed this -- I thought it was unassigned and therefore needed work but since it's likely fixed, I'll leave the assignee as Jukka.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Assigned: (TIKA-314) Initial support for JPEG EXIF metadata extraction

2009-11-06T20:21:41Z

> Initial support for JPEG EXIF metadata extraction
> -------------------------------------------------
>
> Key: TIKA-314
> URL: https://issues.apache.org/jira/browse/TIKA-314
> Project: Tika
> Issue Type: Improvement
> Components: parser
> Reporter: Maxim Valyanskiy
> Assignee: Chris A. Mattmann
> Fix For: 0.5
>
> Attachments: initial-support-for-jpeg-exif-extraction.patch, testJPEG_EXIF.jpg
>
>
> This patch adds initial support for JPEG EXIF metadata extraction

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (TIKA-317) Annotation-based Tika configuration

2009-11-06T20:11:41Z

[ https://issues.apache.org/jira/browse/TIKA-317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774546#action_12774546 ]

Jukka Zitting commented on TIKA-317:
------------------------------------

Re: co-locating metadata with code; Doing so makes it easier to support multiple different configuration mechanisms (default Tika config, programmatic configuration, OSGi services, IoC containers, etc.) as you don't need to duplicate the media type lists for each different way of configuring things.

Re: tika-config.xml vs. META-INF/services/...; The service provider mechanism [1] makes it easy to add custom parser implementations without having to maintain a separate copy of the full Tika configuration file. You could for example create a my-custom-parsers.jar file with a META-INF/services/o.a.tika.parser.Parser file that lists only your custom parser classes. When you add that jar to the classpath, Tika would then automatically pick up those parsers in addition to the standard parser classes from the tika-parsers jar.

[1] http://java.sun.com/j2se/1.5.0/docs/guide/jar/jar.html#Service Provider

> Annotation-based Tika configuration
> -----------------------------------
>
> Key: TIKA-317
> URL: https://issues.apache.org/jira/browse/TIKA-317
> Project: Tika
> Issue Type: Improvement
> Components: parser
> Reporter: Jukka Zitting
> Assignee: Jukka Zitting
> Priority: Minor
> Fix For: 0.5
>
>
> I'd like to simplify Tika configuration and make it easier to customize by pushing the information in tika-config.xml to Parser annotations and Java SPI service files.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (TIKA-317) Annotation-based Tika configuration

2009-11-06T19:47:41Z

[ https://issues.apache.org/jira/browse/TIKA-317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774543#action_12774543 ]

Chris A. Mattmann commented on TIKA-317:
----------------------------------------

Thanks for the more detail Jukka, but I fail to see how co-locating metadata with code (as in the case of JDK annotations) is any better of a mechanism that separating out such configuration into an XML file, Also, what is the difference between having the information in the tika-config.xml file versus locating (some of) that information int a META-INF/services/o.a.tika.parser.Parser file? I guess I just need to understand more b/c I'm missing something?

> Annotation-based Tika configuration
> -----------------------------------
>
> Key: TIKA-317
> URL: https://issues.apache.org/jira/browse/TIKA-317
> Project: Tika
> Issue Type: Improvement
> Components: parser
> Reporter: Jukka Zitting
> Assignee: Jukka Zitting
> Priority: Minor
> Fix For: 0.5
>
>
> I'd like to simplify Tika configuration and make it easier to customize by pushing the information in tika-config.xml to Parser annotations and Java SPI service files.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (TIKA-317) Annotation-based Tika configuration

2009-11-06T19:39:41Z

[ https://issues.apache.org/jira/browse/TIKA-317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774542#action_12774542 ]

Jukka Zitting commented on TIKA-317:
------------------------------------

As Benson mentioned, a pretty typical deployment scenario is one where you want to extend Tika with a few custom Parser classes. Currently you'd either need to maintain a custom version of the full configuration file, or do some CompositeParser magic to inject your custom parsers at runtime. Neither option is ideal.

Another concern of mine is that the current configuration mechanism disconnects the list of supported media types from the parser implementation class. It would be better if that list was maintained in the same Java source file instead of in the XML configuration.

Thinking further, there's some interest in making Tika easy to use in more dynamic environments like an OSGi container where new parser components may be added to or removed from the system at any time. A static configuration file does not work that well in such situations.

So my idea is to move the list of media types supported by a Parser class to a class annotation (or perhaps a getSupportedTypes() method that would work better with composite parsers) and replace the tika-config.xml file with a META-INF/services/org.apache.tika.parser.Parser file that simply lists all the Parser implementations within that jar file.

> Annotation-based Tika configuration
> -----------------------------------
>
> Key: TIKA-317
> URL: https://issues.apache.org/jira/browse/TIKA-317
> Project: Tika
> Issue Type: Improvement
> Components: parser
> Reporter: Jukka Zitting
> Assignee: Jukka Zitting
> Priority: Minor
> Fix For: 0.5
>
>
> I'd like to simplify Tika configuration and make it easier to customize by pushing the information in tika-config.xml to Parser annotations and Java SPI service files.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (TIKA-317) Annotation-based Tika configuration

2009-11-06T17:55:41Z

[ https://issues.apache.org/jira/browse/TIKA-317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774534#action_12774534 ]

Benson Margulies commented on TIKA-317:
---------------------------------------

I'm with Jukka. I needed to replace one processor. Having to copy and modify the xml file, and then forever maintain my mutant version as new Tika releases change the rest of the contents that I don't want to change, is not a good prospect.

> Annotation-based Tika configuration
> -----------------------------------
>
> Key: TIKA-317
> URL: https://issues.apache.org/jira/browse/TIKA-317
> Project: Tika
> Issue Type: Improvement
> Components: parser
> Reporter: Jukka Zitting
> Assignee: Jukka Zitting
> Priority: Minor
> Fix For: 0.5
>
>
> I'd like to simplify Tika configuration and make it easier to customize by pushing the information in tika-config.xml to Parser annotations and Java SPI service files.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (TIKA-317) Annotation-based Tika configuration

2009-11-06T17:05:32Z

[ https://issues.apache.org/jira/browse/TIKA-317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774521#action_12774521 ]

Chris A. Mattmann commented on TIKA-317:
----------------------------------------

Hey Jukka: could you explain how this will be simpler? I, personally, like the tika-config.xml file. Details, please :)

> Annotation-based Tika configuration
> -----------------------------------
>
> Key: TIKA-317
> URL: https://issues.apache.org/jira/browse/TIKA-317
> Project: Tika
> Issue Type: Improvement
> Components: parser
> Reporter: Jukka Zitting
> Assignee: Jukka Zitting
> Priority: Minor
> Fix For: 0.5
>
>
> I'd like to simplify Tika configuration and make it easier to customize by pushing the information in tika-config.xml to Parser annotations and Java SPI service files.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (TIKA-317) Annotation-based Tika configuration

2009-11-06T16:59:32Z

Annotation-based Tika configuration
-----------------------------------

Key: TIKA-317
URL: https://issues.apache.org/jira/browse/TIKA-317
Project: Tika
Issue Type: Improvement
Components: parser
Reporter: Jukka Zitting
Assignee: Jukka Zitting
Priority: Minor
Fix For: 0.5

I'd like to simplify Tika configuration and make it easier to customize by pushing the information in tika-config.xml to Parser annotations and Java SPI service files.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Reopened: (TIKA-309) Mime type application/rdf+xml not correctly detected

2009-11-04T16:03:32Z

[ https://issues.apache.org/jira/browse/TIKA-309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yuan-Fang Li reopened TIKA-309:
-------------------------------

This fix had worked for me till yesterday. When I updated to the latest version (829668) from svn, my test cases on application/rdf+xml mimetype failed again, for URLs "http://www.ai.sri.com/daml/services/owl-s/1.2/Process.owl" and "http://www.w3.org/2002/07/owl#". The mimetype returned is "application/xml" for the first one and "text/html" for the second one. Hence I'm reopening this issue.

> Mime type application/rdf+xml not correctly detected
> ----------------------------------------------------
>
> Key: TIKA-309
> URL: https://issues.apache.org/jira/browse/TIKA-309
> Project: Tika
> Issue Type: Bug
> Components: mime
> Affects Versions: 0.5
> Reporter: Yuan-Fang Li
> Assignee: Jukka Zitting
> Priority: Minor
> Fix For: 0.5
>
>
> Mime type detector using AutoDetectParser and Metadata returns "application/xml" for the URL http://www.w3.org/2002/07/owl#, where it should be "application/rdf+xml". The correct mime type is also suggested here: http://www.w3.org/TR/owl-ref/#MIMEType.
> P.S., Tika was downloaded from svn and built with Maven last week.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Re: Free live video streaming of ApacheCon US 2009

2009-11-04T06:55:16Z

Thanks a lot.

This will be very helpful to me.

As I am not able to attend.

On Wed, Nov 4, 2009 at 8:25 AM, Michael McCandless <
lucene@...> wrote:

> Team,
>
> For those Lucene fanatics not in Oakland this week for ApacheCon US,
> don't miss the FREE live video streaming, starting today:
>
> http://streaming.linux-magazin.de/en/program-apachecon-us-2009.htm
>
> Note that there are many talks available, covering Apache Hadoop,
> Apache HTTPD, Lucene, as well as the Apache Pioneer's Panel and
> keynote presentations.
>
> Lucene's track is this Friday (NOTE these times are UTC -- use
> http://www.timeanddate.com to map to your time zone):
>
> 17:00 Implementing an Information Retrieval Framework for an
> Organizational Repository, Sithu D Sudarsan
>
> 18:00 Apache Mahout - Going from raw data to information
> Isabel Drost
>
> 19:15 MIME Magic with Apache Tika
> Jukka Zitting
>
> 20:15 Keynote: How Open Source Developers Can (Still!) Save The World
> Brian Behlendorf
>
> 22:00 Building Intelligent Search Applications with the Lucene
> Ecosystem, Ted Dunning
>
> 23:00 Realtime Search
> Jason Rutherglen
>
> Happy viewing,
>
> Mike
>

--
"Good Enough" is not good enough.
To give anything less than your best is to sacrifice the gift.
Quality First. Measure Twice. Cut Once.

Free live video streaming of ApacheCon US 2009

2009-11-04T05:25:25Z

Team,

For those Lucene fanatics not in Oakland this week for ApacheCon US,
don't miss the FREE live video streaming, starting today:

http://streaming.linux-magazin.de/en/program-apachecon-us-2009.htm

Note that there are many talks available, covering Apache Hadoop,
Apache HTTPD, Lucene, as well as the Apache Pioneer's Panel and
keynote presentations.

Lucene's track is this Friday (NOTE these times are UTC -- use
http://www.timeanddate.com to map to your time zone):

17:00 Implementing an Information Retrieval Framework for an
Organizational Repository, Sithu D Sudarsan

18:00 Apache Mahout - Going from raw data to information
Isabel Drost

19:15 MIME Magic with Apache Tika
Jukka Zitting

20:15 Keynote: How Open Source Developers Can (Still!) Save The World
Brian Behlendorf

22:00 Building Intelligent Search Applications with the Lucene
Ecosystem, Ted Dunning

23:00 Realtime Search
Jason Rutherglen

Happy viewing,

Mike

[jira] Updated: (TIKA-316) Parsing Visio diagrams with tika-app causes TikaException (Found a chunk with a negative length)

2009-11-03T13:11:32Z

[ https://issues.apache.org/jira/browse/TIKA-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mike Hays updated TIKA-316:
---------------------------

Attachment: repro-TIKA-316.vsd

Run either:
java -jar tika-app-0.4.jar repro-TIKA-316.vsd
java -jar tika-app-0.5-SNAPSHOT.jar repro-TIKA-316.vsd

> Parsing Visio diagrams with tika-app causes TikaException (Found a chunk with a negative length)
> ------------------------------------------------------------------------------------------------
>
> Key: TIKA-316
> URL: https://issues.apache.org/jira/browse/TIKA-316
> Project: Tika
> Issue Type: Bug
> Affects Versions: 0.4, 0.5
> Environment: Windows Server 2003 SP2, JRE 1.6.0_16, tika-app, Visio 2003
> Reporter: Mike Hays
> Attachments: repro-TIKA-316.vsd
>
>
> tika-app (0.4 and 0.5 nightly) return the following when attempting to parse a Visio 2003 file (other versions may be affected):
> Exception in thread "main" org.apache.tika.exception.TikaException: Unexpected RuntimeException from org.apache.tika.parser.microsoft.OfficeParser@145e044
> at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:123)
> at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:103)
> at org.apache.tika.cli.TikaCLI.process(TikaCLI.java:176)
> at org.apache.tika.cli.TikaCLI.main(TikaCLI.java:63)
> Caused by: java.lang.IllegalArgumentException: Found a chunk with a negative length, which isn't allowed
> at org.apache.poi.hdgf.chunks.ChunkFactory.createChunk(ChunkFactory.java:120)
> at org.apache.poi.hdgf.streams.ChunkStream.findChunks(ChunkStream.java:59)
> at org.apache.poi.hdgf.streams.PointerContainingStream.findChildren(PointerContainingStream.java:93)
> at org.apache.poi.hdgf.streams.PointerContainingStream.findChildren(PointerContainingStream.java:100)
> at org.apache.poi.hdgf.streams.PointerContainingStream.findChildren(PointerContainingStream.java:100)
> at org.apache.poi.hdgf.HDGFDiagram.<init>(HDGFDiagram.java:95)
> at org.apache.poi.hdgf.extractor.VisioTextExtractor.<init>(VisioTextExtractor.java:52)
> at org.apache.poi.hdgf.extractor.VisioTextExtractor.<init>(VisioTextExtractor.java:49)
> at org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:118)
> at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:121)
> ... 3 more

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (TIKA-316) Parsing Visio diagrams with tika-app causes TikaException (Found a chunk with a negative length)

2009-11-03T13:07:32Z

Parsing Visio diagrams with tika-app causes TikaException (Found a chunk with a negative length)
------------------------------------------------------------------------------------------------

Key: TIKA-316
URL: https://issues.apache.org/jira/browse/TIKA-316
Project: Tika
Issue Type: Bug
Affects Versions: 0.4, 0.5
Environment: Windows Server 2003 SP2, JRE 1.6.0_16, tika-app, Visio 2003
Reporter: Mike Hays

tika-app (0.4 and 0.5 nightly) return the following when attempting to parse a Visio 2003 file (other versions may be affected):

Exception in thread "main" org.apache.tika.exception.TikaException: Unexpected RuntimeException from org.apache.tika.parser.microsoft.OfficeParser@145e044
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:123)
at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:103)
at org.apache.tika.cli.TikaCLI.process(TikaCLI.java:176)
at org.apache.tika.cli.TikaCLI.main(TikaCLI.java:63)
Caused by: java.lang.IllegalArgumentException: Found a chunk with a negative length, which isn't allowed
at org.apache.poi.hdgf.chunks.ChunkFactory.createChunk(ChunkFactory.java:120)
at org.apache.poi.hdgf.streams.ChunkStream.findChunks(ChunkStream.java:59)
at org.apache.poi.hdgf.streams.PointerContainingStream.findChildren(PointerContainingStream.java:93)
at org.apache.poi.hdgf.streams.PointerContainingStream.findChildren(PointerContainingStream.java:100)
at org.apache.poi.hdgf.streams.PointerContainingStream.findChildren(PointerContainingStream.java:100)
at org.apache.poi.hdgf.HDGFDiagram.<init>(HDGFDiagram.java:95)
at org.apache.poi.hdgf.extractor.VisioTextExtractor.<init>(VisioTextExtractor.java:52)
at org.apache.poi.hdgf.extractor.VisioTextExtractor.<init>(VisioTextExtractor.java:49)
at org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:118)
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:121)
... 3 more

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (TIKA-315) Tika appears to skip over an entire section of a Microsoft Word Document

2009-10-26T09:22:59Z

[ https://issues.apache.org/jira/browse/TIKA-315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjeev Rao updated TIKA-315:
-----------------------------

Description:
I saved this MS word file http://mingo.info-science.uiowa.edu/padmini/Papers/Biolink07.doc to my 32 bit Vista desktop. Then I tried both the command line -t option (output attached) and the GUI option. In either case, a large section of the word got lost.. From the bottom, you can scroll up and look for reference #4 which exists but references 1 - 3 don't.. You will also find that a page or so of the original document is missing! It is not clear what about this document caused this behavior.

I am trying to use tika to convert content from the web.

was:
Using the -gui option, dragged and dropped this file http://mingo.info-science.uiowa.edu/padmini/Papers/Biolink07.doc which was saved on my 32 bit Vista desktop. Attached is the output. (assuming I can attach after I create this issue)

Environment: Microsoft Windows Vista 32 bit; Apache tika 0.4 release; used the -gui and the command line -t option. (was: Microsoft Windows Vista 32 bit; Apache tika 0.4 release; used the -gui option.)

> Tika appears to skip over an entire section of a Microsoft Word Document
> ------------------------------------------------------------------------
>
> Key: TIKA-315
> URL: https://issues.apache.org/jira/browse/TIKA-315
> Project: Tika
> Issue Type: Bug
> Affects Versions: 0.4
> Environment: Microsoft Windows Vista 32 bit; Apache tika 0.4 release; used the -gui and the command line -t option.
> Reporter: Sanjeev Rao
> Attachments: Biolink07_fromTika0.4.txt
>
>
> I saved this MS word file http://mingo.info-science.uiowa.edu/padmini/Papers/Biolink07.doc to my 32 bit Vista desktop. Then I tried both the command line -t option (output attached) and the GUI option. In either case, a large section of the word got lost.. From the bottom, you can scroll up and look for reference #4 which exists but references 1 - 3 don't.. You will also find that a page or so of the original document is missing! It is not clear what about this document caused this behavior.
> I am trying to use tika to convert content from the web.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (TIKA-315) Tika appears to skip over an entire section of a Microsoft Word Document

2009-10-26T09:16:59Z

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (TIKA-315) Tika appears to skip over an entire section of a Microsoft Word Document

2009-10-26T09:14:59Z

Tika appears to skip over an entire section of a Microsoft Word Document
------------------------------------------------------------------------

Key: TIKA-315
URL: https://issues.apache.org/jira/browse/TIKA-315
Project: Tika
Issue Type: Bug
Affects Versions: 0.4
Environment: Microsoft Windows Vista 32 bit; Apache tika 0.4 release; used the -gui option.
Reporter: Sanjeev Rao

Using the -gui option, dragged and dropped this file http://mingo.info-science.uiowa.edu/padmini/Papers/Biolink07.doc which was saved on my 32 bit Vista desktop. Attached is the output. (assuming I can attach after I create this issue)

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (TIKA-314) Initial support for JPEG EXIF metadata extraction

2009-10-24T16:42:59Z

[ https://issues.apache.org/jira/browse/TIKA-314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769736#action_12769736 ]

Jukka Zitting commented on TIKA-314:
------------------------------------

Patch committed in revision 829467. Added missing license headers in revision 829469.

BTW, how about if we moved the JpegParser class to the o.a.tika.parser.image package?

> Initial support for JPEG EXIF metadata extraction
> -------------------------------------------------
>
> Key: TIKA-314
> URL: https://issues.apache.org/jira/browse/TIKA-314
> Project: Tika
> Issue Type: Improvement
> Components: parser
> Reporter: Maxim Valyanskiy
> Fix For: 0.5
>
> Attachments: initial-support-for-jpeg-exif-extraction.patch, testJPEG_EXIF.jpg
>
>
> This patch adds initial support for JPEG EXIF metadata extraction

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (TIKA-314) Initial support for JPEG EXIF metadata extraction

2009-10-24T16:24:59Z

[ https://issues.apache.org/jira/browse/TIKA-314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769731#action_12769731 ]

Jukka Zitting commented on TIKA-314:
------------------------------------

Looks good, thanks! The metadata-extractor library seems to be in the public domain, so it should be fine for us to use. The exact license info I find in the library is:

This is public domain software - that is, you can do whatever you want
with it, and include it software that is licensed under the GNU or the
BSD license, or whatever other licence you choose, including proprietary
closed source licenses. I do ask that you leave this header in tact.

BTW, a git format patch is fine, I'm actually using git myself too (committing via git-svn).

> Initial support for JPEG EXIF metadata extraction
> -------------------------------------------------
>
> Key: TIKA-314
> URL: https://issues.apache.org/jira/browse/TIKA-314
> Project: Tika
> Issue Type: Improvement
> Components: parser
> Reporter: Maxim Valyanskiy
> Fix For: 0.5
>
> Attachments: initial-support-for-jpeg-exif-extraction.patch, testJPEG_EXIF.jpg
>
>
> This patch adds initial support for JPEG EXIF metadata extraction

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Re: MarkUnsupportedException

2009-10-24T12:02:00Z

Hi,

On Sat, Oct 17, 2009 at 2:43 PM, mastcheshmi
<mehran.mastcheshmi@...> wrote:
> I use Tika
> for all document this exception occured.
> org/apache/poi/hpsf/MarkUnsupportedException

That seems like a bug. I guess POI expects the given document stream
to support the mark feature, so Tika should explicitly wrap the stream
into a java.io.BufferedInputStream if it does not already support
marks.

Can you please file a bug report [1] about this? Meanwhile, as a
workaround you can wrap the input stream yourself:

new OfficeParser().parse(new BufferedInputStream(in), handler, metadata);

[1] https://issues.apache.org/jira/browse/TIKA

BR,

Jukka Zitting

[jira] Updated: (TIKA-314) Initial support for JPEG EXIF metadata extraction

2009-10-20T06:25:59Z

[ https://issues.apache.org/jira/browse/TIKA-314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Maxim Valyanskiy updated TIKA-314:
----------------------------------

Attachment: (was: 0001-initial-support-for-jpeg-exif-extraction.patch)

> Initial support for JPEG EXIF metadata extraction
> -------------------------------------------------
>
> Key: TIKA-314
> URL: https://issues.apache.org/jira/browse/TIKA-314
> Project: Tika
> Issue Type: Improvement
> Components: parser
> Reporter: Maxim Valyanskiy
> Fix For: 0.5
>
> Attachments: initial-support-for-jpeg-exif-extraction.patch, testJPEG_EXIF.jpg
>
>
> This patch adds initial support for JPEG EXIF metadata extraction

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (TIKA-314) Initial support for JPEG EXIF metadata extraction

2009-10-20T06:25:59Z

[ https://issues.apache.org/jira/browse/TIKA-314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Maxim Valyanskiy updated TIKA-314:
----------------------------------

Attachment: initial-support-for-jpeg-exif-extraction.patch

oops, I attached patch in git format, this attachement is the same patch in unidiff format

> Initial support for JPEG EXIF metadata extraction
> -------------------------------------------------
>
> Key: TIKA-314
> URL: https://issues.apache.org/jira/browse/TIKA-314
> Project: Tika
> Issue Type: Improvement
> Components: parser
> Reporter: Maxim Valyanskiy
> Fix For: 0.5
>
> Attachments: initial-support-for-jpeg-exif-extraction.patch, testJPEG_EXIF.jpg
>
>
> This patch adds initial support for JPEG EXIF metadata extraction

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.