Camel Mail issue with unsupported charset

[dcheckoway]

I'm using Camel Mail to route email messages to a handler.  Works perfectly...but I've bumped into an issue whenever somebody sends an email with an unsupported charset.  For example:

     Content-type: text/plain; charset=ansi_x3.110-1983

...and since JDK6 has no idea wtf charset that is, I get the following exception:

Caused by: java.io.UnsupportedEncodingException: ansi_x3.110-1983
        at sun.nio.cs.StreamDecoder.forInputStreamReader(StreamDecoder.java:52)
        at java.io.InputStreamReader.<init>(InputStreamReader.java:83)
        at com.sun.mail.handlers.text_plain.getContent(text_plain.java:82)
        at javax.activation.DataSourceDataContentHandler.getContent(DataHandler.java:775)
        at javax.activation.DataHandler.getContent(DataHandler.java:522)
        at javax.mail.internet.MimeMessage.getContent(MimeMessage.java:1380)
        at org.apache.camel.component.mail.MailBinding.extractBodyFromMail(MailBinding.java:108)

This wouldn't be such a big deal if it wasn't for the fact that Camel is now totally hosed and can't "move past" that one message in the inbox.  Every time it subsequently polls the inbox, it throws that same exception and bails.  It can't access any other messages at that point.  I'm forced to go in and manually delete that one message so that Camel can continue processing the inbox.

So here are my questions...

1. Is there any way I can somehow intercept/filter the route and manually tweak the Content-Type header on the MimeMessage?  I would happily munge the charset to a known value if I could...

2. Is there any way Camel's behavior can be configured to "ignore messages on error" or something like that?

3. Is there any way I can add Charset support for ansi_x3.110-1983?  Seems like that would be the ideal solution.

For what it's worth, here's my route:

            from("imaps://imap.gmail.com?" +
                 "username=" + username +
                 "&password=" + password +
                 "&deleteProcessedMessages=false" +
                 "&processOnlyUnseenMessages=true" +
                 "&consumer.delay=10003")
                .process(handler);

Thanks in advance for your help!

[hzbarcea]

If Camel encounters an exception it will invoke on the error handler  
which by default in 1.x tries to reprocess.
If all that fails, the message remains unprocessed. What you need is  
to handle the exception yourself and process it any way you want,  
either delete it or move it to some other folder for review, etc.

I'd suggest looking at the try/catch: http://camel.apache.org/try-catch-finally.html

Cheers
Hadrian


On Jul 31, 2009, at 8:27 AM, dcheckoway wrote:

>
> I'm using Camel Mail to route email messages to a handler.  Works
> perfectly...but I've bumped into an issue whenever somebody sends an  
> email
> with an unsupported charset.  For example:
>
>     Content-type: text/plain; charset=ansi_x3.110-1983
>
> ...and since JDK6 has no idea wtf charset that is, I get the following
> exception:
>
> Caused by: java.io.UnsupportedEncodingException: ansi_x3.110-1983
>        at
> sun.nio.cs.StreamDecoder.forInputStreamReader(StreamDecoder.java:52)
>        at java.io.InputStreamReader.<init>(InputStreamReader.java:83)
>        at  
> com.sun.mail.handlers.text_plain.getContent(text_plain.java:82)
>        at
> javax
> .activation.DataSourceDataContentHandler.getContent(DataHandler.java:
> 775)
>        at javax.activation.DataHandler.getContent(DataHandler.java:
> 522)
>        at  
> javax.mail.internet.MimeMessage.getContent(MimeMessage.java:1380)
>        at
> org
> .apache
> .camel
> .component.mail.MailBinding.extractBodyFromMail(MailBinding.java:108)
>
> This wouldn't be such a big deal if it wasn't for the fact that  
> Camel is now
> totally hosed and can't "move past" that one message in the inbox.  
> Every
> time it subsequently polls the inbox, it throws that same exception  
> and
> bails.  It can't access any other messages at that point.  I'm  
> forced to go
> in and manually delete that one message so that Camel can continue
> processing the inbox.
>
> So here are my questions...
>
> 1. Is there any way I can somehow intercept/filter the route and  
> manually
> tweak the Content-Type header on the MimeMessage?  I would happily  
> munge the
> charset to a known value if I could...
>
> 2. Is there any way Camel's behavior can be configured to "ignore  
> messages
> on error" or something like that?
>
> 3. Is there any way I can add Charset support for ansi_x3.110-1983?  
> Seems
> like that would be the ideal solution.
>
> For what it's worth, here's my route:
>
>            from("imaps://imap.gmail.com?" +
>                 "username=" + username +
>                 "&password=" + password +
>                 "&deleteProcessedMessages=false" +
>                 "&processOnlyUnseenMessages=true" +
>                 "&consumer.delay=10003")
>                .process(handler);
>
> Thanks in advance for your help!
> --
> View this message in context: http://www.nabble.com/Camel-Mail-issue-with-unsupported-charset-tp24755585p24755585.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>

[Claus Ibsen-2]

On Fri, Jul 31, 2009 at 2:27 PM, dcheckoway<dcheckoway@...> wrote:

>
> I'm using Camel Mail to route email messages to a handler.  Works
> perfectly...but I've bumped into an issue whenever somebody sends an email
> with an unsupported charset.  For example:
>
>     Content-type: text/plain; charset=ansi_x3.110-1983
>
> ...and since JDK6 has no idea wtf charset that is, I get the following
> exception:
>
> Caused by: java.io.UnsupportedEncodingException: ansi_x3.110-1983
>        at
> sun.nio.cs.StreamDecoder.forInputStreamReader(StreamDecoder.java:52)
>        at java.io.InputStreamReader.<init>(InputStreamReader.java:83)
>        at com.sun.mail.handlers.text_plain.getContent(text_plain.java:82)
>        at
> javax.activation.DataSourceDataContentHandler.getContent(DataHandler.java:775)
>        at javax.activation.DataHandler.getContent(DataHandler.java:522)
>        at javax.mail.internet.MimeMessage.getContent(MimeMessage.java:1380)
>        at
> org.apache.camel.component.mail.MailBinding.extractBodyFromMail(MailBinding.java:108)
>
> This wouldn't be such a big deal if it wasn't for the fact that Camel is now
> totally hosed and can't "move past" that one message in the inbox.  Every
> time it subsequently polls the inbox, it throws that same exception and
> bails.  It can't access any other messages at that point.  I'm forced to go
> in and manually delete that one message so that Camel can continue
> processing the inbox.
>
> So here are my questions...
>
> 1. Is there any way I can somehow intercept/filter the route and manually
> tweak the Content-Type header on the MimeMessage?  I would happily munge the
> charset to a known value if I could...
>

There is a ContentTypeResolver in camel-mail that can help here
however it currently only supports when sending as the geronomi mail
provider is buggy and it allowed end users to work around issues with
it.

I have created a ticket to extend this to consuming as well, that
would allow you to detect this content type and remove the invalid
charset name, so Camel will just use the platform default instead.
https://issues.apache.org/activemq/browse/CAMEL-1869


> 2. Is there any way Camel's behavior can be configured to "ignore messages
> on error" or something like that?
>

No as this is an issue within the transport itself and no message have
been handed over to Camel routing engine.
The problem lies in that the mail server has invalid/incompatible data
with Java 6.

You can extend the mail component yourself and work around it by
overriding the code in MailBinding and work around the invalid
charset.


> 3. Is there any way I can add Charset support for ansi_x3.110-1983?  Seems
> like that would be the ideal solution.
>

I have newer heard of such a charset. Can you try google this to see
if its actually a valid charset name. I do think that its a mistake
and the mail server do not care anyway.

Which mail server are you using?

And also you should always state which version of Camel you are using!

> For what it's worth, here's my route:
>
>            from("imaps://imap.gmail.com?" +
>                 "username=" + username +
>                 "&password=" + password +
>                 "&deleteProcessedMessages=false" +
>                 "&processOnlyUnseenMessages=true" +
>                 "&consumer.delay=10003")
>                .process(handler);
>
> Thanks in advance for your help!
> --
> View this message in context: http://www.nabble.com/Camel-Mail-issue-with-unsupported-charset-tp24755585p24755585.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>

--
Claus Ibsen
Apache Camel Committer

Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

[dcheckoway]

Claus,

Thanks for the reply.  It is a valid charset name, but java6 has no knowledge of it (yet).  For example, on my linux box, it lives in /usr/share/i18n/charmaps/ANSI_X3.110-1983.gz right alongside all the other typical charsets.

The mail server is whatever google uses...imap.gmail.com is the host I'm having Camel poll.

I'm using Camel 2.0-M1.

--Dan

Claus Ibsen-2 wrote:

I have newer heard of such a charset. Can you try google this to see
if its actually a valid charset name. I do think that its a mistake
and the mail server do not care anyway.

Which mail server are you using?

And also you should always state which version of Camel you are using!

[Claus Ibsen-2]

On Mon, Aug 3, 2009 at 2:53 PM, dcheckoway<dcheckoway@...> wrote:
>
> Claus,
>
> Thanks for the reply.  It is a valid charset name, but java6 has no
> knowledge of it (yet).  For example, on my linux box, it lives in
> /usr/share/i18n/charmaps/ANSI_X3.110-1983.gz right alongside all the other
> typical charsets.
>

Thanks for the findings.

> The mail server is whatever google uses...imap.gmail.com is the host I'm
> having Camel poll.
>
> I'm using Camel 2.0-M1.
>

I will work on the ticket allowing you to remedy this by changing it
to another charset of your choice, such as utf-8 or iso-8859-1 or what
you like.

And I wonder if we should add a simpler boolean flag where you can set
whether camel-mail should ignore unknown charset and default to use
the platforms charset then?

> --Dan
>
>
> Claus Ibsen-2 wrote:
>>
>> I have newer heard of such a charset. Can you try google this to see
>> if its actually a valid charset name. I do think that its a mistake
>> and the mail server do not care anyway.
>>
>> Which mail server are you using?
>>
>> And also you should always state which version of Camel you are using!
>>
>
> --
> View this message in context: http://www.nabble.com/Camel-Mail-issue-with-unsupported-charset-tp24755585p24790290.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>

--
Claus Ibsen
Apache Camel Committer

Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

[Claus Ibsen-2]

Hi

What version of Camel are you using?

We have improved the camel-mail component quite a bit lately.
Can you try with 1.6.1 or 2.0m3?


On Fri, Jul 31, 2009 at 2:27 PM, dcheckoway<dcheckoway@...> wrote:

>
> I'm using Camel Mail to route email messages to a handler.  Works
> perfectly...but I've bumped into an issue whenever somebody sends an email
> with an unsupported charset.  For example:
>
>     Content-type: text/plain; charset=ansi_x3.110-1983
>
> ...and since JDK6 has no idea wtf charset that is, I get the following
> exception:
>
> Caused by: java.io.UnsupportedEncodingException: ansi_x3.110-1983
>        at
> sun.nio.cs.StreamDecoder.forInputStreamReader(StreamDecoder.java:52)
>        at java.io.InputStreamReader.<init>(InputStreamReader.java:83)
>        at com.sun.mail.handlers.text_plain.getContent(text_plain.java:82)
>        at
> javax.activation.DataSourceDataContentHandler.getContent(DataHandler.java:775)
>        at javax.activation.DataHandler.getContent(DataHandler.java:522)
>        at javax.mail.internet.MimeMessage.getContent(MimeMessage.java:1380)
>        at
> org.apache.camel.component.mail.MailBinding.extractBodyFromMail(MailBinding.java:108)
>
> This wouldn't be such a big deal if it wasn't for the fact that Camel is now
> totally hosed and can't "move past" that one message in the inbox.  Every
> time it subsequently polls the inbox, it throws that same exception and
> bails.  It can't access any other messages at that point.  I'm forced to go
> in and manually delete that one message so that Camel can continue
> processing the inbox.
>
> So here are my questions...
>
> 1. Is there any way I can somehow intercept/filter the route and manually
> tweak the Content-Type header on the MimeMessage?  I would happily munge the
> charset to a known value if I could...
>
> 2. Is there any way Camel's behavior can be configured to "ignore messages
> on error" or something like that?
>
> 3. Is there any way I can add Charset support for ansi_x3.110-1983?  Seems
> like that would be the ideal solution.
>
> For what it's worth, here's my route:
>
>            from("imaps://imap.gmail.com?" +
>                 "username=" + username +
>                 "&password=" + password +
>                 "&deleteProcessedMessages=false" +
>                 "&processOnlyUnseenMessages=true" +
>                 "&consumer.delay=10003")
>                .process(handler);
>
> Thanks in advance for your help!
> --
> View this message in context: http://www.nabble.com/Camel-Mail-issue-with-unsupported-charset-tp24755585p24755585.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>

--
Claus Ibsen
Apache Camel Committer

Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

[dcheckoway]

Claus,

I wanted to let you know that this unsupported charset issue is still a problem with Camel 2.0.0 (camel-mail-2.0.0.jar), even though I'm now using "ignoreUnsupportedCharset=true".  Here's the stack trace:

Nov 3, 2009 5:02:50 AM org.apache.camel.processor.Logger log
SEVERE: Failed to extract body due to: ansi_x3.110-1983. Exchange: Exchange[MailMessage: messageNumber=[478], from=["[e-mail]" <[e-mail]>], to=[Weathermeister TFR Notifier <[e-mail]>], subject=[New Email Address], sentDate=[Nov 2, 2009 5:12:54 PM], receivedDate=[Nov 2, 2009 5:12:55 PM]]. Message: com.sun.mail.imap.IMAPMessage@a2bd15
org.apache.camel.RuntimeCamelException: Failed to extract body due to: ansi_x3.110-1983. Exchange: Exchange[MailMessage: messageNumber=[478], from=["[e-mail]" <[e-mail]>], to=[Weathermeister TFR Notifier <[e-mail]>], subject=[New Email Address], sentDate=[Nov 2, 2009 5:12:54 PM], receivedDate=[Nov 2, 2009 5:12:55 PM]]. Message: com.sun.mail.imap.IMAPMessage@a2bd15
        at org.apache.camel.component.mail.MailBinding.extractBodyFromMail(MailBinding.java:215)
        at org.apache.camel.component.mail.MailMessage.createBody(MailMessage.java:86)
        at org.apache.camel.impl.MessageSupport.getBody(MessageSupport.java:43)
        at org.apache.camel.impl.MessageSupport.copyFrom(MessageSupport.java:133)
        at org.apache.camel.impl.DefaultMessage.copyFrom(DefaultMessage.java:51)
        at org.apache.camel.component.mail.MailMessage.copyFrom(MailMessage.java:117)
        at org.apache.camel.impl.MessageSupport.copy(MessageSupport.java:127)
        at org.apache.camel.component.mail.MailMessage.copy(MailMessage.java:61)
        at org.apache.camel.component.mail.MailMessage.copy(MailMessage.java:40)
        at org.apache.camel.impl.DefaultUnitOfWork.<init>(DefaultUnitOfWork.java:61)
        at org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:44)
        at org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
        at org.apache.camel.component.mail.MailConsumer.processExchange(MailConsumer.java:195)
        at org.apache.camel.component.mail.MailConsumer.processBatch(MailConsumer.java:153)
        at org.apache.camel.component.mail.MailConsumer.poll(MailConsumer.java:111)
        at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:99)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.UnsupportedEncodingException: ansi_x3.110-1983
        at sun.nio.cs.StreamDecoder.forInputStreamReader(StreamDecoder.java:52)
        at java.io.InputStreamReader.<init>(InputStreamReader.java:83)
        at com.sun.mail.handlers.text_plain.getContent(text_plain.java:82)
        at javax.activation.DataSourceDataContentHandler.getContent(DataHandler.java:775)
        at javax.activation.DataHandler.getContent(DataHandler.java:522)
        at javax.mail.internet.MimeMessage.getContent(MimeMessage.java:1380)
        at org.apache.camel.component.mail.MailBinding.extractBodyFromMail(MailBinding.java:213)
        ... 24 more

I'm attaching the source email message, whose Content-Type header is:

Content-type: text/plain; charset=ansi_x3.110-1983

Am I doing something wrong, or does ignoreUnsupportedCharset not work in this case?

Thanks,
Dan
message-ansi_x3.110-1983

Claus Ibsen-2 wrote:

On Mon, Aug 3, 2009 at 2:53 PM, dcheckoway<[e-mail]> wrote:
>
> Claus,
>
> Thanks for the reply.  It is a valid charset name, but java6 has no
> knowledge of it (yet).  For example, on my linux box, it lives in
> /usr/share/i18n/charmaps/ANSI_X3.110-1983.gz right alongside all the other
> typical charsets.
>

Thanks for the findings.

> The mail server is whatever google uses...imap.gmail.com is the host I'm
> having Camel poll.
>
> I'm using Camel 2.0-M1.
>

I will work on the ticket allowing you to remedy this by changing it
to another charset of your choice, such as utf-8 or iso-8859-1 or what
you like.

And I wonder if we should add a simpler boolean flag where you can set
whether camel-mail should ignore unknown charset and default to use
the platforms charset then?




> --Dan
>
>
> Claus Ibsen-2 wrote:
>>
>> I have newer heard of such a charset. Can you try google this to see
>> if its actually a valid charset name. I do think that its a mistake
>> and the mail server do not care anyway.
>>
>> Which mail server are you using?
>>
>> And also you should always state which version of Camel you are using!
>>
>
> --
> View this message in context: http://www.nabble.com/Camel-Mail-issue-with-unsupported-charset-tp24755585p24790290.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>



--
Claus Ibsen
Apache Camel Committer

Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

[Claus Ibsen-2]

Hi

Can you try with 2.1-SNAPSHOT as we have messed with the camel-mail
component since it was released in 2.0.0.

On Tue, Nov 3, 2009 at 2:10 PM, dcheckoway <dcheckoway@...> wrote:

>
> Claus,
>
> I wanted to let you know that this unsupported charset issue is still a
> problem with Camel 2.0.0 (camel-mail-2.0.0.jar), even though I'm now using
> "ignoreUnsupportedCharset=true".  Here's the stack trace:
>
> Nov 3, 2009 5:02:50 AM org.apache.camel.processor.Logger log
> SEVERE: Failed to extract body due to: ansi_x3.110-1983. Exchange:
> Exchange[MailMessage: messageNumber=[478],
> from=["do-not-reply@..." <art@...>],
> to=[Weathermeister TFR Notifier <tfr@...>], subject=[New
> Email Address], sentDate=[Nov 2, 2009 5:12:54 PM], receivedDate=[Nov 2, 2009
> 5:12:55 PM]]. Message: com.sun.mail.imap.IMAPMessage@a2bd15
> org.apache.camel.RuntimeCamelException: Failed to extract body due to:
> ansi_x3.110-1983. Exchange: Exchange[MailMessage: messageNumber=[478],
> from=["do-not-reply@..." <art@...>],
> to=[Weathermeister TFR Notifier <tfr@...>], subject=[New
> Email Address], sentDate=[Nov 2, 2009 5:12:54 PM], receivedDate=[Nov 2, 2009
> 5:12:55 PM]]. Message: com.sun.mail.imap.IMAPMessage@a2bd15
>        at
> org.apache.camel.component.mail.MailBinding.extractBodyFromMail(MailBinding.java:215)
>        at
> org.apache.camel.component.mail.MailMessage.createBody(MailMessage.java:86)
>        at
> org.apache.camel.impl.MessageSupport.getBody(MessageSupport.java:43)
>        at
> org.apache.camel.impl.MessageSupport.copyFrom(MessageSupport.java:133)
>        at
> org.apache.camel.impl.DefaultMessage.copyFrom(DefaultMessage.java:51)
>        at
> org.apache.camel.component.mail.MailMessage.copyFrom(MailMessage.java:117)
>        at
> org.apache.camel.impl.MessageSupport.copy(MessageSupport.java:127)
>        at
> org.apache.camel.component.mail.MailMessage.copy(MailMessage.java:61)
>        at
> org.apache.camel.component.mail.MailMessage.copy(MailMessage.java:40)
>        at
> org.apache.camel.impl.DefaultUnitOfWork.<init>(DefaultUnitOfWork.java:61)
>        at
> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:44)
>        at
> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
>        at
> org.apache.camel.component.mail.MailConsumer.processExchange(MailConsumer.java:195)
>        at
> org.apache.camel.component.mail.MailConsumer.processBatch(MailConsumer.java:153)
>        at
> org.apache.camel.component.mail.MailConsumer.poll(MailConsumer.java:111)
>        at
> org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:99)
>        at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>        at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>        at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>        at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
>        at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.UnsupportedEncodingException: ansi_x3.110-1983
>        at
> sun.nio.cs.StreamDecoder.forInputStreamReader(StreamDecoder.java:52)
>        at java.io.InputStreamReader.<init>(InputStreamReader.java:83)
>        at com.sun.mail.handlers.text_plain.getContent(text_plain.java:82)
>        at
> javax.activation.DataSourceDataContentHandler.getContent(DataHandler.java:775)
>        at javax.activation.DataHandler.getContent(DataHandler.java:522)
>        at javax.mail.internet.MimeMessage.getContent(MimeMessage.java:1380)
>        at
> org.apache.camel.component.mail.MailBinding.extractBodyFromMail(MailBinding.java:213)
>        ... 24 more
>
> I'm attaching the source email message, whose Content-Type header is:
>
> Content-type: text/plain; charset=ansi_x3.110-1983
>
> Am I doing something wrong, or does ignoreUnsupportedCharset not work in
> this case?
>
> Thanks,
> Dan
> http://old.nabble.com/file/p26160108/message-ansi_x3.110-1983
> message-ansi_x3.110-1983
>
>
> Claus Ibsen-2 wrote:
>>
>> On Mon, Aug 3, 2009 at 2:53 PM, dcheckoway<dcheckoway@...> wrote:
>>>
>>> Claus,
>>>
>>> Thanks for the reply.  It is a valid charset name, but java6 has no
>>> knowledge of it (yet).  For example, on my linux box, it lives in
>>> /usr/share/i18n/charmaps/ANSI_X3.110-1983.gz right alongside all the
>>> other
>>> typical charsets.
>>>
>>
>> Thanks for the findings.
>>
>>> The mail server is whatever google uses...imap.gmail.com is the host I'm
>>> having Camel poll.
>>>
>>> I'm using Camel 2.0-M1.
>>>
>>
>> I will work on the ticket allowing you to remedy this by changing it
>> to another charset of your choice, such as utf-8 or iso-8859-1 or what
>> you like.
>>
>> And I wonder if we should add a simpler boolean flag where you can set
>> whether camel-mail should ignore unknown charset and default to use
>> the platforms charset then?
>>
>>
>>
>>
>>> --Dan
>>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> I have newer heard of such a charset. Can you try google this to see
>>>> if its actually a valid charset name. I do think that its a mistake
>>>> and the mail server do not care anyway.
>>>>
>>>> Which mail server are you using?
>>>>
>>>> And also you should always state which version of Camel you are using!
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Camel-Mail-issue-with-unsupported-charset-tp24755585p24790290.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/Camel-Mail-issue-with-unsupported-charset-tp24755585p26160108.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>

--
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

[dcheckoway]

I'll definitely try 2.1-SNAPSHOT.  And in the meantime, how would I go about "catching" that exception so that my app can at least be aware of this error condition?

As of right now, it literally clogs the whole process, since Camel can't get past that offending message and process anything else in the inbox.  It just repeatedly fails & bails, every time it polls the inbox.  I'd at least like to catch the exception and have my app notify me of the condition (which I currently clear by manually deleting the offending message from the inbox).

Should I use try/catch in the route builder?  I'm painfully amateur at error handling with camel...sorry...  :-)

Claus Ibsen-2 wrote:

Hi

Can you try with 2.1-SNAPSHOT as we have messed with the camel-mail
component since it was released in 2.0.0.

On Tue, Nov 3, 2009 at 2:10 PM, dcheckoway <[e-mail]> wrote:
>
> Claus,
>
> I wanted to let you know that this unsupported charset issue is still a
> problem with Camel 2.0.0 (camel-mail-2.0.0.jar), even though I'm now using
> "ignoreUnsupportedCharset=true".  Here's the stack trace:
>
> Nov 3, 2009 5:02:50 AM org.apache.camel.processor.Logger log
> SEVERE: Failed to extract body due to: ansi_x3.110-1983. Exchange:
> Exchange[MailMessage: messageNumber=[478],
> from=["[e-mail]" <[e-mail]>],
> to=[Weathermeister TFR Notifier <[e-mail]>], subject=[New
> Email Address], sentDate=[Nov 2, 2009 5:12:54 PM], receivedDate=[Nov 2, 2009
> 5:12:55 PM]]. Message: com.sun.mail.imap.IMAPMessage@a2bd15
> org.apache.camel.RuntimeCamelException: Failed to extract body due to:
> ansi_x3.110-1983. Exchange: Exchange[MailMessage: messageNumber=[478],
> from=["[e-mail]" <[e-mail]>],
> to=[Weathermeister TFR Notifier <[e-mail]>], subject=[New
> Email Address], sentDate=[Nov 2, 2009 5:12:54 PM], receivedDate=[Nov 2, 2009
> 5:12:55 PM]]. Message: com.sun.mail.imap.IMAPMessage@a2bd15
>        at
> org.apache.camel.component.mail.MailBinding.extractBodyFromMail(MailBinding.java:215)
>        at
> org.apache.camel.component.mail.MailMessage.createBody(MailMessage.java:86)
>        at
> org.apache.camel.impl.MessageSupport.getBody(MessageSupport.java:43)
>        at
> org.apache.camel.impl.MessageSupport.copyFrom(MessageSupport.java:133)
>        at
> org.apache.camel.impl.DefaultMessage.copyFrom(DefaultMessage.java:51)
>        at
> org.apache.camel.component.mail.MailMessage.copyFrom(MailMessage.java:117)
>        at
> org.apache.camel.impl.MessageSupport.copy(MessageSupport.java:127)
>        at
> org.apache.camel.component.mail.MailMessage.copy(MailMessage.java:61)
>        at
> org.apache.camel.component.mail.MailMessage.copy(MailMessage.java:40)
>        at
> org.apache.camel.impl.DefaultUnitOfWork.<init>(DefaultUnitOfWork.java:61)
>        at
> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:44)
>        at
> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
>        at
> org.apache.camel.component.mail.MailConsumer.processExchange(MailConsumer.java:195)
>        at
> org.apache.camel.component.mail.MailConsumer.processBatch(MailConsumer.java:153)
>        at
> org.apache.camel.component.mail.MailConsumer.poll(MailConsumer.java:111)
>        at
> org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:99)
>        at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>        at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>        at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>        at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
>        at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.UnsupportedEncodingException: ansi_x3.110-1983
>        at
> sun.nio.cs.StreamDecoder.forInputStreamReader(StreamDecoder.java:52)
>        at java.io.InputStreamReader.<init>(InputStreamReader.java:83)
>        at com.sun.mail.handlers.text_plain.getContent(text_plain.java:82)
>        at
> javax.activation.DataSourceDataContentHandler.getContent(DataHandler.java:775)
>        at javax.activation.DataHandler.getContent(DataHandler.java:522)
>        at javax.mail.internet.MimeMessage.getContent(MimeMessage.java:1380)
>        at
> org.apache.camel.component.mail.MailBinding.extractBodyFromMail(MailBinding.java:213)
>        ... 24 more
>
> I'm attaching the source email message, whose Content-Type header is:
>
> Content-type: text/plain; charset=ansi_x3.110-1983
>
> Am I doing something wrong, or does ignoreUnsupportedCharset not work in
> this case?
>
> Thanks,
> Dan
> http://old.nabble.com/file/p26160108/message-ansi_x3.110-1983
> message-ansi_x3.110-1983
>
>
> Claus Ibsen-2 wrote:
>>
>> On Mon, Aug 3, 2009 at 2:53 PM, dcheckoway<[e-mail]> wrote:
>>>
>>> Claus,
>>>
>>> Thanks for the reply.  It is a valid charset name, but java6 has no
>>> knowledge of it (yet).  For example, on my linux box, it lives in
>>> /usr/share/i18n/charmaps/ANSI_X3.110-1983.gz right alongside all the
>>> other
>>> typical charsets.
>>>
>>
>> Thanks for the findings.
>>
>>> The mail server is whatever google uses...imap.gmail.com is the host I'm
>>> having Camel poll.
>>>
>>> I'm using Camel 2.0-M1.
>>>
>>
>> I will work on the ticket allowing you to remedy this by changing it
>> to another charset of your choice, such as utf-8 or iso-8859-1 or what
>> you like.
>>
>> And I wonder if we should add a simpler boolean flag where you can set
>> whether camel-mail should ignore unknown charset and default to use
>> the platforms charset then?
>>
>>
>>
>>
>>> --Dan
>>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> I have newer heard of such a charset. Can you try google this to see
>>>> if its actually a valid charset name. I do think that its a mistake
>>>> and the mail server do not care anyway.
>>>>
>>>> Which mail server are you using?
>>>>
>>>> And also you should always state which version of Camel you are using!
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Camel-Mail-issue-with-unsupported-charset-tp24755585p24790290.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/Camel-Mail-issue-with-unsupported-charset-tp24755585p26160108.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>



--
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

[Claus Ibsen-2]

On Wed, Nov 4, 2009 at 4:57 PM, dcheckoway <dcheckoway@...> wrote:

>
> I'll definitely try 2.1-SNAPSHOT.  And in the meantime, how would I go about
> "catching" that exception so that my app can at least be aware of this error
> condition?
>
> As of right now, it literally clogs the whole process, since Camel can't get
> past that offending message and process anything else in the inbox.  It just
> repeatedly fails & bails, every time it polls the inbox.  I'd at least like
> to catch the exception and have my app notify me of the condition (which I
> currently clear by manually deleting the offending message from the inbox).
>
> Should I use try/catch in the route builder?  I'm painfully amateur at error
> handling with camel...sorry...  :-)
>

Unfortunately I guess the camel-mail component will pre extract the
data before its being routed, and thus before the error handling kicks
in.

And the stacktrace indicates the root problem is deep inside the SUN
mail jar which makes it hard to fix the unsupported charset.

>
> Claus Ibsen-2 wrote:
>>
>> Hi
>>
>> Can you try with 2.1-SNAPSHOT as we have messed with the camel-mail
>> component since it was released in 2.0.0.
>>
>> On Tue, Nov 3, 2009 at 2:10 PM, dcheckoway <dcheckoway@...> wrote:
>>>
>>> Claus,
>>>
>>> I wanted to let you know that this unsupported charset issue is still a
>>> problem with Camel 2.0.0 (camel-mail-2.0.0.jar), even though I'm now
>>> using
>>> "ignoreUnsupportedCharset=true".  Here's the stack trace:
>>>
>>> Nov 3, 2009 5:02:50 AM org.apache.camel.processor.Logger log
>>> SEVERE: Failed to extract body due to: ansi_x3.110-1983. Exchange:
>>> Exchange[MailMessage: messageNumber=[478],
>>> from=["do-not-reply@..." <art@...>],
>>> to=[Weathermeister TFR Notifier <tfr@...>], subject=[New
>>> Email Address], sentDate=[Nov 2, 2009 5:12:54 PM], receivedDate=[Nov 2,
>>> 2009
>>> 5:12:55 PM]]. Message: com.sun.mail.imap.IMAPMessage@a2bd15
>>> org.apache.camel.RuntimeCamelException: Failed to extract body due to:
>>> ansi_x3.110-1983. Exchange: Exchange[MailMessage: messageNumber=[478],
>>> from=["do-not-reply@..." <art@...>],
>>> to=[Weathermeister TFR Notifier <tfr@...>], subject=[New
>>> Email Address], sentDate=[Nov 2, 2009 5:12:54 PM], receivedDate=[Nov 2,
>>> 2009
>>> 5:12:55 PM]]. Message: com.sun.mail.imap.IMAPMessage@a2bd15
>>>        at
>>> org.apache.camel.component.mail.MailBinding.extractBodyFromMail(MailBinding.java:215)
>>>        at
>>> org.apache.camel.component.mail.MailMessage.createBody(MailMessage.java:86)
>>>        at
>>> org.apache.camel.impl.MessageSupport.getBody(MessageSupport.java:43)
>>>        at
>>> org.apache.camel.impl.MessageSupport.copyFrom(MessageSupport.java:133)
>>>        at
>>> org.apache.camel.impl.DefaultMessage.copyFrom(DefaultMessage.java:51)
>>>        at
>>> org.apache.camel.component.mail.MailMessage.copyFrom(MailMessage.java:117)
>>>        at
>>> org.apache.camel.impl.MessageSupport.copy(MessageSupport.java:127)
>>>        at
>>> org.apache.camel.component.mail.MailMessage.copy(MailMessage.java:61)
>>>        at
>>> org.apache.camel.component.mail.MailMessage.copy(MailMessage.java:40)
>>>        at
>>> org.apache.camel.impl.DefaultUnitOfWork.<init>(DefaultUnitOfWork.java:61)
>>>        at
>>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:44)
>>>        at
>>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
>>>        at
>>> org.apache.camel.component.mail.MailConsumer.processExchange(MailConsumer.java:195)
>>>        at
>>> org.apache.camel.component.mail.MailConsumer.processBatch(MailConsumer.java:153)
>>>        at
>>> org.apache.camel.component.mail.MailConsumer.poll(MailConsumer.java:111)
>>>        at
>>> org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:99)
>>>        at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>>        at
>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>>        at
>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>>        at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>>>        at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
>>>        at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
>>>        at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>        at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>        at java.lang.Thread.run(Thread.java:619)
>>> Caused by: java.io.UnsupportedEncodingException: ansi_x3.110-1983
>>>        at
>>> sun.nio.cs.StreamDecoder.forInputStreamReader(StreamDecoder.java:52)
>>>        at java.io.InputStreamReader.<init>(InputStreamReader.java:83)
>>>        at com.sun.mail.handlers.text_plain.getContent(text_plain.java:82)
>>>        at
>>> javax.activation.DataSourceDataContentHandler.getContent(DataHandler.java:775)
>>>        at javax.activation.DataHandler.getContent(DataHandler.java:522)
>>>        at
>>> javax.mail.internet.MimeMessage.getContent(MimeMessage.java:1380)
>>>        at
>>> org.apache.camel.component.mail.MailBinding.extractBodyFromMail(MailBinding.java:213)
>>>        ... 24 more
>>>
>>> I'm attaching the source email message, whose Content-Type header is:
>>>
>>> Content-type: text/plain; charset=ansi_x3.110-1983
>>>
>>> Am I doing something wrong, or does ignoreUnsupportedCharset not work in
>>> this case?
>>>
>>> Thanks,
>>> Dan
>>> http://old.nabble.com/file/p26160108/message-ansi_x3.110-1983
>>> message-ansi_x3.110-1983
>>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> On Mon, Aug 3, 2009 at 2:53 PM, dcheckoway<dcheckoway@...> wrote:
>>>>>
>>>>> Claus,
>>>>>
>>>>> Thanks for the reply.  It is a valid charset name, but java6 has no
>>>>> knowledge of it (yet).  For example, on my linux box, it lives in
>>>>> /usr/share/i18n/charmaps/ANSI_X3.110-1983.gz right alongside all the
>>>>> other
>>>>> typical charsets.
>>>>>
>>>>
>>>> Thanks for the findings.
>>>>
>>>>> The mail server is whatever google uses...imap.gmail.com is the host
>>>>> I'm
>>>>> having Camel poll.
>>>>>
>>>>> I'm using Camel 2.0-M1.
>>>>>
>>>>
>>>> I will work on the ticket allowing you to remedy this by changing it
>>>> to another charset of your choice, such as utf-8 or iso-8859-1 or what
>>>> you like.
>>>>
>>>> And I wonder if we should add a simpler boolean flag where you can set
>>>> whether camel-mail should ignore unknown charset and default to use
>>>> the platforms charset then?
>>>>
>>>>
>>>>
>>>>
>>>>> --Dan
>>>>>
>>>>>
>>>>> Claus Ibsen-2 wrote:
>>>>>>
>>>>>> I have newer heard of such a charset. Can you try google this to see
>>>>>> if its actually a valid charset name. I do think that its a mistake
>>>>>> and the mail server do not care anyway.
>>>>>>
>>>>>> Which mail server are you using?
>>>>>>
>>>>>> And also you should always state which version of Camel you are using!
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/Camel-Mail-issue-with-unsupported-charset-tp24755585p24790290.html
>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> Apache Camel Committer
>>>>
>>>> Open Source Integration: http://fusesource.com
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Twitter: http://twitter.com/davsclaus
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/Camel-Mail-issue-with-unsupported-charset-tp24755585p26160108.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/Camel-Mail-issue-with-unsupported-charset-tp24755585p26198992.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>

--
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

[dcheckoway]

Is there any way to "intercept" the raw mail message content before it gets passed to the java mail API?  I would be happy to manually munge the Content-Type header if I detect one of the known-to-be-funky charsets in there.

Claus Ibsen-2 wrote:

On Wed, Nov 4, 2009 at 4:57 PM, dcheckoway <[e-mail]> wrote:
Unfortunately I guess the camel-mail component will pre extract the
data before its being routed, and thus before the error handling kicks
in.

And the stacktrace indicates the root problem is deep inside the SUN
mail jar which makes it hard to fix the unsupported charset.

[Claus Ibsen-2]

On Wed, Nov 4, 2009 at 7:24 PM, dcheckoway <dcheckoway@...> wrote:
>
> Is there any way to "intercept" the raw mail message content before it gets
> passed to the java mail API?  I would be happy to manually munge the
> Content-Type header if I detect one of the known-to-be-funky charsets in
> there.
>

You can use a custom MailBinding class, e.g. extending the on in camel-maill.

But from the stack trace it appears as if when using getContent() on
he Mail API you get that exception.
I wonder if you can munge it to remove the charset header etc.

A remedy we can build in is to detect in the stacktrace hierachy that
is the unsupported charset and then have a kind of invalid message.
You can probably not read its content but Camel should be able to move
on. That could be a plan b.

>
>
> Claus Ibsen-2 wrote:
>>
>> On Wed, Nov 4, 2009 at 4:57 PM, dcheckoway <dcheckoway@...> wrote:
>> Unfortunately I guess the camel-mail component will pre extract the
>> data before its being routed, and thus before the error handling kicks
>> in.
>>
>> And the stacktrace indicates the root problem is deep inside the SUN
>> mail jar which makes it hard to fix the unsupported charset.
>>
>>
>
> --
> View this message in context: http://old.nabble.com/Camel-Mail-issue-with-unsupported-charset-tp24755585p26201924.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>

--
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

[dcheckoway]

Claus, I'm not a contributor on the Camel project, but here's what I would advocate trying in MailBinding.java, expanding the meaning of "ignoreUnsupportedCharset" a bit:

    public Object extractBodyFromMail(Exchange exchange, Message message) {
        return extractBodyFromMail(exchange, message, false);
    }

    private Object extractBodyFromMail(Exchange exchange, Message message, boolean isSecondAttempt) {
        try {
            return message.getContent();
        } catch (Exception e) {
            if (!isSecondAttempt &&
                (e instanceof java.io.UnsupportedEncodingException ||
                 (e.getCause() != null &&
                  e.getCause() instanceof java.io.UnsupportedEncodingException)) &&
                configuration.isIgnoreUnsupportedCharset()) {
                // The charset is unsupported, but we've been configured to
                // ingore that condition.  Let's at least attempt to rewrite
                // the Content-Type header and omit the unsupported charset.
                // This may wreak havoc down the line, but it's better than
                // Camel just bailing on this exchange, and ending up retrying
                // this same message over and over, and failing every time...
                String contentType = message.getContentType();
                if (contentType != null) {
                    // Wipe out the charset=... from the Content-Type header
                    contentType = contentType.replaceAll("\\s*charset=[^;\\s]+", "");
                    part.setHeader("Content-Type", contentType);
                    // Try again...pass isSecondAttempt=true this time
                    return extractBodyFromMail(exchange, message, true);
                }
            }
            else {
                throw new RuntimeCamelException("Failed to extract body due to: " + e.getMessage()
                                                + ". Exchange: " + exchange + ". Message: " + message, e);
            }
        }
    }

What do you think?

Claus Ibsen-2 wrote:

On Wed, Nov 4, 2009 at 7:24 PM, dcheckoway <[e-mail]> wrote:
>
> Is there any way to "intercept" the raw mail message content before it gets
> passed to the java mail API?  I would be happy to manually munge the
> Content-Type header if I detect one of the known-to-be-funky charsets in
> there.
>

You can use a custom MailBinding class, e.g. extending the on in camel-maill.

[Claus Ibsen-2]

On Thu, Nov 5, 2009 at 12:50 AM, dcheckoway <dcheckoway@...> wrote:

>
> Claus, I'm not a contributor on the Camel project, but here's what I would
> advocate trying in MailBinding.java, expanding the meaning of
> "ignoreUnsupportedCharset" a bit:
>
>    public Object extractBodyFromMail(Exchange exchange, Message message) {
>        return extractBodyFromMail(exchange, message, false);
>    }
>
>    private Object extractBodyFromMail(Exchange exchange, Message message,
> boolean isSecondAttempt) {
>        try {
>            return message.getContent();
>        } catch (Exception e) {
>            if (!isSecondAttempt &&
>                (e instanceof java.io.UnsupportedEncodingException ||
>                 (e.getCause() != null &&
>                  e.getCause() instanceof
> java.io.UnsupportedEncodingException)) &&
>                configuration.isIgnoreUnsupportedCharset()) {
>                // The charset is unsupported, but we've been configured to
>                // ingore that condition.  Let's at least attempt to rewrite
>                // the Content-Type header and omit the unsupported charset.
>                // This may wreak havoc down the line, but it's better than
>                // Camel just bailing on this exchange, and ending up
> retrying
>                // this same message over and over, and failing every
> time...
>                String contentType = message.getContentType();
>                if (contentType != null) {
>                    // Wipe out the charset=... from the Content-Type header
>                    contentType =
> contentType.replaceAll("\\s*charset=[^;\\s]+", "");
>                    part.setHeader("Content-Type", contentType);
>                    // Try again...pass isSecondAttempt=true this time
>                    return extractBodyFromMail(exchange, message, true);
>                }
>            }
>            else {
>                throw new RuntimeCamelException("Failed to extract body due
> to: " + e.getMessage()
>                                                + ". Exchange: " + exchange
> + ". Message: " + message, e);
>            }
>        }
>    }
>
> What do you think?
>

Yeah we could add such a work around.
Does it fix you issue when you manually remove that charset from the
header and can you read it on the 2nd attempt then?

And btw you are a Camel contributor :) And kind of activity is
measured for your metrics.
See some of the Apache blogs about how that works :)

Please create a JIRA ticket and submit your patch. And if possible an
unit test that demonstrates this if that is possible.

And Camel have some code to extract the exception from the hierarchy a
bit nicer and cleaner.

UnsupportedEncodingException uce =
ObjectHelper.getException(UnsupportedEncodingException.class, e);
if (uce != null) {
   // okay there was such kind of exception
}

>
>
> Claus Ibsen-2 wrote:
>>
>> On Wed, Nov 4, 2009 at 7:24 PM, dcheckoway <dcheckoway@...> wrote:
>>>
>>> Is there any way to "intercept" the raw mail message content before it
>>> gets
>>> passed to the java mail API?  I would be happy to manually munge the
>>> Content-Type header if I detect one of the known-to-be-funky charsets in
>>> there.
>>>
>>
>> You can use a custom MailBinding class, e.g. extending the on in
>> camel-maill.
>>
>
> --
> View this message in context: http://old.nabble.com/Camel-Mail-issue-with-unsupported-charset-tp24755585p26206832.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>

--
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

[Claus Ibsen-2]

Hi

Did you try with this code and did it work for you?


On Thu, Nov 5, 2009 at 12:50 AM, dcheckoway <dcheckoway@...> wrote:

>
> Claus, I'm not a contributor on the Camel project, but here's what I would
> advocate trying in MailBinding.java, expanding the meaning of
> "ignoreUnsupportedCharset" a bit:
>
>    public Object extractBodyFromMail(Exchange exchange, Message message) {
>        return extractBodyFromMail(exchange, message, false);
>    }
>
>    private Object extractBodyFromMail(Exchange exchange, Message message,
> boolean isSecondAttempt) {
>        try {
>            return message.getContent();
>        } catch (Exception e) {
>            if (!isSecondAttempt &&
>                (e instanceof java.io.UnsupportedEncodingException ||
>                 (e.getCause() != null &&
>                  e.getCause() instanceof
> java.io.UnsupportedEncodingException)) &&
>                configuration.isIgnoreUnsupportedCharset()) {
>                // The charset is unsupported, but we've been configured to
>                // ingore that condition.  Let's at least attempt to rewrite
>                // the Content-Type header and omit the unsupported charset.
>                // This may wreak havoc down the line, but it's better than
>                // Camel just bailing on this exchange, and ending up
> retrying
>                // this same message over and over, and failing every
> time...
>                String contentType = message.getContentType();
>                if (contentType != null) {
>                    // Wipe out the charset=... from the Content-Type header
>                    contentType =
> contentType.replaceAll("\\s*charset=[^;\\s]+", "");
>                    part.setHeader("Content-Type", contentType);
>                    // Try again...pass isSecondAttempt=true this time
>                    return extractBodyFromMail(exchange, message, true);
>                }
>            }
>            else {
>                throw new RuntimeCamelException("Failed to extract body due
> to: " + e.getMessage()
>                                                + ". Exchange: " + exchange
> + ". Message: " + message, e);
>            }
>        }
>    }
>
> What do you think?
>
>
>
> Claus Ibsen-2 wrote:
>>
>> On Wed, Nov 4, 2009 at 7:24 PM, dcheckoway <dcheckoway@...> wrote:
>>>
>>> Is there any way to "intercept" the raw mail message content before it
>>> gets
>>> passed to the java mail API?  I would be happy to manually munge the
>>> Content-Type header if I detect one of the known-to-be-funky charsets in
>>> there.
>>>
>>
>> You can use a custom MailBinding class, e.g. extending the on in
>> camel-maill.
>>
>
> --
> View this message in context: http://old.nabble.com/Camel-Mail-issue-with-unsupported-charset-tp24755585p26206832.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>

--
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

[dcheckoway]

Sorry, I haven't yet.  I've been bogged with other stuff and haven't even looked into how to build camel from source...excuses, excuses.  I hope to get around to it in the next week or so and will definitely let you know!

Thanks,
Dan

Claus Ibsen-2 wrote:

Hi

Did you try with this code and did it work for you?


On Thu, Nov 5, 2009 at 12:50 AM, dcheckoway <[e-mail]> wrote:
>
> Claus, I'm not a contributor on the Camel project, but here's what I would
> advocate trying in MailBinding.java, expanding the meaning of
> "ignoreUnsupportedCharset" a bit:
>
>    public Object extractBodyFromMail(Exchange exchange, Message message) {
>        return extractBodyFromMail(exchange, message, false);
>    }
>
>    private Object extractBodyFromMail(Exchange exchange, Message message,
> boolean isSecondAttempt) {
>        try {
>            return message.getContent();
>        } catch (Exception e) {
>            if (!isSecondAttempt &&
>                (e instanceof java.io.UnsupportedEncodingException ||
>                 (e.getCause() != null &&
>                  e.getCause() instanceof
> java.io.UnsupportedEncodingException)) &&
>                configuration.isIgnoreUnsupportedCharset()) {
>                // The charset is unsupported, but we've been configured to
>                // ingore that condition.  Let's at least attempt to rewrite
>                // the Content-Type header and omit the unsupported charset.
>                // This may wreak havoc down the line, but it's better than
>                // Camel just bailing on this exchange, and ending up
> retrying
>                // this same message over and over, and failing every
> time...
>                String contentType = message.getContentType();
>                if (contentType != null) {
>                    // Wipe out the charset=... from the Content-Type header
>                    contentType =
> contentType.replaceAll("\\s*charset=[^;\\s]+", "");
>                    part.setHeader("Content-Type", contentType);
>                    // Try again...pass isSecondAttempt=true this time
>                    return extractBodyFromMail(exchange, message, true);
>                }
>            }
>            else {
>                throw new RuntimeCamelException("Failed to extract body due
> to: " + e.getMessage()
>                                                + ". Exchange: " + exchange
> + ". Message: " + message, e);
>            }
>        }
>    }
>
> What do you think?
>
>
>
> Claus Ibsen-2 wrote:
>>
>> On Wed, Nov 4, 2009 at 7:24 PM, dcheckoway <[e-mail]> wrote:
>>>
>>> Is there any way to "intercept" the raw mail message content before it
>>> gets
>>> passed to the java mail API?  I would be happy to manually munge the
>>> Content-Type header if I detect one of the known-to-be-funky charsets in
>>> there.
>>>
>>
>> You can use a custom MailBinding class, e.g. extending the on in
>> camel-maill.
>>
>
> --
> View this message in context: http://old.nabble.com/Camel-Mail-issue-with-unsupported-charset-tp24755585p26206832.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>



--
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

[Claus Ibsen-2]

On Sun, Nov 8, 2009 at 9:38 AM, dcheckoway <dcheckoway@...> wrote:
>
> Sorry, I haven't yet.  I've been bogged with other stuff and haven't even
> looked into how to build camel from source...excuses, excuses.  I hope to
> get around to it in the next week or so and will definitely let you know!
>

Hi

I have attached a .jar with a fix I have just coded. Can you test it
on your system.
The commit is rev 833857.

> Thanks,
> Dan
>
>
>
> Claus Ibsen-2 wrote:
>>
>> Hi
>>
>> Did you try with this code and did it work for you?
>>
>>
>> On Thu, Nov 5, 2009 at 12:50 AM, dcheckoway <dcheckoway@...> wrote:
>>>
>>> Claus, I'm not a contributor on the Camel project, but here's what I
>>> would
>>> advocate trying in MailBinding.java, expanding the meaning of
>>> "ignoreUnsupportedCharset" a bit:
>>>
>>>    public Object extractBodyFromMail(Exchange exchange, Message message)
>>> {
>>>        return extractBodyFromMail(exchange, message, false);
>>>    }
>>>
>>>    private Object extractBodyFromMail(Exchange exchange, Message message,
>>> boolean isSecondAttempt) {
>>>        try {
>>>            return message.getContent();
>>>        } catch (Exception e) {
>>>            if (!isSecondAttempt &&
>>>                (e instanceof java.io.UnsupportedEncodingException ||
>>>                 (e.getCause() != null &&
>>>                  e.getCause() instanceof
>>> java.io.UnsupportedEncodingException)) &&
>>>                configuration.isIgnoreUnsupportedCharset()) {
>>>                // The charset is unsupported, but we've been configured
>>> to
>>>                // ingore that condition.  Let's at least attempt to
>>> rewrite
>>>                // the Content-Type header and omit the unsupported
>>> charset.
>>>                // This may wreak havoc down the line, but it's better
>>> than
>>>                // Camel just bailing on this exchange, and ending up
>>> retrying
>>>                // this same message over and over, and failing every
>>> time...
>>>                String contentType = message.getContentType();
>>>                if (contentType != null) {
>>>                    // Wipe out the charset=... from the Content-Type
>>> header
>>>                    contentType =
>>> contentType.replaceAll("\\s*charset=[^;\\s]+", "");
>>>                    part.setHeader("Content-Type", contentType);
>>>                    // Try again...pass isSecondAttempt=true this time
>>>                    return extractBodyFromMail(exchange, message, true);
>>>                }
>>>            }
>>>            else {
>>>                throw new RuntimeCamelException("Failed to extract body
>>> due
>>> to: " + e.getMessage()
>>>                                                + ". Exchange: " +
>>> exchange
>>> + ". Message: " + message, e);
>>>            }
>>>        }
>>>    }
>>>
>>> What do you think?
>>>
>>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> On Wed, Nov 4, 2009 at 7:24 PM, dcheckoway <dcheckoway@...> wrote:
>>>>>
>>>>> Is there any way to "intercept" the raw mail message content before it
>>>>> gets
>>>>> passed to the java mail API?  I would be happy to manually munge the
>>>>> Content-Type header if I detect one of the known-to-be-funky charsets
>>>>> in
>>>>> there.
>>>>>
>>>>
>>>> You can use a custom MailBinding class, e.g. extending the on in
>>>> camel-maill.
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/Camel-Mail-issue-with-unsupported-charset-tp24755585p26206832.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/Camel-Mail-issue-with-unsupported-charset-tp24755585p26251752.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>

--
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

[dcheckoway]

Claus,

Thanks...that got us further along but didn't quite work.  I finally got my butt in gear and started doing camel builds, and I managed to get this problem fixed.  Here's the scoop:

The Message instance we're working with is read-only.  So when calling setHeader it threw:

javax.mail.IllegalWriteException: IMAPMessage is read-only
        at com.sun.mail.imap.IMAPMessage.setHeader(IMAPMessage.java:739)
        ...

The fix was to make a copy of the message object, and then we can modify the copy.  But that's not all there is to it.  The problem cascaded out into other accessors down the line, i.e. when headers and attachments get processed.  Those actions would bump into the same UnsupportedEncodingException, because they're working with that same original instance of Message that has the funky charset.

So...the key is to call MailMessage.setMessage(ourModifiedInstance) so that the modified copy gets used in other calls down the line.

But...of course that introduced a new problem: when marking the message "SEEN" we were now marking the *copy* seen, not the original.  So it was necessary for MailMessage to hold not only the modified instance, but also the original Message instance, so that MailConsumer.processCommit could setFlag on the original.  Otherwise, the message never got marked read.

Attached is a patch for camel/trunk that works.
dcheckoway-charset-fix.diff

Thanks for all your help on this and for putting up with me.  Maybe someday I can become a committer...  :-)

Dan

Claus Ibsen-2 wrote:

On Sun, Nov 8, 2009 at 9:38 AM, dcheckoway <[e-mail]> wrote:
>
> Sorry, I haven't yet.  I've been bogged with other stuff and haven't even
> looked into how to build camel from source...excuses, excuses.  I hope to
> get around to it in the next week or so and will definitely let you know!
>

Hi

I have attached a .jar with a fix I have just coded. Can you test it
on your system.
The commit is rev 833857.

[Claus Ibsen-2]

Hi

yikes ugly story.

Could you create a JIRA ticket and attach your patch.
And if you have any kind of unit test that can verify this that would
be great as well.


On Mon, Nov 9, 2009 at 2:17 PM, dcheckoway <dcheckoway@...> wrote:

>
> Claus,
>
> Thanks...that got us further along but didn't quite work.  I finally got my
> butt in gear and started doing camel builds, and I managed to get this
> problem fixed.  Here's the scoop:
>
> The Message instance we're working with is read-only.  So when calling
> setHeader it threw:
>
> javax.mail.IllegalWriteException: IMAPMessage is read-only
>        at com.sun.mail.imap.IMAPMessage.setHeader(IMAPMessage.java:739)
>        ...
>
> The fix was to make a copy of the message object, and then we can modify the
> copy.  But that's not all there is to it.  The problem cascaded out into
> other accessors down the line, i.e. when headers and attachments get
> processed.  Those actions would bump into the same
> UnsupportedEncodingException, because they're working with that same
> original instance of Message that has the funky charset.
>
> So...the key is to call MailMessage.setMessage(ourModifiedInstance) so that
> the modified copy gets used in other calls down the line.
>
> But...of course that introduced a new problem: when marking the message
> "SEEN" we were now marking the *copy* seen, not the original.  So it was
> necessary for MailMessage to hold not only the modified instance, but also
> the original Message instance, so that MailConsumer.processCommit could
> setFlag on the original.  Otherwise, the message never got marked read.
>
> Attached is a patch for camel/trunk that works.
> http://old.nabble.com/file/p26266245/dcheckoway-charset-fix.diff
> dcheckoway-charset-fix.diff
>
> Thanks for all your help on this and for putting up with me.  Maybe someday
> I can become a committer...  :-)
>
> Dan
>
>
>
> Claus Ibsen-2 wrote:
>>
>> On Sun, Nov 8, 2009 at 9:38 AM, dcheckoway <dcheckoway@...> wrote:
>>>
>>> Sorry, I haven't yet.  I've been bogged with other stuff and haven't even
>>> looked into how to build camel from source...excuses, excuses.  I hope to
>>> get around to it in the next week or so and will definitely let you know!
>>>
>>
>> Hi
>>
>> I have attached a .jar with a fix I have just coded. Can you test it
>> on your system.
>> The commit is rev 833857.
>>
> --
> View this message in context: http://old.nabble.com/Camel-Mail-issue-with-unsupported-charset-tp24755585p26266245.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>

--
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

[dcheckoway]

You bet.  http://issues.apache.org/activemq/browse/CAMEL-2152

I'll see about trying to add a standalone unit test...

Claus Ibsen-2 wrote:

Could you create a JIRA ticket and attach your patch.
And if you have any kind of unit test that can verify this that would
be great as well.