Bug in BODYSTRUCTURE

[Chris St Denis-4]

Courier-imap is incorrectly parsing the message structure of some
emails. Replies to replies from hotmail are known to be affected as an
example.

Procedure to reproduce:
1. Send a message to Hotmail from a mail client.
2. Reply to that message from Hotmail
3. Reply to the reply from mail client.
4. Reply to the reply in Hotmail
5. Message BODYSTRUCTURE data returned by courier-imap is incorrect.

For the sample message source attached, courier-imap returns
BODYSTRUCTURE ("text" "plain" NIL NIL NIL "8bit" 2630 103 NIL NIL NIL)


Tested in both courier-imap versions 4.4.1 and 4.6.0

--
Chris St Denis
Programmer
SmarttNet (www.smartt.com)
Ph: 604-473-9700 Ext. 200
-------------------------------------------
"Smart Internet Solutions For Businesses"


Return-Path: <cstdenis@...>
Delivered-To: auto@...
Received: from spam.smartt.com (spam.smartt.com [69.67.187.103])
        (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
        (No client certificate requested)
        by barium.smartt.com (Postfix) with ESMTPS id B233710E594
        for <auto@...>; Thu, 24 Sep 2009 15:40:33 -0700 (PDT)
X-ASG-Debug-ID: 1253832021-4dfb02eb0000-snjJ1A
X-Barracuda-URL: http://spam.smartt.com:8000/cgi-bin/mark.cgi
Received: from blu0-omc3-s7.blu0.hotmail.com (localhost [127.0.0.1])
        by spam.smartt.com (Spam & Virus Firewall) with ESMTP id B5A1D2CC3AD
        for <auto@...>; Thu, 24 Sep 2009 15:40:21 -0700 (PDT)
Received: from blu0-omc3-s7.blu0.hotmail.com (blu0-omc3-s7.blu0.hotmail.com [65.55.116.82]) by spam.smartt.com with ESMTP id BLZls48wHV3xknxI for <auto@...>; Thu, 24 Sep 2009 15:40:21 -0700 (PDT)
X-Barracuda-Envelope-From: cstdenis@...
Received: from BLU149-W18 ([65.55.116.72]) by blu0-omc3-s7.blu0.hotmail.com with Microsoft SMTPSVC(6.0.3790.3959);
         Thu, 24 Sep 2009 15:40:21 -0700
Message-ID: <BLU149-W188A8E33FDBDB7BDD81E9EDEDA0@...>
X-Barracuda-BBL-IP: 65.55.116.72
X-Barracuda-RBL-IP: 65.55.116.72
Content-Type: multipart/alternative;
        boundary="_733ece41-8f84-4bc8-817a-fc8ba74dd366_"
X-Originating-IP: [69.31.174.220]
From: "cstdenis ." <cstdenis@...>
To: <auto@...>
X-ASG-Orig-Subj: RE: test message
Subject: RE: test message
Date: Thu, 24 Sep 2009 22:40:21 +0000
Importance: Normal
In-Reply-To: <83db9c3d5d5ca5197bb08c465092ceb5@localhost>
References: <ae83f2d67c8fe1634630b9ea99c9839f@localhost>
 <BLU149-W55BFF0C1D417E5DD1201F6DEDA0@...>
X-OriginalArrivalTime: 24 Sep 2009 22:40:21.0047 (UTC) FILETIME=[FF782470:01CA3D67]
X-Barracuda-Connect: blu0-omc3-s7.blu0.hotmail.com[65.55.116.82]
X-Barracuda-Start-Time: 1253832021
X-Barracuda-Virus-Scanned: by SmarttNet Spam Firewall at smartt.com
X-Barracuda-Spam-Score: 0.86
X-Barracuda-Spam-Status: No, SCORE=0.86 using per-user scores of TAG_LEVEL=9.0 QUARANTINE_LEVEL=8.0 KILL_LEVEL=7.0 tests=HTML_MESSAGE, MIME_HEADER_CTYPE_ONLY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.2.9891
        Rule breakdown below
         pts rule name              description
        ---- ---------------------- --------------------------------------------------
        0.00 HTML_MESSAGE           BODY: HTML included in message
        0.86 MIME_HEADER_CTYPE_ONLY 'Content-Type' found without required MIME
                                   headers


 <83db9c3d5d5ca5197bb08c465092ceb5@localhost>
MIME-Version: 1.0

--_733ece41-8f84-4bc8-817a-fc8ba74dd366_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable


reply Back from hotmail


Date: Thu=2C 24 Sep 2009 15:40:09 -0700
From: auto@...
To: cstdenis@...
Subject: RE: test message

Another reply from Roundcube

=20

On Thu=2C 24 Sep 2009 22:39:16 +0000=2C "cstdenis ." <cstdenis@...>=
 wrote:

 This is a test reply




Date: Thu=2C 24 Sep 2009 15:38:38 -0700
From: auto@...
To: cstdenis@...
Subject: test message

test message body


--=20
this si a test sig




Insert movie times and more without leaving Hotmail=AE. See how.
=20


--=20
this si a test sig
     =0A=
_________________________________________________________________=0A=
Lauren found her dream laptop. Find the PC that=92s right for you.=0A=
http://www.microsoft.com/windows/choosepc/?ocid=3Dftp_val_wl_290=

--_733ece41-8f84-4bc8-817a-fc8ba74dd366_
Content-Type: text/html; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<style><!--
.hmmessage P
{
margin:0px=3B
padding:0px
}
body.hmmessage
{
font-size: 10pt=3B
font-family:Verdana
}
--></style>
</head>
<body class=3D'hmmessage'>
reply Back from hotmail<br><br><br><hr id=3D"stopSpelling">Date: Thu=2C 24 =
Sep 2009 15:40:09 -0700<br>From: auto@...<br>To: cstdenis@...=
om<br>Subject: RE: test message<br><br>Another reply from Roundcube<BR>
 =3B<BR>
On Thu=2C 24 Sep 2009 22:39:16 +0000=2C "cstdenis ." <=3Bcstdenis@hotmail=
.com>=3B wrote:<BR>
<blockquote style=3D"border-left: 2px solid rgb(16=2C 16=2C 255)=3B padding=
-left: 5px=3B margin-left: 5px=3B width: 100%=3B"> This is a test reply<br>=
<br><br>
<hr id=3D"ecxstopSpelling">
Date: Thu=2C 24 Sep 2009 15:38:38 -0700<br>From: auto@...<br>To: cs=
tdenis@...<br>Subject: test message<br><br>test message body<br>
<div>
<pre>-- <br>this si a test sig</pre>
</div>
<br>
<hr>
Insert movie times and more without leaving Hotmail=AE. <a href=3D"http://w=
indowslive.com/Tutorial/Hotmail/QuickAdd?ocid=3DTXT_TAGLM_WL_HM_Tutorial_Qu=
ickAdd_062009">See how.</a></blockquote>
 =3B<BR>
<div>
<pre>-- <br>this si a test sig</pre>
</div>    <br /><hr />Lauren found her dream laptop. <a href=3D'htt=
p://www.microsoft.com/windows/choosepc/?ocid=3Dftp_val_wl_290' target=3D'_n=
ew'>Find the PC that=92s right for you.</a></body>
</html>=

--_733ece41-8f84-4bc8-817a-fc8ba74dd366_--
------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
Courier-imap mailing list
Courier-imap@...
Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-imap

[Sam Varshavchik]

Chris St Denis writes:

> Courier-imap is incorrectly parsing the message structure of some
> emails.

Courier-IMAP parsed your sample message correctly, according to RFC 2822.

This appears to be a bug in your "Barracuda spam firewall" product, which
corrupted the original headers, when it processed it.




------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
Courier-imap mailing list
Courier-imap@...
Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-imap

[Chris St Denis-4]

Sam Varshavchik wrote:
> Chris St Denis writes:
>
>> Courier-imap is incorrectly parsing the message structure of some
>> emails.
>
> Courier-IMAP parsed your sample message correctly, according to RFC 2822.
>
> This appears to be a bug in your "Barracuda spam firewall" product,
> which corrupted the original headers, when it processed it.
What header(s) in particular are wrong and what should they be for it to
be correct? I will pass the data as a bug report to Barracuda and get
this resolved.

However this situation does appear to be specific to courier-imap.
Dovecot is able to parse it and Thunderbird (with courier-imap as the
server) displays it correctly (therefore it must not use BODYSTRUCTURE).

------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
Courier-imap mailing list
Courier-imap@...
Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-imap

[Sam Varshavchik]

Chris St Denis writes:

> Sam Varshavchik wrote:
>> Chris St Denis writes:
>>
>>> Courier-imap is incorrectly parsing the message structure of some
>>> emails.
>>
>> Courier-IMAP parsed your sample message correctly, according to RFC 2822.
>>
>> This appears to be a bug in your "Barracuda spam firewall" product,
>> which corrupted the original headers, when it processed it.
>
> What header(s) in particular are wrong and what should they be for it to
> be correct? I will pass the data as a bug report to Barracuda and get
> this resolved.

An RFC 2822 message consists of one or more header lines, then an empty
line, then followed by the body of the message. Then, in order for a message
to be a valid MIME message, it must include the MIME-Version: header. See
section 4 of RFC 2045. It's clear, and unambiguous. A MIME-Version: header
must be present. Your sample message does not contain a "MIME-Version:"
header, in the header portion of the message; as such it is not a MIME
message. Without a valid MIME-Version: header present, none of the MIME
headers, including Content-Type: carry any meaning.

There is a line in your message that reads "MIME-Version:", however it is
not a part of this message's header portion. The message's headers precede
the first empty line of the message, see above. In the example message
"MIME-Version:" occurs after the first empty line.

If you actually examine the message closely, Barracuda inserted its junk *in
the middle* of an existing References: header! After all of that garbage,
you can see what's obviously the last line of the original References:
header, containing the last message ID, followed by a "Mime-Version: 1.0".
However, since the junk inserted by Barracuda included a bunch of empty
lines, everything below that junk is considered a part of the message's
contents, and not its headers.

Ready!… Fire!… Aim???

> However this situation does appear to be specific to courier-imap.
> Dovecot is able to parse it

If so, it violates RFC 2045, section 4. Its wording is clear:

   Messages composed in accordance with this document MUST include such
   a header field, with the following verbatim text:

     MIME-Version: 1.0

   The presence of this header field is an assertion that the message
   has been composed in compliance with this document.

If so, it fails to check for the presence of the MIME-Version: header, so it
processes the Content-Type: header even if MIME-Version: is missing.

>                              and Thunderbird (with courier-imap as the
> server) displays it correctly (therefore it must not use BODYSTRUCTURE).

Correct. Thunderbird does not use BODYSTRUCTURE. And, it has the same sloppy
logic as Dovecot.

This is somewhat sad. Internet standards are supposed to have some meaning.
I could see ignoring something that's may be burdensome or onerous, but this
is basic, elementary stuff.




------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
Courier-imap mailing list
Courier-imap@...
Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-imap

[Chris St Denis-4]

Sam Varshavchik wrote:

> Chris St Denis writes:
>
>> Sam Varshavchik wrote:
>>> Chris St Denis writes:
>>>
>>>> Courier-imap is incorrectly parsing the message structure of some
>>>> emails.
>>>
>>> Courier-IMAP parsed your sample message correctly, according to RFC
>>> 2822.
>>>
>>> This appears to be a bug in your "Barracuda spam firewall" product,
>>> which corrupted the original headers, when it processed it.
>>
>> What header(s) in particular are wrong and what should they be for it
>> to be correct? I will pass the data as a bug report to Barracuda and
>> get this resolved.
>
> An RFC 2822 message consists of one or more header lines, then an
> empty line, then followed by the body of the message. Then, in order
> for a message to be a valid MIME message, it must include the
> MIME-Version: header. See section 4 of RFC 2045. It's clear, and
> unambiguous. A MIME-Version: header must be present. Your sample
> message does not contain a "MIME-Version:" header, in the header
> portion of the message; as such it is not a MIME message. Without a
> valid MIME-Version: header present, none of the MIME headers,
> including Content-Type: carry any meaning.
>
> There is a line in your message that reads "MIME-Version:", however it
> is not a part of this message's header portion. The message's headers
> precede the first empty line of the message, see above. In the example
> message "MIME-Version:" occurs after the first empty line.
>
> If you actually examine the message closely, Barracuda inserted its
> junk *in the middle* of an existing References: header! After all of
> that garbage, you can see what's obviously the last line of the
> original References: header, containing the last message ID, followed
> by a "Mime-Version: 1.0". However, since the junk inserted by
> Barracuda included a bunch of empty lines, everything below that junk
> is considered a part of the message's contents, and not its headers.
>
> Ready!… Fire!… Aim???
>
>> However this situation does appear to be specific to courier-imap.
>> Dovecot is able to parse it
>
> If so, it violates RFC 2045, section 4. Its wording is clear:
>
> Messages composed in accordance with this document MUST include such
> a header field, with the following verbatim text:
>
> MIME-Version: 1.0
>
> The presence of this header field is an assertion that the message
> has been composed in compliance with this document.
>
> If so, it fails to check for the presence of the MIME-Version: header,
> so it processes the Content-Type: header even if MIME-Version: is
> missing.
>
>> and Thunderbird (with courier-imap as the server) displays it
>> correctly (therefore it must not use BODYSTRUCTURE).
>
> Correct. Thunderbird does not use BODYSTRUCTURE. And, it has the same
> sloppy logic as Dovecot.
>
> This is somewhat sad. Internet standards are supposed to have some
> meaning. I could see ignoring something that's may be burdensome or
> onerous, but this is basic, elementary stuff.

Thank you for the explanation, I did not notice the broken header, I
thought it was a problem with the body portion of the mime construct.

I'll open a tick with Barracuda for them to, well, probably not do
anything about, but it's all I can do. The barracuda hardware actually
uses Postfix for it's core, so it may be a bug in Postfix's header
rewriting. Actually, this part it probably generated by it's
SpamAssassin, so maybe it's to blame. Or, maybe hotmail is just
outputting broken headers with a line break in the References header.
Either way I'll have to do some investigation.

------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
Courier-imap mailing list
Courier-imap@...
Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-imap

[Chris St Denis-4]

Chris St Denis wrote:

Sam Varshavchik wrote:
  

Chris St Denis writes:

    

Sam Varshavchik wrote:
      

Chris St Denis writes:

        

Courier-imap is incorrectly parsing the message structure of some 
emails.
          

Courier-IMAP parsed your sample message correctly, according to RFC 
2822.

This appears to be a bug in your "Barracuda spam firewall" product, 
which corrupted the original headers, when it processed it.
        

What header(s) in particular are wrong and what should they be for it 
to be correct? I will pass the data as a bug report to Barracuda and 
get this resolved.
      

An RFC 2822 message consists of one or more header lines, then an 
empty line, then followed by the body of the message. Then, in order 
for a message to be a valid MIME message, it must include the 
MIME-Version: header. See section 4 of RFC 2045. It's clear, and 
unambiguous. A MIME-Version: header must be present. Your sample 
message does not contain a "MIME-Version:" header, in the header 
portion of the message; as such it is not a MIME message. Without a 
valid MIME-Version: header present, none of the MIME headers, 
including Content-Type: carry any meaning.

There is a line in your message that reads "MIME-Version:", however it 
is not a part of this message's header portion. The message's headers 
precede the first empty line of the message, see above. In the example 
message "MIME-Version:" occurs after the first empty line.

If you actually examine the message closely, Barracuda inserted its 
junk *in the middle* of an existing References: header! After all of 
that garbage, you can see what's obviously the last line of the 
original References: header, containing the last message ID, followed 
by a "Mime-Version: 1.0". However, since the junk inserted by 
Barracuda included a bunch of empty lines, everything below that junk 
is considered a part of the message's contents, and not its headers.

Ready!… Fire!… Aim???

    

However this situation does appear to be specific to courier-imap. 
Dovecot is able to parse it
      

If so, it violates RFC 2045, section 4. Its wording is clear:

Messages composed in accordance with this document MUST include such
a header field, with the following verbatim text:

MIME-Version: 1.0

The presence of this header field is an assertion that the message
has been composed in compliance with this document.

If so, it fails to check for the presence of the MIME-Version: header, 
so it processes the Content-Type: header even if MIME-Version: is 
missing.

    

and Thunderbird (with courier-imap as the server) displays it 
correctly (therefore it must not use BODYSTRUCTURE).
      

Correct. Thunderbird does not use BODYSTRUCTURE. And, it has the same 
sloppy logic as Dovecot.

This is somewhat sad. Internet standards are supposed to have some 
meaning. I could see ignoring something that's may be burdensome or 
onerous, but this is basic, elementary stuff.
    

Thank you for the explanation, I did not notice the broken header, I 
thought it was a problem with the body portion of the mime construct.

I'll open a tick with Barracuda for them to, well, probably not do 
anything about, but it's all I can do. The barracuda hardware actually 
uses Postfix for it's core, so it may be a bug in Postfix's header 
rewriting. Actually, this part it probably generated by it's 
SpamAssassin, so maybe it's to blame. Or, maybe hotmail is just 
outputting broken headers with a line break in the References header. 
Either way I'll have to do some investigation

For the archives, it definitely looks like a bug in the Hotmail side. A packet sniff shows it is coming in with References already broken by X-OriginalArrivalTime.

#
T 65.55.116.49:23588 -> [removed]:25 [A]
Received: from BLU149-W41 ([65.55.116.8]) by blu0-omc1-s38.blu0.hotmail.com with Microsoft SMTPSVC(6.0.3790.3959);.
. Thu, 1 Oct 2009 14:47:35 -0700.
Message-ID: BLU149-W41208AD07A27269B47CCBBDED30@....
Return-Path: cstdenis@....
Content-Type: multipart/alternative;.
.boundary="_1adbe0ca-ce21-4c29-a6e0-ea642d006ed0_".
X-Originating-IP: [removed].
From: "cstdenis ." cstdenis@....
To: <[removed]>.
Subject: RE: test message.
Date: Thu, 1 Oct 2009 21:47:34 +0000.
Importance: Normal.
In-Reply-To: <83db9c3d5d5ca5197bb08c465092ceb5@localhost>.
References: <ae83f2d67c8fe1634630b9ea99c9839f@localhost>.
 BLU149-W55BFF0C1D417E5DD1201F6DEDA0@....
X-OriginalArrivalTime: 01 Oct 2009 21:47:35.0093 (UTC) FILETIME=[C94E2650:01CA42E0].
.
.
 <83db9c3d5d5ca5197bb08c465092ceb5@localhost>.
MIME-Version: 1.0.
.
--_1adbe0ca-ce21-4c29-a6e0-ea642d006ed0_.
Content-Type: text/plain; charset="iso-8859-1".
Content-Transfer-Encoding: quoted-printable.

------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
Courier-imap mailing list
Courier-imap@...
Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-imap

[Rolf Eike Beer-5]

Chris St Denis wrote:

> For the archives, it definitely looks like a bug in the Hotmail side. A
> packet sniff shows it is coming in with References already broken by
> X-OriginalArrivalTime.

Looks like that didn't get the point that a header line can be wrapped
multiple times. They seem to append their stuff at the end of the header and
missing that multiple-break is still header. Afterwards Barracuda appends it's
own stuff at the end of the header which was already damaged by Hotmail, so
from his point of view at the end of the header.

Eike


------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
Courier-imap mailing list
Courier-imap@...
Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-imap