Gen-ART review of draft-ietf-eai-imap-utf8-07

[Black_David]

I have been selected as the General Area Review Team (Gen-ART)
reviewer for this draft (for background on Gen-ART, please see
http://www.alvestrand.no/ietf/gen/art/gen-art-FAQ.html).

Please resolve these comments along with any other Last Call
comments you may receive.

Document: draft-ietf-eai-imap-utf8-07
Reviewer: David L. Black
Review Date: August 31, 2009
IETF LC End Date: August 31, 2009
IESG Telechat Date: September 10, 2009
 
Summary:

This draft is on the right track, but has open issues, described
in the review.

The draft appears to be in good shape, although one has to be an
IMAP expert to understand all of its implications (and I am not
an IMAP expert).  I found two open issues:

- The header upconversion behavior specification for non-UTF-8
        mailstores appears to be incomplete.
- The recommendation to support MIME header upconversion for
        "Other widely deployed MIME charsets" strikes me as
        too vague to be useful guidance to implementers.

Major issues:

Section 3.2, upconversion behavior specification appears to be
incomplete:

   If the mailstore is not UTF-8
   header native and the SELECT or EXAMINE command with UTF-8 header
   modifier succeeds, then the server MUST return results as if the
   mailstore was UTF-8 header native with upconversion requirements as
   described in Section 8.  

What happens if a header that is not upconverted is accessed with
a UTF-8 comparison string (e.g., by SEARCH)?  I presume that no
matches occur courtesy of the charset mismatch, but that needs to
be explained, as it will be a surprise to users.

Section 8 lists a number of 8859 character sets for which upconversion
of MIME headers MUST be supported, and then says "Other widely deployed
MIME charsets SHOULD be supported."  How does an implementer figure
out which character sets those would be?  As an alternative, I suggest
saying something along the lines of: any server-supported character
set that is a superset of ASCII should be supported for upconversion.
That probably leads to fewer client surprises caused by UTF-8 not
working as expected.

Minor issues:

Section 3.1, next to last paragraph needs a couple of RFC 2119
keywords:

   Mailbox
   names must comply with the Net-Unicode Definition (section 2 of
MUST >-->^^^^

   [RFC5198]) with the specific exception that they may not contain
MUST NOT >----------------------------------------->^^^^^^^

   control characters (0000-001F, 0080-009F), delete (007F), line
   separator (2028) or paragraph separator (2029).

The ABNF in this draft is extensions to ABNF specified elsewhere.
I hope that the combined ABNF grammars have been run through an
ABNF checker, but didn't see any mention of that in the IESG
comment log.  This would normally be covered by a shepherd's
report, but I did not see one.

Section 7 recommends that all IMAP clients be modified to display a
clear error when the server advertises UTF8=ONLY.  What's the
expected behavior of existing, unmodified clients?

Nits/editorial comments:

Section 2 ought to introduce what's being added to the protocol.
Adaptations of the first two sentences in Section 10 (IANA
Considerations) would suffice.

While not strictly a security consideration, it would be useful for
section 11 to point out the potential for user confusion caused by
SEARCH command match strings that have different UTF-8 representations
but display identically or similarly (strings that look like they
should match don't).

idnits 2.11.12 found a few things (I've deleted a couple of
obviously incorrect "Missing Reference:" warnings):

  Checking nits according to http://www.ietf.org/ID-Checklist.html:
 
------------------------------------------------------------------------
----

  ** There is 1 instance of too long lines in the document, the longest
one
     being 14 characters in excess of 72.


  Miscellaneous warnings:
 
------------------------------------------------------------------------
----

  == The document seems to lack the recommended RFC 2119 boilerplate,
even if
     it appears to use RFC 2119 keywords -- however, there's a paragraph
with
     a matching beginning. Boilerplate error?

     (The document does seem to have the reference to RFC 2119 which the
     ID-Checklist requires).

  Checking references for intended status: Experimental
 
------------------------------------------------------------------------
----

  == Unused Reference: 'RFC2045' is defined on line 475, but no explicit
     reference was found in the text

  == Unused Reference: 'RFC2183' is defined on line 486, but no explicit
     reference was found in the text

  ** Obsolete normative reference: RFC 1341 (Obsoleted by RFC 1521)

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
[phone]             FAX: [phone]
black_david@...        Mobile: [phone]
----------------------------------------------------
_______________________________________________
IMA mailing list
IMA@...
https://www.ietf.org/mailman/listinfo/ima

[John C Klensin-3]

David,

A comment on one issue -- speaking for myself, not the WG...

--On Monday, August 31, 2009 03:03 -0400 Black_David@...
wrote:

> I have been selected as the General Area Review Team (Gen-ART)
> reviewer for this draft (for background on Gen-ART, please see
> http://www.alvestrand.no/ietf/gen/art/gen-art-FAQ.html).
>...
 

> Major issues:
>...
> Section 8 lists a number of 8859 character sets for which
> upconversion of MIME headers MUST be supported, and then says
> "Other widely deployed MIME charsets SHOULD be supported."
> How does an implementer figure out which character sets those
> would be?  As an alternative, I suggest saying something along
> the lines of: any server-supported character set that is a
> superset of ASCII should be supported for upconversion. That
> probably leads to fewer client surprises caused by UTF-8 not
> working as expected.

FWIW, "superset of ASCII" turns out to be a nightmare
definition.  There are languages out there that use Latin-based
scripts but don't use all ASCII characters.  While I'm not aware
of any of them that use coded character sets that don't include
all of the undecorated "Latin" characters that appear in ASCII,
that combination is certainly plausible and probably does occur
somewhere.  More important, the issues with local character
coding systems are far more severe with non-Latin-based writing
systems than they are with Latin-based ones, partially because
there are different models for coding (precomposed characters
versus combining ones, format and style effectors that may
actually change the characters versus greater numbers of
precomposed forms, etc.), rendering conversion to or from
Unicode a context-dependent translation activity rather than a
character-to-character mapping one.

On the other hand, if the specification says "this needs to be
Unicode, presented in UTF-8, on the wire" then servers pretty
much know what they use (or have to deliver to mail-reading
MUAs) locally and have to support and convert to and from,
without our doing anything more than reminding them that, if
they don't do those conversions, things won't work at all well.

So, while I agree with you that the text doesn't sound
sufficiently specific, I fear that the reality is that trying to
actually make it more specific will set us up for either
requiring support for conversions that are completely
unnecessary in context or for leaving out ones that are
important.  Indeed, I suspect that the listing of specific 8859
standards will turn out to be a bad idea in the long term, but
that is something the WG decided on and, for an Experimental
document, I think it is reasonable to leave it there and examine
operational experience when that is relevant.

best,
    john

_______________________________________________
IMA mailing list
IMA@...
https://www.ietf.org/mailman/listinfo/ima

[Alexey Melnikov]

Hi  David,
Thank you for your detailed review.

Black_David@... wrote:

>I have been selected as the General Area Review Team (Gen-ART)
>reviewer for this draft (for background on Gen-ART, please see
>http://www.alvestrand.no/ietf/gen/art/gen-art-FAQ.html).
>
>Please resolve these comments along with any other Last Call
>comments you may receive.
>
>Document: draft-ietf-eai-imap-utf8-07
>Reviewer: David L. Black
>Review Date: August 31, 2009
>IETF LC End Date: August 31, 2009
>IESG Telechat Date: September 10, 2009
>
>Summary:
>
>This draft is on the right track, but has open issues, described
>in the review.
>
>The draft appears to be in good shape, although one has to be an
>IMAP expert to understand all of its implications (and I am not
>an IMAP expert).  I found two open issues:
>
>- The header upconversion behavior specification for non-UTF-8
> mailstores appears to be incomplete.
>- The recommendation to support MIME header upconversion for
> "Other widely deployed MIME charsets" strikes me as
> too vague to be useful guidance to implementers.
>
>Major issues:
>
>Section 3.2, upconversion behavior specification appears to be
>incomplete:
>
>   If the mailstore is not UTF-8
>   header native and the SELECT or EXAMINE command with UTF-8 header
>   modifier succeeds, then the server MUST return results as if the
>   mailstore was UTF-8 header native with upconversion requirements as
>   described in Section 8.  
>
>What happens if a header that is not upconverted is accessed with
>a UTF-8 comparison string (e.g., by SEARCH)?  I presume that no
>matches occur courtesy of the charset mismatch, but that needs to
>be explained, as it will be a surprise to users.
>

Actually no, I believe this is already fully specified in RFC 3501 (base
IMAP spec), Section 6.4.4:

      The OPTIONAL [CHARSET] specification consists of the word
      "CHARSET" followed by a registered [CHARSET].  It indicates the
      [CHARSET] of the strings that appear in the search criteria.
      [MIME-IMB] content transfer encodings, and [MIME-HDRS] strings in
      [RFC-2822]/[MIME-IMB] headers, MUST be decoded before comparing
      text in a [CHARSET] other than US-ASCII.  US-ASCII MUST be
      supported; other [CHARSET]s MAY be supported.

So any of the recognized charsets will be decoded by the server and the
right thing would happen.

>Section 8 lists a number of 8859 character sets for which upconversion
>of MIME headers MUST be supported, and then says "Other widely deployed
>MIME charsets SHOULD be supported."  How does an implementer figure
>out which character sets those would be?  As an alternative, I suggest
>saying something along the lines of: any server-supported character
>set that is a superset of ASCII should be supported for upconversion.
>That probably leads to fewer client surprises caused by UTF-8 not
>working as expected.
>  
>

I believe John Klensin has adequately responded to this point.

>Minor issues:
>
>Section 3.1, next to last paragraph needs a couple of RFC 2119
>keywords:
>
>   Mailbox
>   names must comply with the Net-Unicode Definition (section 2 of
>MUST >-->^^^^
>
>   [RFC5198]) with the specific exception that they may not contain
>MUST NOT >----------------------------------------->^^^^^^^
>
>   control characters (0000-001F, 0080-009F), delete (007F), line
>   separator (2028) or paragraph separator (2029).
>

Agreed. I've entered these as RFC Editor notes.

>The ABNF in this draft is extensions to ABNF specified elsewhere.
>I hope that the combined ABNF grammars have been run through an
>ABNF checker, but didn't see any mention of that in the IESG
>comment log.
>
Yes, both Harald Alvestrand and myself checked the ABNF.

>This would normally be covered by a shepherd's
>report, but I did not see one.
>
This is coming.

>Section 7 recommends that all IMAP clients be modified to display a
>clear error when the server advertises UTF8=ONLY.  What's the
>expected behavior of existing, unmodified clients?
>  
>
Such clients will not be using the UTF8 parameter to SELECT/EXAMINE
(mailbox opening) commands, so they will fail to do anything useful. But
this is to be expected.

>Nits/editorial comments:
>
>Section 2 ought to introduce what's being added to the protocol.
>Adaptations of the first two sentences in Section 10 (IANA
>Considerations) would suffice.
>
>While not strictly a security consideration, it would be useful for
>section 11 to point out the potential for user confusion caused by
>SEARCH command match strings that have different UTF-8 representations
>but display identically or similarly (strings that look like they
>should match don't).
>  
>

I will let editors respond/address these 2 comments.

>
>------------------------------------------------------------------------
>----
>
>  ** There is 1 instance of too long lines in the document, the longest
>one
>     being 14 characters in excess of 72.
>
>
>  Miscellaneous warnings:
>
>------------------------------------------------------------------------
>----
>
>  == The document seems to lack the recommended RFC 2119 boilerplate,
>even if
>     it appears to use RFC 2119 keywords -- however, there's a paragraph
>with
>     a matching beginning. Boilerplate error?
>
>     (The document does seem to have the reference to RFC 2119 which the
>     ID-Checklist requires).
>
>  Checking references for intended status: Experimental
>
>------------------------------------------------------------------------
>----
>
>idnits 2.11.12 found a few things (I've deleted a couple of
>obviously incorrect "Missing Reference:" warnings):
>
>  Checking nits according to http://www.ietf.org/ID-Checklist.html:  == Unused Reference: 'RFC2045' is defined on line 475, but no explicit
>     reference was found in the text
>  
>

Not sure if this one is needed or not.

>  == Unused Reference: 'RFC2183' is defined on line 486, but no explicit
>     reference was found in the text
>  
>
I've entered an RFC Editor note that updates the document to reference
this RFC.

>  ** Obsolete normative reference: RFC 1341 (Obsoleted by RFC 1521)  
>
According to authors this reference is intentional.

_______________________________________________
IMA mailing list
IMA@...
https://www.ietf.org/mailman/listinfo/ima

[Black_David]

John,

I freely admit to not being a character set expert.  The core
question from the review was:

> > Section 8 lists a number of 8859 character sets for which
> > upconversion of MIME headers MUST be supported, and then says
> > "Other widely deployed MIME charsets SHOULD be supported."
> > How does an implementer figure out which character sets those
> > would be?

My attempt to improve on this seems to have missed the mark:

> FWIW, "superset of ASCII" turns out to be a nightmare definition.

I won't disagree with that, despite the number of 8859 character
sets that are listed in the draft.

Rather, what would you suggest to address the original concern?
Specifically, you wrote:

> On the other hand, if the specification says "this needs to be
> Unicode, presented in UTF-8, on the wire" then servers pretty
> much know what they use (or have to deliver to mail-reading
> MUAs) locally and have to support and convert to and from,
> without our doing anything more than reminding them that, if
> they don't do those conversions, things won't work at all well.

Are you suggesting that the "SHOULD" for upconversion to UTF-8
ought to be on all character sets supported by the server, so
that a client can just deal with UTF-8?

Thanks,
--David
 

> -----Original Message-----
> From: John C Klensin [mailto:klensin@...]
> Sent: Tuesday, September 01, 2009 2:27 PM
> To: Black, David; presnick@...;
> chris.newman@...; gen-art@...; harald@...
> Cc: ima@...
> Subject: Re: [EAI] Gen-ART review of draft-ietf-eai-imap-utf8-07
>
> David,
>
> A comment on one issue -- speaking for myself, not the WG...
>
> --On Monday, August 31, 2009 03:03 -0400 Black_David@...
> wrote:
>
> > I have been selected as the General Area Review Team (Gen-ART)
> > reviewer for this draft (for background on Gen-ART, please see
> > http://www.alvestrand.no/ietf/gen/art/gen-art-FAQ.html).
> >...
>  
> > Major issues:
> >...
> > Section 8 lists a number of 8859 character sets for which
> > upconversion of MIME headers MUST be supported, and then says
> > "Other widely deployed MIME charsets SHOULD be supported."
> > How does an implementer figure out which character sets those
> > would be?  As an alternative, I suggest saying something along
> > the lines of: any server-supported character set that is a
> > superset of ASCII should be supported for upconversion. That
> > probably leads to fewer client surprises caused by UTF-8 not
> > working as expected.
>
> FWIW, "superset of ASCII" turns out to be a nightmare
> definition.  There are languages out there that use Latin-based
> scripts but don't use all ASCII characters.  While I'm not aware
> of any of them that use coded character sets that don't include
> all of the undecorated "Latin" characters that appear in ASCII,
> that combination is certainly plausible and probably does occur
> somewhere.  More important, the issues with local character
> coding systems are far more severe with non-Latin-based writing
> systems than they are with Latin-based ones, partially because
> there are different models for coding (precomposed characters
> versus combining ones, format and style effectors that may
> actually change the characters versus greater numbers of
> precomposed forms, etc.), rendering conversion to or from
> Unicode a context-dependent translation activity rather than a
> character-to-character mapping one.
>
> On the other hand, if the specification says "this needs to be
> Unicode, presented in UTF-8, on the wire" then servers pretty
> much know what they use (or have to deliver to mail-reading
> MUAs) locally and have to support and convert to and from,
> without our doing anything more than reminding them that, if
> they don't do those conversions, things won't work at all well.
>
> So, while I agree with you that the text doesn't sound
> sufficiently specific, I fear that the reality is that trying to
> actually make it more specific will set us up for either
> requiring support for conversions that are completely
> unnecessary in context or for leaving out ones that are
> important.  Indeed, I suspect that the listing of specific 8859
> standards will turn out to be a bad idea in the long term, but
> that is something the WG decided on and, for an Experimental
> document, I think it is reasonable to leave it there and examine
> operational experience when that is relevant.
>
> best,
>     john
>
>
>

_______________________________________________
IMA mailing list
IMA@...
https://www.ietf.org/mailman/listinfo/ima

[Pete Resnick]

On 9/1/09 at 10:36 PM +0100, Alexey Melnikov wrote:

>Black_David@... wrote:
>
>>- The header upconversion behavior specification for non-UTF-8
>> mailstores appears to be incomplete.
>>- The recommendation to support MIME header upconversion for
>> "Other widely deployed MIME charsets" strikes me as
>> too vague to be useful guidance to implementers.

I'll leave these as they are without further instruction.

>>Section 3.1, next to last paragraph needs a couple of RFC 2119
>>keywords:
>>
>>   Mailbox
>>   names must comply with the Net-Unicode Definition (section 2 of
>>MUST >-->^^^^
>>
>>   [RFC5198]) with the specific exception that they may not contain
>>MUST NOT >----------------------------------------->^^^^^^^
>>
>>   control characters (0000-001F, 0080-009F), delete (007F), line
>>   separator (2028) or paragraph separator (2029).
>
>Agreed. I've entered these as RFC Editor notes.

Fixed.

>>Section 7 recommends that all IMAP clients be modified to display a
>>clear error when the server advertises UTF8=ONLY.  What's the
>>expected behavior of existing, unmodified clients?
>
>Such clients will not be using the UTF8 parameter to SELECT/EXAMINE
>(mailbox opening) commands, so they will fail to do anything useful.
>But this is to be expected.

No change.

>>Nits/editorial comments:
>>
>>Section 2 ought to introduce what's being added to the protocol.
>>Adaptations of the first two sentences in Section 10 (IANA
>>Considerations) would suffice.

Done.

>>While not strictly a security consideration, it would be useful for
>>section 11 to point out the potential for user confusion caused by
>>SEARCH command match strings that have different UTF-8 representations
>>but display identically or similarly (strings that look like they
>>should match don't).

(*mumble*) This seems to me a job for RFC 3629, not this document.
(And, BTW, it does so.)

>>  ** There is 1 instance of too long lines in the document

I'll find that.

>>  == The document seems to lack the recommended RFC 2119 boilerplate

Huh? It's in there. Or is this document so old that it doesn't use
the precise recommended wording? I'll leave that for the RFC Editor.

>>idnits 2.11.12 found a few things (I've deleted a couple of
>>obviously incorrect "Missing Reference:" warnings):
>>
>>  Checking nits according to http://www.ietf.org/ID-Checklist.html: 
>>== Unused Reference: 'RFC2045' is defined on line 475, but no
>>explicit
>>     reference was found in the text
>
>Not sure if this one is needed or not.

Added a reference.

>>  == Unused Reference: 'RFC2183' is defined on line 486, but no explicit
>>     reference was found in the text
>>
>I've entered an RFC Editor note that updates the document to
>reference this RFC.

Added.

>>  ** Obsolete normative reference: RFC 1341 (Obsoleted by RFC 1521)
>According to authors this reference is intentional.

It is.

Thanks David (and Alexey).

pr
--
Pete Resnick <http://www.qualcomm.com/~presnick/>
Qualcomm Incorporated - Direct phone: [phone], Fax: [phone]
_______________________________________________
IMA mailing list
IMA@...
https://www.ietf.org/mailman/listinfo/ima

[Joseph Yee]

On 3-Sep-09, at 4:28 PM, Pete Resnick wrote:

[snip]

>
>>> While not strictly a security consideration, it would be useful for
>>> section 11 to point out the potential for user confusion caused by
>>> SEARCH command match strings that have different UTF-8  
>>> representations
>>> but display identically or similarly (strings that look like they
>>> should match don't).
>
> (*mumble*) This seems to me a job for RFC 3629, not this document.  
> (And, BTW, it does so.)

Agreed with Pete, the security concerns are user name and password.  
SEARCH is more of usability than security.  Maybe mention it in design  
rational instead? (Just suggesting, not saying it MUST or SHOULD)

Regards,
Joseph

>
>>> ** There is 1 instance of too long lines in the document
>
> I'll find that.
>
>>> == The document seems to lack the recommended RFC 2119 boilerplate
>
> Huh? It's in there. Or is this document so old that it doesn't use  
> the precise recommended wording? I'll leave that for the RFC Editor.
>
>>> idnits 2.11.12 found a few things (I've deleted a couple of
>>> obviously incorrect "Missing Reference:" warnings):
>>>
>>> Checking nits according to http://www.ietf.org/ID-Checklist.html: 
>>> == Unused Reference: 'RFC2045' is defined on line 475, but no  
>>> explicit
>>>    reference was found in the text
>>
>> Not sure if this one is needed or not.
>
> Added a reference.
>
>>> == Unused Reference: 'RFC2183' is defined on line 486, but no  
>>> explicit
>>>    reference was found in the text
>>>
>> I've entered an RFC Editor note that updates the document to  
>> reference this RFC.
>
> Added.
>
>>> ** Obsolete normative reference: RFC 1341 (Obsoleted by RFC 1521)
>> According to authors this reference is intentional.
>
> It is.
>
> Thanks David (and Alexey).
>
> pr
> --
> Pete Resnick <http://www.qualcomm.com/~presnick/>
> Qualcomm Incorporated - Direct phone: [phone], Fax:  
> [phone]
> _______________________________________________
> IMA mailing list
> IMA@...
> https://www.ietf.org/mailman/listinfo/ima

_______________________________________________
IMA mailing list
IMA@...
https://www.ietf.org/mailman/listinfo/ima

[Black_David]

Alexey,

> >- The header upconversion behavior specification for non-UTF-8
> > mailstores appears to be incomplete.
...
> Actually no, I believe this is already fully specified in RFC
> 3501 (base IMAP spec), Section 6.4.4:

I concur - this is not an issue.  Thanks for the pointer to the
relevant portion of RFC 3501.

> >- The recommendation to support MIME header upconversion for
> > "Other widely deployed MIME charsets" strikes me as
> > too vague to be useful guidance to implementers.
...
> I believe John Klensin has adequately responded to this point.

John has thoroughly demolished the change I suggested, but I
don't think that demolition resolves this concern, as the original
text remains vague.

OTOH, this draft is intended to be an Experimental RFC - would
it be reasonable to say that determining which charsets should
be upconverted is part of the experiment?

> >While not strictly a security consideration, it would be useful for
> >section 11 to point out the potential for user confusion caused by
> >SEARCH command match strings that have different UTF-8
representations
> >but display identically or similarly (strings that look like they
> >should match don't).

That was a nit - leaving the Section 11 text unmodified is fine,
as Section 11 already references RFC 3629.

Thanks,
--David
 

> -----Original Message-----
> From: Alexey Melnikov [mailto:alexey.melnikov@...]
> Sent: Tuesday, September 01, 2009 5:36 PM
> To: Black, David
> Cc: presnick@...; chris.newman@...;
> gen-art@...; harald@...; lee@...; ima@...
> Subject: Re: Gen-ART review of draft-ietf-eai-imap-utf8-07
>
> Hi  David,
> Thank you for your detailed review.
>
> Black_David@... wrote:
>
> >I have been selected as the General Area Review Team (Gen-ART)
> >reviewer for this draft (for background on Gen-ART, please see
> >http://www.alvestrand.no/ietf/gen/art/gen-art-FAQ.html).
> >
> >Please resolve these comments along with any other Last Call
> >comments you may receive.
> >
> >Document: draft-ietf-eai-imap-utf8-07
> >Reviewer: David L. Black
> >Review Date: August 31, 2009
> >IETF LC End Date: August 31, 2009
> >IESG Telechat Date: September 10, 2009
> >
> >Summary:
> >
> >This draft is on the right track, but has open issues, described
> >in the review.
> >
> >The draft appears to be in good shape, although one has to be an
> >IMAP expert to understand all of its implications (and I am not
> >an IMAP expert).  I found two open issues:
> >
> >- The header upconversion behavior specification for non-UTF-8
> > mailstores appears to be incomplete.
> >- The recommendation to support MIME header upconversion for
> > "Other widely deployed MIME charsets" strikes me as
> > too vague to be useful guidance to implementers.
> >
> >Major issues:
> >
> >Section 3.2, upconversion behavior specification appears to be
> >incomplete:
> >
> >   If the mailstore is not UTF-8
> >   header native and the SELECT or EXAMINE command with UTF-8 header
> >   modifier succeeds, then the server MUST return results as if the
> >   mailstore was UTF-8 header native with upconversion
> requirements as
> >   described in Section 8.  
> >
> >What happens if a header that is not upconverted is accessed with
> >a UTF-8 comparison string (e.g., by SEARCH)?  I presume that no
> >matches occur courtesy of the charset mismatch, but that needs to
> >be explained, as it will be a surprise to users.
> >
> Actually no, I believe this is already fully specified in RFC
> 3501 (base
> IMAP spec), Section 6.4.4:
>
>       The OPTIONAL [CHARSET] specification consists of the word
>       "CHARSET" followed by a registered [CHARSET].  It indicates the
>       [CHARSET] of the strings that appear in the search criteria.
>       [MIME-IMB] content transfer encodings, and [MIME-HDRS]
> strings in
>       [RFC-2822]/[MIME-IMB] headers, MUST be decoded before comparing
>       text in a [CHARSET] other than US-ASCII.  US-ASCII MUST be
>       supported; other [CHARSET]s MAY be supported.
>
> So any of the recognized charsets will be decoded by the
> server and the
> right thing would happen.
>
> >Section 8 lists a number of 8859 character sets for which
> upconversion
> >of MIME headers MUST be supported, and then says "Other
> widely deployed
> >MIME charsets SHOULD be supported."  How does an implementer figure
> >out which character sets those would be?  As an alternative,
> I suggest
> >saying something along the lines of: any server-supported character
> >set that is a superset of ASCII should be supported for upconversion.
> >That probably leads to fewer client surprises caused by UTF-8 not
> >working as expected.
> >  
> >
> I believe John Klensin has adequately responded to this point.
>
> >Minor issues:
> >
> >Section 3.1, next to last paragraph needs a couple of RFC 2119
> >keywords:
> >
> >   Mailbox
> >   names must comply with the Net-Unicode Definition (section 2 of
> >MUST >-->^^^^
> >
> >   [RFC5198]) with the specific exception that they may not contain
> >MUST NOT >----------------------------------------->^^^^^^^
> >
> >   control characters (0000-001F, 0080-009F), delete (007F), line
> >   separator (2028) or paragraph separator (2029).
> >
> Agreed. I've entered these as RFC Editor notes.
>
> >The ABNF in this draft is extensions to ABNF specified elsewhere.
> >I hope that the combined ABNF grammars have been run through an
> >ABNF checker, but didn't see any mention of that in the IESG
> >comment log.
> >
> Yes, both Harald Alvestrand and myself checked the ABNF.
>
> >This would normally be covered by a shepherd's
> >report, but I did not see one.
> >
> This is coming.
>
> >Section 7 recommends that all IMAP clients be modified to display a
> >clear error when the server advertises UTF8=ONLY.  What's the
> >expected behavior of existing, unmodified clients?
> >  
> >
> Such clients will not be using the UTF8 parameter to SELECT/EXAMINE
> (mailbox opening) commands, so they will fail to do anything
> useful. But
> this is to be expected.
>
> >Nits/editorial comments:
> >
> >Section 2 ought to introduce what's being added to the protocol.
> >Adaptations of the first two sentences in Section 10 (IANA
> >Considerations) would suffice.
> >
> >While not strictly a security consideration, it would be useful for
> >section 11 to point out the potential for user confusion caused by
> >SEARCH command match strings that have different UTF-8
> representations
> >but display identically or similarly (strings that look like they
> >should match don't).
> >  
> >
> I will let editors respond/address these 2 comments.
>
> >
> >-------------------------------------------------------------
> -----------
> >----
> >
> >  ** There is 1 instance of too long lines in the document,
> the longest
> >one
> >     being 14 characters in excess of 72.
> >
> >
> >  Miscellaneous warnings:
> >
> >-------------------------------------------------------------
> -----------
> >----
> >
> >  == The document seems to lack the recommended RFC 2119 boilerplate,
> >even if
> >     it appears to use RFC 2119 keywords -- however, there's
> a paragraph
> >with
> >     a matching beginning. Boilerplate error?
> >
> >     (The document does seem to have the reference to RFC
> 2119 which the
> >     ID-Checklist requires).
> >
> >  Checking references for intended status: Experimental
> >
> >-------------------------------------------------------------
> -----------
> >----
> >
> >idnits 2.11.12 found a few things (I've deleted a couple of
> >obviously incorrect "Missing Reference:" warnings):
> >
> >  Checking nits according to
> http://www.ietf.org/ID-Checklist.html:  == Unused Reference:
> 'RFC2045' is defined on line 475, but no explicit
> >     reference was found in the text
> >  
> >
> Not sure if this one is needed or not.
>
> >  == Unused Reference: 'RFC2183' is defined on line 486, but
> no explicit
> >     reference was found in the text
> >  
> >
> I've entered an RFC Editor note that updates the document to
> reference
> this RFC.
>
> >  ** Obsolete normative reference: RFC 1341 (Obsoleted by RFC 1521)  
> >
> According to authors this reference is intentional.
>
>
>

_______________________________________________
IMA mailing list
IMA@...
https://www.ietf.org/mailman/listinfo/ima

[Alexey Melnikov]

Black_David@... wrote:

>Alexey,
>  
>
Hi David,

>>>- The recommendation to support MIME header upconversion for
>>> "Other widely deployed MIME charsets" strikes me as
>>> too vague to be useful guidance to implementers.  
>>>
>...
>  
>
>>I believe John Klensin has adequately responded to this point.
>>    
>>
>John has thoroughly demolished the change I suggested, but I
>don't think that demolition resolves this concern, as the original
>text remains vague.
>
>OTOH, this draft is intended to be an Experimental RFC - would
>it be reasonable to say that determining which charsets should
>be upconverted is part of the experiment?
>

I am quite skeptical that we can come to a consensus after running such
experiment. The answer to the question is going to be very
region/country specific.
This is not the first time we are discussing the rathole (the same
applies to MIME, IMAP SEARCH, IMAP CONVERT extension) and this looks
like a big rathole to me.

I think you mentioned earlier that maybe an implementation should
support conversion to UTF-8 of all charsets it already supports in
SEARCH. After thinking more about this I think this proposal is quite
sensible.

Alternatively, the sentence saying:

  Other widely deployed MIME charsets SHOULD be supported

should just be removed, as I tend to agree that it doesn't provide much
useful information.

_______________________________________________
IMA mailing list
IMA@...
https://www.ietf.org/mailman/listinfo/ima

[Harald Alvestrand]

Alexey Melnikov skrev:

> Black_David@... wrote:
>
>> Alexey,
>>  
>>
> Hi David,
>
>>>> - The recommendation to support MIME header upconversion for
>>>>     "Other widely deployed MIME charsets" strikes me as
>>>>     too vague to be useful guidance to implementers.  
>> ...
>>  
>>
>>> I believe John Klensin has adequately responded to this point.
>>>  
>> John has thoroughly demolished the change I suggested, but I
>> don't think that demolition resolves this concern, as the original
>> text remains vague.
>>
>> OTOH, this draft is intended to be an Experimental RFC - would
>> it be reasonable to say that determining which charsets should
>> be upconverted is part of the experiment?
>>
> I am quite skeptical that we can come to a consensus after running
> such experiment. The answer to the question is going to be very
> region/country specific.
> This is not the first time we are discussing the rathole (the same
> applies to MIME, IMAP SEARCH, IMAP CONVERT extension) and this looks
> like a big rathole to me.
>
> I think you mentioned earlier that maybe an implementation should
> support conversion to UTF-8 of all charsets it already supports in
> SEARCH. After thinking more about this I think this proposal is quite
> sensible.

It makes quite a lot of sense to me that if an IMAP server supports a
specific charset in some other context (such as IMAP SEARCH and
friends), it should be consistent about supporting the same charset in
UTF8 too.

Inconsistency here would be unreasonable for the user - and by pointing
to that, we have "reduced to a previously unsolved problem".

Suggested language:

If the server supports other charsets in IMAP SEARCH or IMAP CONVERT
(add references to taste), it SHOULD also support those charsets in this
conversion.


              Harald

_______________________________________________
IMA mailing list
IMA@...
https://www.ietf.org/mailman/listinfo/ima

[John C Klensin-3]

--On Sunday, September 06, 2009 8:07 PM +0200 Harald Tveit
Alvestrand <harald@...> wrote:

> It makes quite a lot of sense to me that if an IMAP server
> supports a specific charset in some other context (such as
> IMAP SEARCH and friends), it should be consistent about
> supporting the same charset in UTF8 too.
>
> Inconsistency here would be unreasonable for the user - and by
> pointing to that, we have "reduced to a previously unsolved
> problem".
>
> Suggested language:
>
> If the server supports other charsets in IMAP SEARCH or IMAP
> CONVERT (add references to taste), it SHOULD also support
> those charsets in this conversion.

Wfm
    john





_______________________________________________
IMA mailing list
IMA@...
https://www.ietf.org/mailman/listinfo/ima

[Alexey Melnikov]

John C Klensin wrote:

> --On Sunday, September 06, 2009 8:07 PM +0200 Harald Tveit Alvestrand
> <harald@...> wrote:
>
>> It makes quite a lot of sense to me that if an IMAP server
>> supports a specific charset in some other context (such as
>> IMAP SEARCH and friends), it should be consistent about
>> supporting the same charset in UTF8 too.
>>
>> Inconsistency here would be unreasonable for the user - and by
>> pointing to that, we have "reduced to a previously unsolved
>> problem".
>>
>> Suggested language:
>>
>> If the server supports other charsets in IMAP SEARCH or IMAP
>> CONVERT (add references to taste), it SHOULD also support
>> those charsets in this conversion.
>
> Wfm

Ok, I've added this as an RFC Editor note.

_______________________________________________
IMA mailing list
IMA@...
https://www.ietf.org/mailman/listinfo/ima

[Black_David]

> John C Klensin wrote:
>
> > --On Sunday, September 06, 2009 8:07 PM +0200 Harald Tveit
> Alvestrand
> > <harald@...> wrote:
> >
> >> It makes quite a lot of sense to me that if an IMAP server
> >> supports a specific charset in some other context (such as
> >> IMAP SEARCH and friends), it should be consistent about
> >> supporting the same charset in UTF8 too.
> >>
> >> Inconsistency here would be unreasonable for the user - and by
> >> pointing to that, we have "reduced to a previously unsolved
> >> problem".
> >>
> >> Suggested language:
> >>
> >> If the server supports other charsets in IMAP SEARCH or IMAP
> >> CONVERT (add references to taste), it SHOULD also support
> >> those charsets in this conversion.
> >
> > Wfm
>
> Ok, I've added this as an RFC Editor note.

Also works for me.  Thanks for working through the details,
--David
_______________________________________________
IMA mailing list
IMA@...
https://www.ietf.org/mailman/listinfo/ima