Nabble - Python - email-sig

Re: Design Thoughts Summary

2009-11-15T10:01:21Z

On Nov 14, 2009, at 5:12 PM, Matthew Dixon Cowles wrote:

> Thank you. I am virtually 100% in agreement that this document
> represents what people have agreed on and that it represents what is
> sensible to do.

As am I. Fantastic work in pulling this all together David.

I'm a bit slammed right now, but a quick comment...

>> * The API needs to at a minimum have hooks available for an
>> application to store data on disk rather than holding everything in
>> memory.
>
> I remain unconvinced that this is worth the trouble. Yes, the Twisted
> folks say that they can't use the email module because they may be
> receiving hundreds of messages at once. But can anyone do anything
> with hundreds of messages at once other than write them to disk?
>
> And would anything actually be improved by reading hundreds of files
> at once, in small chunks, looking for MIME separators?

Mailman has a similar problem. Even if we get just a few big
messages, they can crush the system. You could argue that the MTA
should just block messages with 50MB bodies if the underlying Mailman
code can't handle it, but I still think we can do better.

I think we're fine if all the headers and MIME structure were kept in
memory it would be fine. But I do think we just want to be able to
never store the raw body content in memory (perhaps unless needed, on
demand). Mailman for example rarely cares about the bytes of say an
image/jpeg body.

I'm very psyched to see this work progress!

-Barry

_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

PGP.sig (201 bytes) Download Attachment

Re: Design Thoughts Summary

2009-11-14T15:12:36Z

[R. David Murray]

> I'm including it here to make it easier for people to comment.

Thank you. I am virtually 100% in agreement that this document
represents what people have agreed on and that it represents what is
sensible to do.

> - For non-conforming input, when is it OK to raise an error and
> return to the calling client rather than handle it? (The
> answer to this is probably 'never').

I continue to vote for never. Mostly because it would amount to
providing the same information in more than one way.

Consider code like this:

m=email.message_from_bytes(b)
for defect in m:
# See how bad the message is and
# decide what to do with it

If the parser raised parsing exceptions, I'd need something like:

try:
m=email.message_from_bytes(b)
except email.ParseError:
# Message is very bad
else:
for defect in m:
# See how bad the message is

Hardly impossible to write, but I don't see the point in making every
use of the parser look like that.

> * The API needs to at a minimum have hooks available for an
> application to store data on disk rather than holding everything in
> memory.

I remain unconvinced that this is worth the trouble. Yes, the Twisted
folks say that they can't use the email module because they may be
receiving hundreds of messages at once. But can anyone do anything
with hundreds of messages at once other than write them to disk?

And would anything actually be improved by reading hundreds of files
at once, in small chunks, looking for MIME separators?

Regards,
Matt

_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

Design Thoughts Summary

2009-11-14T14:01:49Z

I've gone through the messages from the design discussion thread and
summarized what seems to me to be the sense of the discussion [*]. Some of
this restates what is already in the Design Overview I put on the wiki,
but seemed worth repeating.

I've posted this on the wiki:

http://wiki.python.org/moin/Email%20SIG/DesignThoughts

I'm including it here to make it easier for people to comment.

[*] "sense of the discussion" as analogous to the "Sense of the
Meeting" in Quaker terminology.

Design Thoughts
===============

Guiding Thoughts
----------------

* To decide what to do with broken email we need to decide:

- To what level does the email module promise to parse
conforming wire-format into useful objects?

- For non-conforming input, when is it OK to raise an error and
return to the calling client rather than handle it? (The answer
to this is probably 'never').

- What is the API for accessing and/or mutating unparsable data,
and requesting a reparse?

* We should treat backward compatibility the way Python3 did: as something
desirable, but not something that prevents us from fixing the warts
in the current API. We can worry about a migration strategy later.

* A more property based API would be nice.

Required Elements
-----------------

* The code and documentation should state clearly what RFCs are
implemented.

* There are two parallel APIs at the outer level: a bytes API and a
string API. We expect that all user facing APIs will have both a
string and a bytes version, except in cases where there is no
sensible string version of the data (eg: binary attachments).

* Only developers working on the internals of the email package should
need to worry about how the data is actually represented in the
model.

* When raw data is fed in to the system, serializing the resulting
object back to the same data-type as the input data should result
in the exact same data whenever possible. ("Invertability")

* Once a model has been serialized, serializing it again in the
absence of model mutations should produce the same result
("Idempotence")

* All headers are represented internally as Header objects. As with
all user facing APIs, the Header object must have both a bytes and a
string API.

* Ideally, accessing data from the model should never raise an exception.

* Mutating the model should raise exceptions as early as possible.

* As in the current email package, the fundamental model is the Message
object. Message objects are recursively defined, and consist of
the following parts:

- headers: an ordered list of Header objects (in RFC 5322 terms,
this is the model of the 'header section').

+ Duplicate headers must be supported
+ Header order must be preserved
+ As in the current package, a dictionary-like interface will
probably also be provided.

- body/payload: the data encapsulated by the Message (the model
of the RFC 5322 'body')

+ A Message can be either a terminal node, in which case an
object corresponding to its MIME type can be retrieved,
or an instance of one of the multipart MIME types, in which
case a list of sub-Messages can be retrieved.

+ Access to the raw data of the body must be available

* A Message object should provide access to the following for
*any* MIME type if the input is parsable:

- the raw data that was parsed to create the Message

- the Headers built from the header section

- the transfer-decoded bytes

- if the body is not a MIME multipart, the body instantiated into
an object of some sort. The system should provide a way for
application programs to determine how payloads of specific MIME
types are instantiated into objects. This will presumably be a
registration system with a series of default handlers registered by
the email package itself for common and useful types, and a generic
MIME object for types for which there is no specific handler.
All registrations, even the one for the default 'fallback' object,
should be overridable by the application.

* The mainline APIs should emit only RFC compliant data. Access to
malformed data (for, eg, error recovery attempts) should be
via alternate APIs.

* When parsing raw data, the email package should do the right thing
wherever possible (respecting the Postel Principle). When it cannot,
it should

- construct a defect list that is exposed through the appropriate API
(the defect list should include anything not RFC compliant, even
if parsable following the Postel Principle).

- return an object that is as close as possible to the object that
would have been returned if the raw data had been fully parsable,
but whose attributes will raise errors if an attempt is made to
access an API that would rely on data that was not in fact valid.
(ex: suppose the text body cannot be decoded to Unicode via the
declared charset. An object would be returned where accessing the
string API would raise an error, while the alternate APIs would
still provide access to the transfer-decoded data).

- the raw data that was parsed to build the object should
always be accessible, whether the parse succeeded or failed.

* The API needs to at a minimum have hooks available for an application
to store data on disk rather than holding everything in memory.
This API should provide the ability to trigger disk storage based on

- an aggregate message size threshold

- a message part size threshold

- both

The email package should provide a demonstration implementation
of these hooks in the spirit of wsgiref.

* Different use cases require different levels of RFC conformance.
The API should provide both a relaxed and strict API for modifying
the model. (ex: an application might want to use the email
module to manipulate email-like messages but without the 998 character
line length limit). The parser can use the non-strict API when
building the model-plus-defect-list for a non-conformant data
stream. (It may also need additional non-API model methods).

* A registration system should also be provided for transfer-encodings,
with the RFC standard encodings provided by default. This provides
a way to handle new RFC defined encodings and for applications
to implement X- encodings.

* It should be possible for third party extensions to add converters
to the MIME part registration system (see timezones in the datetime
module for a model).

Interesting Ideas
-----------------

* An API for telling the system to store the decoded content of a
MIME part in the filename specified in the MIME headers. (If
the part is already stored on disk by the disk storage hooks,
this might be a simple rename, thus avoiding the data transfer
required if this API is not provided).

* The parsing could be lazy, only parsing the MIME sub-parts on
request. If so, there should be an API available that requests
"full parsing" of a Message be done immediately.

* Parsing of the header section and body section could also
possibly be done lazily (though parsing the body requires that at
least a certain minimum set of headers be parsed).

* After the API is fleshed out (and *only* after) we can map
__string__ and __bytes__ to appropriate elements of that API.

* It might be useful for the email package to be able to parse
the non-RFC but common '+' mailbox extension notation. If so this
should be clearly documented as an extension.

* Even the fully-decoded content of a Message could be a Property,
if we also expose a decode method that allows the application to do
the decoding itself (for example to handle weird cases such as Shift
JIS data being labeled as ISO-2022-JP). (This may simply be a specific
case of using the alternate "dirty data" interface.)

Issues
------

* How does the desire for not-quite-ducklike objects for badly formed
input data mesh with the desire for a plugin system for instantiating
the objects that represent the payloads?

* How does the registration system interact with threading? Can different
threads have different registrations?

* We believe that there are no cross-part context items that would
prevent the lazy decoding of MIME parts, but this has not been
confirmed. It is the case that when parsing an inner MIME part, access
to the boundary delimiter for the outer parts is required. There may
also be some issues for semantic parsing with Content-ID references.

* Multipart/report could be both a Message-with-sub-Messages *and* a
single specialized object type; this possibility should be
considered in the API design.

_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

Re: RFC list page added to wiki

2009-11-03T14:07:21Z

On Nov 3, 2009, at 4:17 PM, R. David Murray wrote:

> After letting myself get distracted by other matters, I'm back to
> poking
> at the email package. I've added an RFC reference page to the wiki
> based on Stephen's list:
>
> http://wiki.python.org/moin/Email%20SIG/RelevantRFCs
>
> Corrections or additions welcome.

Thanks David.

-B

_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

PGP.sig (849 bytes) Download Attachment

RFC list page added to wiki

2009-11-03T13:17:45Z

After letting myself get distracted by other matters, I'm back to poking
at the email package. I've added an RFC reference page to the wiki
based on Stephen's list:

http://wiki.python.org/moin/Email%20SIG/RelevantRFCs

Corrections or additions welcome.

--David
_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

Re: invertability and idempotence

2009-10-22T11:00:33Z

Andrew McNamara writes:

> > > didn't want to introduce new terminology. But referring to this:
> > >
> > > > generate(parse(msg)) == msg
> > >
> > > as "idempotency" is perfectly valid in my opinion (as in, applying an
> > > operation multiple times produces the same result).
> >
> >That would be generate(generate(msg)) == generate(msg) or
> >parse(parse(email)) == parse(email). The input and output of
> >these functions are of *different types*, they cannot possibly be
> >idempotent.
>
> You're splitting hairs - the operation "generate(parse(X))" is
> idempotent, and that's what I was referring to.

Yes and no. The equation above does imply idempotency, but it is a
much stronger statement: generate(parse()) is the identity. That
stronger statement could be useful in practice, but it could also be
expensive to implement. That tension could engender flamewars if the
requirement is expressed by the word "idempotency" but the intent is
"identity".

For example, suppose that for MIME multipart messages, generate() uses
"$%$%$%$%$%$" as the separator as long as no component contains that
string. Then generate(parse(msg)) will be *equivalent* but not
*identical* to msg for most messages received from non-Python-email-
using MUAs. generate(parse()) is idempotent, though. I don't think
the folks who ask for "idempotency" would be satisfied with that!

As I said earlier, if we're going to use the word "idempotent" to mean
"invertible", that's established practice, so we footnote the
Humpty-Dumpty-ism, and I can live with that. But if we're going to
try to be more accurate, let's be fully accurate.
_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

Re: invertability and idempotence

2009-10-22T04:36:12Z

On Oct 21, 2009, at 6:58 PM, R. David Murray wrote:

> But it turns out that idempotence does have a meaning in the context
> of the email module, so I think I need to remove 'depreciated' from
> my glossary[1] entry for it, and explain what it means in the context
> of the email module.

I think you're onto something here.

> For background, see issue 7119[2].
>
> Here's what I propose: _invertability_ applies to the data path
> into the parser and out of the generator. That is:
>
> generate(parse(msg)) == msg
>
> should be true whenever possible.

Agreed, where 'msg' in this context means the message text or bytes.

> On the other hand, when _constructing_ a message, sometimes not all
> data
> is filled in (in the example above, it is the MIME boundary marker).
> In that case, it is important (I think, please discuss :) that
> generating
> the message maintain _idempotency_: once you have generated the
> message,
> then if you have not further mutated the message, generating the
> message
> again should produce the _same_ output. That is:
>
> generate(msg) == generate(msg)
>
> even though the state of msg may change after the _first_ generate
> call.

"Idempotent" means: "multiple applications of the operation do not
change the result". So here where the operation is to take a message
object and generate a stream of text or bytes, this should absolutely
return the same stream if the object is not mutated between calls. I
think it's fair though that if the model is manipulated in any way, we
make no guarantees of idempotency, though we should strive for minimal
differences.

-Barry

_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

PGP.sig (849 bytes) Download Attachment

Re: invertability and idempotence

2009-10-22T02:42:43Z

> > didn't want to introduce new terminology. But referring to this:
> >
> > > generate(parse(msg)) == msg
> >
> > as "idempotency" is perfectly valid in my opinion (as in, applying an
> > operation multiple times produces the same result).
>
>That would be generate(generate(msg)) == generate(msg) or
>parse(parse(email)) == parse(email). The input and output of
>these functions are of *different types*, they cannot possibly be
>idempotent.

You're splitting hairs - the operation "generate(parse(X))" is idempotent,
and that's what I was referring to.

--
Andrew McNamara, Senior Developer, Object Craft
http://www.object-craft.com.au/
_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

Re: invertability and idempotence

2009-10-22T01:00:13Z

Andrew McNamara writes:

> The discussion had referred to idempotency up until that point, and I
> didn't want to introduce new terminology. But referring to this:
>
> > generate(parse(msg)) == msg
>
> as "idempotency" is perfectly valid in my opinion (as in, applying an
> operation multiple times produces the same result).

That would be generate(generate(msg)) == generate(msg) or
parse(parse(email)) == parse(email). The input and output of
these functions are of *different types*, they cannot possibly be
idempotent.

I'm +1 on changing to use "invertible", -0 on continuing to use
"idempotent" (since it's the traditional idiom), and -1 on using
"idempotent" to mean "is deterministic", ie, generate(msg) ==
generate(msg).

If msg changes state in an irrelevant way, it would be nice to produce
the same output from generate. But that is not "idempotency".

And we would need to specify precisely what irrelevant means. For
example, if a client of the Message class decides to specify the MIME
boundary explicitly, then the output of generate has to change IMO.
OTOH, many MIME implementations put the time of day or the generating
process into the MIME boundary. This is unnecessary (boundaries need
to be unique only message-wide, and the email package can adjust the
boundary to not conflict with message content, eg, Emacs/Gnus uses
something like "-=-=-=-=-" by default), and I would hope that email
avoids such practices when possible.

_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

Re: invertability and idempotence

2009-10-21T21:58:24Z

>You are referring here to invertability, rather than idempotence.

The discussion had referred to idempotency up until that point, and I
didn't want to introduce new terminology. But referring to this:

> generate(parse(msg)) == msg

as "idempotency" is perfectly valid in my opinion (as in, applying an
operation multiple times produces the same result).

--
Andrew McNamara, Senior Developer, Object Craft
http://www.object-craft.com.au/
_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

invertability and idempotence

2009-10-21T15:58:42Z

On Mon, 19 Oct 2009 at 15:39, Andrew McNamara wrote:
>> Just to ramble a little longer, it's been argued that we should give
>> up on idempotency, but I'm not convinced. I think people want to see
>> an email message they throw into the system come out the other end as
>> closely as possible (well, /exactly/ for well-formed messages).
>
> I, for one, would be disappointed if we lost idempotency. If people want
> a use-case, think of SpamBayes, where we read the message, do our best
> to analyse it, then insert a header or two. If this mangled messages,
> the email module would be nearly useless to SB.

You are referring here to invertability, rather than idempotence.

But it turns out that idempotence does have a meaning in the context
of the email module, so I think I need to remove 'depreciated' from
my glossary[1] entry for it, and explain what it means in the context
of the email module.

For background, see issue 7119[2].

Here's what I propose: _invertability_ applies to the data path
into the parser and out of the generator. That is:

generate(parse(msg)) == msg

should be true whenever possible.

On the other hand, when _constructing_ a message, sometimes not all data
is filled in (in the example above, it is the MIME boundary marker).
In that case, it is important (I think, please discuss :) that generating
the message maintain _idempotency_: once you have generated the message,
then if you have not further mutated the message, generating the message
again should produce the _same_ output. That is:

generate(msg) == generate(msg)

even though the state of msg may change after the _first_ generate call.

--David

[1] http://wiki.python.org/moin/Email%20SIG/Glossary
[2] http://bugs.python.org/issue7119
_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

Re: fixing the current email module

2009-10-18T22:05:10Z

>wire-format is potentially misleading because the RFCs define line-
>endings as CRLF, but we accept system native line-endings, and
>sometimes output them too.

And, in some contexts, when forwarding e-mail it is important that we
emit exactly the line endings we received, without trying to be "helpful"
and "fix" them. But, in the case of text content inserted into a message,
I think we should convert the system line endings into CRLF (possibly with
some way to override this - a "literal" mode).

--
Andrew McNamara, Senior Developer, Object Craft
http://www.object-craft.com.au/
_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

Re: fixing the current email module

2009-10-18T21:50:26Z

> > Your "hit me with your best shot" comment indicates that you want a
> > failure code or exception when the data is bad, and then a way to
> > "retry accepting errors"?
>
>My curent thinking is that the email module should return an object
>representing a partial parse. The way that you find out if it is
>partial is to try to access some data that "should" be in the object.
>If the parse succeeded, the accessor returns the data (which might be
>empty). If the parse did not succeed, you get an AttributeError.
>(This is just a paraphrase of what I wrote in response to Oleg.)

I agree - try to extract as much intelligence as we can from the malformed
message, and hold the unparseable bits in a "bad chunk" object. If
possible, when reserialising the message, e-mail the bad chunk verbatim,
or possibly with minor fixes to keep the containing MIME structure legal
if we have to. But I'd rather see "garbage-in and same garbage-out",
than "garbage-in and even worse garbage out".

Maybe the parsing should lazy where possible: don't recurse deeper
into the structure if all we're doing is looking at a top level header,
for instance.

--
Andrew McNamara, Senior Developer, Object Craft
http://www.object-craft.com.au/
_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

Re: fixing the current email module

2009-10-18T21:39:06Z

>Just to ramble a little longer, it's been argued that we should give
>up on idempotency, but I'm not convinced. I think people want to see
>an email message they throw into the system come out the other end as
>closely as possible (well, /exactly/ for well-formed messages).

I, for one, would be disappointed if we lost idempotency. If people want
a use-case, think of SpamBayes, where we read the message, do our best
to analyse it, then insert a header or two. If this mangled messages,
the email module would be nearly useless to SB.

--
Andrew McNamara, Senior Developer, Object Craft
http://www.object-craft.com.au/
_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

Re: fixing the current email module

2009-10-12T20:59:13Z

Barry Warsaw writes:

> I would proposal a radical suggestion: we treat backward compatibility
> the way Python 3 did. Nice to keep, but we can throw it over the side
> in order to fix the warts. We'll worry about migration strategy later.

+1

> Aside: I would really like to have a much more @property based API
> where appropriate.

+1

> E.g. Message.get_content_type() would be Message.content_type. And
> in this case we'd probably have message.payload_bytes or some such.
> Decoding may require additional parameters so it will probably be a
> method.

Maybe, but in general those parameters can be deduced from the
metadata. If we can use those defaults often enough, then the
default-decoded version can be a property too.

We would have to provide alternatives, though. I've seen Shift JIS
encoded Japanese labelled "ISO-2022-JP", and apparently many Japanese
MUAs actually decode that to Japanese! Not suggesting that we should
do the same, but probably the generic function that is used to decode
should be exposed as a method so that clients who encounter such
nonsense can deal with it, and override any of the metadata.
_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

Re: fixing the current email module

2009-10-12T13:54:17Z

On Oct 10, 2009, at 11:23 PM, Stephen J. Turnbull wrote:

> Right. To riff on the RFC vs. not theme ["Barry, pick up the bass
> line, need more bottom here!"], I think we should pick a list of RFCs
> we "promise" to implement as "defining" email; if we reserve any
> structures as "too obscure for us to parse," we should say so (and
> reference chapter and verse of the Holy RFC). On the other hand, of
> course as we discover common use cases for which precise
> specifications can be given, we should be flexible and implement them.
> But there should be no rush.

Although of course Rush is the most awesomest band EVAR. But I'm
slappin' and poppin' to your groove here my bruthah.

> Which RFCs?
>
> First of all, the STD 11 series (RFCs 733, 822, 2822, 5322). Here we
> have to worry about the standard's recommended format vs. the obsolete
> format because of the Postel principle. AFAIK, there is no reason not
> to insist on *producing* strictly RFC 5322 conformant messages, but I
> think we should implement both strict and lax parsers. The lax parser
> is for "daily use", the strict parser for validation.
>
> Second, the basic MIME structure RFCs: 2045-2049, 2231. (Some of
> these have been at least partially superseded by now, I think.)
>
> The mailing list header RFCs: 2369 and 2919.

Yep, yep, and yep.

> Not RFCs, per se, but an auxiliary module should provide the
> registered IANA data for the above RFCs.
>
> Strictly speaking outside of the email module, but we make use of URLs
> (RFC 3986 -- superseded?) and mimetypes data (this overlaps
> substantially with the "registered IANA data". We need to coordinate
> with the responsible maintainers for those.
>
> Ditto coordinating with modules that we share a lot of structure with,
> the "not email but very similar" like HTTP (RFC 2616), and netnews
> (NNTP = 3397 and RFC 1036).
>
> Which extensions?
>
> Er, don't you think the above is enough for now?<wink>

Surely is, at least until that U$1M grant from the PSF comes through
<wink>. Oh wait, we blew that on lunch at Pycon 2009.

-Barry

_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

PGP.sig (849 bytes) Download Attachment

Re: fixing the current email module

2009-10-12T13:47:31Z

On Oct 10, 2009, at 8:47 PM, Stephen J. Turnbull wrote:

> The question of "what should we do about broken mail" at this point
> has three components:
>
> (1) To what level do we (ie, the email module) promise to parse
> conforming wire format into useful objects?
>
> (2) For nonconforming input, when is it OK to raise an error and
> return to the calling client rather than handle it ourselves?
>
> (3) What is the API for accessing and/or mutating unparsed data, and
> requesting a reparse?
>
> I don't think we should go any farther than that.

Agreed!

-Barry

_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

PGP.sig (849 bytes) Download Attachment

Re: fixing the current email module

2009-10-12T13:45:09Z

On Oct 10, 2009, at 5:20 PM, R. David Murray wrote:

> The other is a Glossary[2]. I think most of it accurately reflects
> the
> consensus here, but in it I'm proposing to use the term 'transfer-
> decoded'
> for #3, and 'transfer-encoded' as an alternative to 'wire-format' just
> for symmetry. Comments and suggestions welcome.

wire-format is potentially misleading because the RFCs define line-
endings as CRLF, but we accept system native line-endings, and
sometimes output them too.

ready-for-another-can-of-worms-yum!-ly y'rs,
-Barry

_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

PGP.sig (849 bytes) Download Attachment

Re: fixing the current email module

2009-10-12T13:41:48Z

On Oct 9, 2009, at 2:59 PM, Glenn Linderman wrote:

> It would be good though to have standardized terms for easier
> communication. Maybe as they are chosen, they could be added to
> that Wiki RDM set up?

I like raw, transfer-decoded, decoded (or maybe fully-decoded). As
I've mentioned before, I don't think the Message or Header APIs need
to directly support transfer-decoded.

> Separate APIs would be clearer, but for compatibility,
> should .get_payload() be retained, with the flag?

No. It was a mistake that should be taken out back and shot.

I would proposal a radical suggestion: we treat backward compatibility
the way Python 3 did. Nice to keep, but we can throw it over the side
in order to fix the warts. We'll worry about migration strategy later.

Aside: I would really like to have a much more @property based API
where appropriate. E.g. Message.get_content_type() would be
Message.content_type. And in this case we'd probably have
message.payload_bytes or some such. Decoding may require additional
parameters so it will probably be a method.

> Sure, a registration system is fine. It could work for any type
> that has a method that can be registered, that accepts a binary BLOB
> and returns an appropriate typed and functioning object that can
> manipulate that type. That would mean that the application would
> have to make all the registration calls up front, instead of making
> the API calls when the objects are retrieved. Basically, if the
> email package doesn't have a registration system that the
> application can use, the application has to invent its own, so this
> is work that could benefit all applications.

I'm sure there will be lots of default content-types registered, and
there ought to be a "default" or fallback converter that can be
overridden. It should also be possible for third party extensions to
add additional converters. Models for this would be timzeone
additions for datetime, and codecs.

> Actually, although it is not common practice to have encodings other
> than the RFC defined base64 and quoted-printable, a registration
> system for converting from #1 to #3, with appropriate defaults for
> base64, quoted-printable, binary, 7bit, 8bit, would be appropriate,

That makes sense.

-Barry

_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

PGP.sig (849 bytes) Download Attachment

Re: fixing the current email module

2009-10-12T13:30:28Z

On Oct 10, 2009, at 9:59 AM, Stephen J. Turnbull wrote:

> Both. I *believe* (but it needs to be checked) that in a correctly
> formed multipart MIME object (message or part), any internal structure
> is context-free within the MIME boundaries. If that is so, then
> individual parts of the object can be stored in raw form and parsed
> lazily.

I too /think/ that's correct. There are some MIME content-types that
cause parts to be related (e.g. multipart/alternative and multipart/
related), but those are all operating at a higher level.

In practice it probably makes sense to parse all the headers right
away. Content-Type has the most bearing on parsing the rest of the
stuff, so by that time you already need to parse parameters to e.g.
get the boundary. Early on I claimed that headers were so manageable
in practice that we could implement an ordered-dictionary with
duplicates as a simple list, with linear searching and nobody would
notice. I think nobody has noticed ;).

Lazy parsing of the body does make sense. You only need to parse
enough to find end boundaries, or recurse into parsing an embedded
part. This is how the parser currently works anyway.

-Barry

_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

PGP.sig (849 bytes) Download Attachment

Re: fixing the current email module

2009-10-12T13:19:34Z

On Oct 9, 2009, at 8:54 PM, Glenn Linderman wrote:

> That certainly works, but it isn't very helpful... that forces the
> client application to reproduce the logic to parse the header value
> and decode the parts that can be decoded successfully, and that is
> exactly the sort of thing Stephen was complaining about when he
> thought I was suggesting that to be a requirement (but he was
> confused about what I was suggesting).

There are/will be utilities in the email package to make this easy. I
don't think there's a ton of benefit to be had by supporting transport-
decoded directly in the Message or Header API.

-Barry

_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

PGP.sig (849 bytes) Download Attachment

Re: fixing the current email module

2009-10-12T13:18:32Z

On Oct 9, 2009, at 7:20 PM, R. David Murray wrote:

> IMO, the appropriate way for the email package to provide the API you
> are talking about is it provide the client with a way to get at the
> raw
> byte string, which I think everyone agrees on. If the client wants to
> decode it as if it were latin-1 to process it, it can then do that.

I agree. I'm running out of time to participate in this lengthy
thread, but I just wanted to say that of the 3 accessors (raw,
transport-decoded, fully-decoded) I'm not sure transport-decoded is
all that interesting. I wouldn't support it directly in the API. I
think they library's clients are mostly going to be interested in raw
or fully-decoded values, and there will be plenty of library utilities
to get from raw to transport-decoded if they really want it.

-Barry

_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

PGP.sig (849 bytes) Download Attachment

Re: fixing the current email module

2009-10-11T02:03:21Z

Glenn Linderman writes:
> On approximately 10/10/2009 8:23 PM, came the following characters from
> the keyboard of Stephen J. Turnbull:

> > I don't think your use case of prepending a mailing list's topic or
> > serial number to an unparseable subject is realistic; in all lists I
> > know of such a message would be held for moderation, or even discarded
> > outright as spam.
>
> So if the subject is unparseable, what is the moderator to do?

That's her problem, not ours. I can think of a number of things she
can do, starting with bouncing the mail back to sender with a note
that it was broken, please fix. If the moderator is me, I might load
the mail into XEmacs and see if Gnus can grok it. Etc.

If and when we discover there are "best practices" for this situation,
we should help automate them. Until then, "it broke -- here are all
the pieces" is what we should say, IMO.

> The more we understand/discuss about how different client can function,
> the better we can design the email package.

Sure, but about this level of discussion ... "Although never is often
better than *right* now" applies, I think.

_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

Re: fixing the current email module

2009-10-11T01:34:11Z

Glenn Linderman writes:

> conformant is not in the dictionaries I've consulted.

Try these (top 3 goggle results for "conformant"):

conformant- WordWeb dictionary definition
(computing) conforming to a particular specification or standard "In
this paper we present a new approach to conformant planning". Nearest
...
www.wordwebonline.com/en/CONFORMANT - Cached - Similar -
conformant - Definition from the Merriam-Webster Online Dictionary
conformant can be found at Merriam-WebsterUnabridged.com. Click here
to start your free trial! Click here to search for another word in the
Merriam-Webster ...
www.merriam-webster.com/dictionary/conformant - Cached - Similar -
Conformance
The notion of TEI conformance is intended as an aid in describing the
format and contents of a particular document or set of documents. ...
www.tei-c.org/Guidelines/P4/html/CF.html - Cached - Similar -

A quick look at some of the results show that the word "conformant" is
typically used in a section called "conformance", which defines what
criteria are used to determine if an application is following the
standard or not. OTOH, the fact that the top three results are
dictionary definitions suggests an awful lot of people are looking up
the word in dictionaries....

> Conforming is mostly a verb, not an adjective.

Goggling gives "Results 1 - 10 of about 3,680,000 for conforming
application," but " Results 1 - 10 of about 324,000 for conformant
application." Looks like "conforming" is the preferred adjectival
form.

> but conformable and compliant are synonyms.

When used to mean "submissive." "Conformable" won't do.

> English is hard enough for ESL folks when they can find the
> words in the dictionary.

Compliant does seem to be the winner. "Results 1 - 10 of about
13,900,000 for compliant application." Conformant or conforming is
better IMHO but much less popular. Tie goes to the lusers, as usual.
_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

Re: fixing the current email module

2009-10-11T01:17:38Z

Glenn Linderman writes:

> > (3) What is the API for accessing and/or mutating unparsed data, and
> > requesting a reparse?
> >
> > I don't think we should go any farther than that.
>
> I agree with your three components; but I think the answer to (3)
> requires discussion/speculation of what clients might want to to when
> faced with errors,

I could be wrong, but I don't think it does. We don't to implement
YAGNIs.

> otherwise the API won't likely help them much, without
> reimplementing email package logic.

(1) That's why I propose parsing as much as possible, but no more.
The parts that are in email package will not only be implemented
and available, but they will already have been done. What hasn't
been done yet, the email module doesn't know how to do anyway.

(2) DRY simply doesn't apply. The logic for dealing with erroneous
data is not the same as dealing with conforming data. If it were,
we would have succeeded in the first place.

> Except it may be perfectly valid input using a standard that post-dates
> the application. Doing something reasonable with it is appropriate.

I have no idea what you're thinking of. If it's a standard we
implement, we'll handle it. If it isn't, it's not our problem.

Discussing "possibilities" is out of the realm of "useful" already.
Useful is "Existing client X does Y, and Z does it too. We can do Y
for them, faster, better, cheaper."

_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

Re: fixing the current email module

2009-10-10T22:51:50Z

On approximately 10/10/2009 10:12 PM, came the following characters from
the keyboard of R. David Murray:
> But perhaps it should be added to the Glossary itself :)

That would, to me, make it more acceptable for use. Like I said, I knew
what was meant, but tried several printed and internet dictionaries, and
didn't find it. Didn't try wordwebonline, as you might suppose!

--
Glenn -- http://nevcal.com/
===========================
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking

_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

Re: fixing the current email module

2009-10-10T22:49:25Z

On approximately 10/10/2009 5:47 PM, came the following characters from
the keyboard of Stephen J. Turnbull:

> Glenn Linderman writes:
> > On approximately 10/10/2009 8:40 AM, came the following characters from
> > the keyboard of Stephen J. Turnbull:
>
> > > So why are we discussing this? We don't even know what our mainline
> > > APIs are going to look like, why are we discussing forcibly operating
> > > on broken input?
> >
> > Use case generation. If the only way to access header values is to
> > successfully, fully, decode them, then some uses may be rendered
> > impossible, or at least difficult, even by choice of APIs.
>
> Since invertibility is a requirement, "successfully fully decoding" a
> header field is not a prerequisite to accessing it.
>
> The question of "what should we do about broken mail" at this point
> has three components:
>
> (1) To what level do we (ie, the email module) promise to parse
> conforming wire format into useful objects?
>
> (2) For nonconforming input, when is it OK to raise an error and
> return to the calling client rather than handle it ourselves?
>
> (3) What is the API for accessing and/or mutating unparsed data, and
> requesting a reparse?
>
> I don't think we should go any farther than that.
>

I agree with your three components; but I think the answer to (3)
requires discussion/speculation of what clients might want to to when
faced with errors, otherwise the API won't likely help them much,
without reimplementing email package logic. It is easy to design
"sufficient", but unhelpful, APIs. So I've been willing to discuss such
things. Maybe at too much length, and maybe with insufficient clarity
that that is what I'm discussing, for which I apologize. But I don't
think that not discussing it helps to answer (3).

> > > "Re" is a Latin abbreviation; there is no appropriate translation. ;-)
> > >
> >
> > Nonetheless, I have seen both Re: and Fwd: translated to other languages
> > (besides Latin or geek) :)
>
> Sure. This is an aspect of question (1): is this the responsibility
> of the email module?
>

I don't think the old RFCs even discuss the use of Re: and Fwd:, nor
whether they should be collapsed or translated, or even used at all.
Just checked: RFC 822 had an example that showed Re:, but RFC 2822 does
discuss it a bit, and suggests not adding duplicate Re:. Fwd: is not
mentioned at all, in those two RFCs. So no, adding and collapsing
Re:/Fwd: is not the responsibility of the email package. But making it
easy to do so, might be, as it is a common client operation. Lots of
email style guides discuss it.

> > > Maybe they are, but the email module doesn't know or care about what
> > > they do. Let's stick within what the email module is supposed to
> > > handle
> >
> > Yep, this is just use case exploration.
>
> But since by definition this is broken input, discussing what
> applications are going to want to do with it is inappropriate, IMO.
> We don't care if the app is going to prefix, suffix, or crucifix it.
> We need to specify
>
> (a) what object will hold the raw data we couldn't handle
> (b) how a calling client can retrieve the raw data
> (c) how the client can replace (or more generally mutate) that data
> (d) how the client can request a reparse from us if it attempted to
> repair the breakage at a low level rather than parse it
>
> Manipulations of text or bytes are in principle not the responsibility
> of the email module IMO; that will be done *by* the client *using* raw
> Python, not methods provided by email. I don't see how discussion of
> *what* manipulations can be done with one hand up our nose is anything
> but useless bikeshedding.
>
> If we decide that the email module can usefully provide sufficiently
> general facilities that would be convenient and hard to implement by
> general client programmers (eg, the Mailman Developers collective
> wisdom about foreign equivalents for "re" and "fwd" is surely greater
> than that of the average American programmer), we will do it by
> calling low-level methods to get and put the data, and raw Python to
> manipulate it as text or bytes

Except it may be perfectly valid input using a standard that post-dates
the application. Doing something reasonable with it is appropriate.
The email RFCs go to great lengths to make new features work reasonably
in old clients that have limited understanding; with fallback
interpretations for unknown MIME subtypes and even MIME types, and
ensuring that some type of reasonable interpretation might be done. The
RFCs define ways that new MIME types and subtypes might be defined, and
new charsets, it seems reasonable to attempt to accommodate the
possibility that such may actually be defined in the future.

If we don't discuss some of the possibilities, we'll never learn enough
to "decide that the email module can usefully provide sufficiently
general facilities that would be convenient and hard to implement by
general client programmers" :)

To me, "hard" would mean that they would have to rewrite portions of
logic that already exists in the email package, and then tweak it
slightly to compensate for not-quite-perfect data, or maybe I should
switch to saying "not-quite-perfect-or-possibly-later-standardized data" :)

--
Glenn -- http://nevcal.com/
===========================
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking

_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

Re: fixing the current email module

2009-10-10T22:15:49Z

On approximately 10/10/2009 8:23 PM, came the following characters from
the keyboard of Stephen J. Turnbull:

> Glenn Linderman writes:
> > On approximately 10/10/2009 6:59 AM, came the following characters from
> > the keyboard of Stephen J. Turnbull:
> > > Glenn Linderman writes:
> > > > On approximately 10/9/2009 8:10 AM, came the following characters from
> > > > the keyboard of Stephen J. Turnbull:
>
> > > correctly decoded data normally is stored, and is accessible in the
> > > same way. But I gather that's not what you were talking about, my
> > > mistake.
> >
> > Well, the client tells us where to store it, and we can't prevent it
> > from being the same place.
>
> Huh? No way! We decide where our data is stored. This isn't C where
> you pass around arbitrary pointers for efficiency. In particular,
> strings (whether Unicode or bytes) are not mutable. So the client can
> keep a copy if it likes, but once it hands us raw message text as
> bytes, after that we decide where we put parsed pieces and/or slices
> of the unparsed original.
>

Yes, email package can figure out where to store its copy, client
figures out where to store its copy.

We're getting better at communicating, but not 100% there yet :) I was
thinking of the case where the client asks the email package for data,
and stores it in its variable; you seem to be thinking of the case where
the client gives the email package data.

> > > So when you wrote about saving and converting to text form, without
> > > mentioning that the specific APIs, I assumed you meant the "mainline"
> > > APIs for parsing and accessing parts of a correctly formatted message.
> >
> > Mostly, I hadn't bothered about APIs yet;
>
> You may not bother about APIs, but it sure looks like you do to me.
> You can't talk about where to store stuff without touching the API.
>

Well, I'm sure there will be APIs; the names and parameters is what I
haven't bothered about yet, much, except if the discussion seemed to
require such.

> > I think that the email package should require that some special action
> > needs to be taken by the client to request not-quite-perfect data,
> > either a special parameter value, or different API, etc.
>
> That's all I need to hear, until we're ready to write specs for that
> API. (Note that a special parameter value is part of the API in a
> sense, if we specify and document what it means, so I tend to use API
> for that, too, not just for whole functions.)
>

Yes, I was just trying to be clear that it could be either case.

> > But there is nothing that says that some client might not pass that
> > all the time, and ignore the defect reports. Whether that is easy
> > to identify or not, and whether the email package wants to require
> > that the normal APIs be tried before the not-quite-perfect APIs are
> > issues for discussion.
>
> The answers are obvious to me: yes and no. You can identify whether a
> particular API has been used with standard text search tools like M-x
> occur. (For non-Emacsers, that is an Emacs command that finds all
> occurances of a particular string in the buffer.) If a program wants
> to call the quick & dirty APIs first, that's none of our business,
> except that if parsing is being done lazily we should be careful to
> update the defect list, so that the program can check them when it
> wants to.
>
> > Ultimately, the email package cannot enforce that proper case is taken
> > by the client; only code reviews of the client can encourage that.
>
> My point is not to enforce anything, not even code reviews. But by
> having separate APIs for parsed and unparsed data, code review can be
> made easier and more accurate.
>

You have to analyze the control flow as well, not just search for
existence of the API. In normal code, that should be straightforward,
but there is no guarantee that the client doesn't use spaghetti code, or
even obfuscated code, where the analysis would be hard. The API call
could exist, but never be invoked; the API call could take parameters
that never have particular values of interest at run-time. Hence, it
may or may not be easy to search the client code and figure it out. But
I agree with your stated point: we can't enforce anything about the
client code, unless we write it ourself, or have some sort of authority
over it. I intend to write a client, so I'll have control over that
one, and don't plan to obfuscate it.

> > Yes, agreed. And a special way or ways to get various algorithms for
> > attempting to interpret not-quite-perfect data, when the client thinks
> > that might be useful.
>
> I don't think we should be talking about special ways (plural) or
> "not-quite-perfect" data. At this point in the design process, we
> have *parsed* and *unparsed* data. Heuristic algorithms for
> recovering from unparsable input can be layered on top of these two
> sets of APIs, when we have *real* use cases for them. For example, I
> don't think your use case of prepending a mailing list's topic or
> serial number to an unparseable subject is realistic; in all lists I
> know of such a message would be held for moderation, or even discarded
> outright as spam.
>

So if the subject is unparseable, what is the moderator to do? He can't
read the subject if it unparseable. Perhaps he can read the body, but
it might be in the same unparseable charset. Let's say he can read the
body, and the message seems to be valid for the list, and he marks it to
be forwarded to list members. Now what is the mailing list to do, it
still can't parse the subject?

And if there is no moderator, it still may not be spam, just a mailing
list manager that doesn't understand a valid charset, likely because it
predates the definition of the charset.

> And again:
>
> > Right. And it is the more detailed structure that I was referring to...
>
> But why? There is no need to discuss it at this point, and bringing
> it up is confusing as all get-out.
>

The more we understand/discuss about how different client can function,
the better we can design the email package. We'll still not likely
cover all the possibilities, but we don't want to have tunnel vision and
declare that because Mailman works this way, that all mailing list
managers work this way, or that because we haven't discussed that some
client might do something this way, that it won't. So I have no problem
bringing clients into the discussion, to make sure that we don't
preclude their reasonable behaviors as use cases.

> > How a particular email server interprets the "stuff before the @" is
> > pretty much up to it... so as long as it does something appropriate, it
> > can interpret all or a fraction of it as a mailbox name, or could it
> > intuit a mailbox name from the body content if it wants, or even from a
> > special header. So yeah, particular interpretations of the address is
> > non-RFC stuff.
>
> Right. To riff on the RFC vs. not theme ["Barry, pick up the bass
> line, need more bottom here!"], I think we should pick a list of RFCs
> we "promise" to implement as "defining" email; if we reserve any
> structures as "too obscure for us to parse," we should say so (and
> reference chapter and verse of the Holy RFC). On the other hand, of
> course as we discover common use cases for which precise
> specifications can be given, we should be flexible and implement them.
> But there should be no rush.
>
> Which RFCs?
>
> First of all, the STD 11 series (RFCs 733, 822, 2822, 5322). Here we
> have to worry about the standard's recommended format vs. the obsolete
> format because of the Postel principle. AFAIK, there is no reason not
> to insist on *producing* strictly RFC 5322 conformant messages, but I
> think we should implement both strict and lax parsers. The lax parser
> is for "daily use", the strict parser for validation.
>
> Second, the basic MIME structure RFCs: 2045-2049, 2231. (Some of
> these have been at least partially superseded by now, I think.)
>
> The mailing list header RFCs: 2369 and 2919.
>
> Not RFCs, per se, but an auxiliary module should provide the
> registered IANA data for the above RFCs.
>
> Strictly speaking outside of the email module, but we make use of URLs
> (RFC 3986 -- superseded?) and mimetypes data (this overlaps
> substantially with the "registered IANA data". We need to coordinate
> with the responsible maintainers for those.
>
> Ditto coordinating with modules that we share a lot of structure with,
> the "not email but very similar" like HTTP (RFC 2616), and netnews
> (NNTP = 3397 and RFC 1036).
>
> Which extensions?
>
> Er, don't you think the above is enough for now?<wink>
>

It's a good list, yes.

> > Just to point out that good data can be obtained from bad email
> > messages, I think, and that that is a use case.
>
> But we already know that, and the basic idea of how to treat bad data
> (send it to a locked room without any supper). No need to rehash
> that, AFAICS from your use case.
>

Locked room is the first pass; unlocking it belongs to the heuristics,
for determined clients.

The use case wasn't at http://wiki.python.org/moin/Email%20SIG/UseCases
so I've added it there, as "Handling pathological data #2"

--
Glenn -- http://nevcal.com/
===========================
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking

_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

Re: fixing the current email module

2009-10-10T22:12:27Z

On Sun, 11 Oct 2009 at 13:01, Stephen J. Turnbull wrote:
> Glenn Linderman writes:
> > (I switched conformant to compliant,
>
> Conformant is in common use. You might be more comfortable with
> conforming.
>
> Richard Stallman points out that you comply with the law, but you
> conform to a standard. I think it's useful to make that semantic
> distinction, cf. RFC 2119 MUST vs. SHOULD or MAY.

Indeed. My regular dictionary doesn't have it, but WordWeb does:

http://www.wordwebonline.com/en/CONFORMANT

Seems to be a 'term of art' in computing rather than a regular
English word, and the most appropriate word in the context in
which I used it. But perhaps it should be added to the Glossary
itself :)

--David
_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

Re: fixing the current email module

2009-10-10T21:37:48Z

On approximately 10/10/2009 9:01 PM, came the following characters from
the keyboard of Stephen J. Turnbull:

> Glenn Linderman writes:
> > (I switched conformant to compliant,
>
> Conformant is in common use. You might be more comfortable with
> conforming.
>
> Richard Stallman points out that you comply with the law, but you
> conform to a standard. I think it's useful to make that semantic
> distinction, cf. RFC 2119 MUST vs. SHOULD or MAY.
>

conformant is not in the dictionaries I've consulted. Conforming is
mostly a verb, not an adjective.

Richard Stallman is a great programmer, but conformable and compliant
are synonyms. I don't like the word conformable, but if you appreciate
his distinction, then we should use the word conformable even though I
don't like it. But we shouldn't use the letter sequence conformant,
because although I know what you mean by it, it appears not to be a
word, and English is hard enough for ESL folks when they can find the
words in the dictionary.

--
Glenn -- http://nevcal.com/
===========================
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking

_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

Re: fixing the current email module

2009-10-10T21:01:50Z

Glenn Linderman writes:
> (I switched conformant to compliant,

Conformant is in common use. You might be more comfortable with
conforming.

Richard Stallman points out that you comply with the law, but you
conform to a standard. I think it's useful to make that semantic
distinction, cf. RFC 2119 MUST vs. SHOULD or MAY.
_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

Re: fixing the current email module

2009-10-10T20:49:38Z

R. David Murray writes:
> I have set up two more documents on the wiki. One is UseCases[1], [...].
> The other is a Glossary[2].

Thank you, very much!

> I think most of it accurately reflects the consensus here, but in
> it I'm proposing to use the term 'transfer-decoded' for #3, and
> 'transfer-encoded' as an alternative to 'wire-format' just for
> symmetry. Comments and suggestions welcome.

'Wire-format' means "you can cat it to the wire", ie, RFC-conforming
(in fact, it's the only meaning in the RFCs by definition), and for
email itself it's always bytes AFAIK (Mama don' 'low no XML roun'
here, Lord, Lord!). That's not true of all our applications, though,
especially stuff like doctests. There are also some RFCs we use such
as BASE64 (specifically relevant to transfer encodings) that are
defined in terms of characters, not bytes, so 'transfer-encoded' is
slightly different from 'wire-format'.

I think in general that kind of comment should be applied directly to
the Glossary, but what deserves general discussion is "how pedantic do
we want to be? I think the distinction made here between 'wire-format'
and 'transfer-encoded' is useful *to us*, and in general lean toward
"high pedantry" (cf how much smoke and how little fire Glenn and I are
generating!) WDOT?
_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

Re: fixing the current email module

2009-10-10T20:15:33Z

Glenn Linderman writes:
> On approximately 10/10/2009 6:59 AM, came the following characters from
> the keyboard of Stephen J. Turnbull:
> > Glenn Linderman writes:
> > > On approximately 10/9/2009 8:10 AM, came the following characters from
> > > the keyboard of Stephen J. Turnbull:

> > correctly decoded data normally is stored, and is accessible in the
> > same way. But I gather that's not what you were talking about, my
> > mistake.
>
> Well, the client tells us where to store it, and we can't prevent it
> from being the same place.

Huh? No way! We decide where our data is stored. This isn't C where
you pass around arbitrary pointers for efficiency. In particular,
strings (whether Unicode or bytes) are not mutable. So the client can
keep a copy if it likes, but once it hands us raw message text as
bytes, after that we decide where we put parsed pieces and/or slices
of the unparsed original.

> > So when you wrote about saving and converting to text form, without
> > mentioning that the specific APIs, I assumed you meant the "mainline"
> > APIs for parsing and accessing parts of a correctly formatted message.
>
> Mostly, I hadn't bothered about APIs yet;

You may not bother about APIs, but it sure looks like you do to me.
You can't talk about where to store stuff without touching the API.

> I think that the email package should require that some special action
> needs to be taken by the client to request not-quite-perfect data,
> either a special parameter value, or different API, etc.

That's all I need to hear, until we're ready to write specs for that
API. (Note that a special parameter value is part of the API in a
sense, if we specify and document what it means, so I tend to use API
for that, too, not just for whole functions.)

> But there is nothing that says that some client might not pass that
> all the time, and ignore the defect reports. Whether that is easy
> to identify or not, and whether the email package wants to require
> that the normal APIs be tried before the not-quite-perfect APIs are
> issues for discussion.

The answers are obvious to me: yes and no. You can identify whether a
particular API has been used with standard text search tools like M-x
occur. (For non-Emacsers, that is an Emacs command that finds all
occurances of a particular string in the buffer.) If a program wants
to call the quick & dirty APIs first, that's none of our business,
except that if parsing is being done lazily we should be careful to
update the defect list, so that the program can check them when it
wants to.

> Ultimately, the email package cannot enforce that proper case is taken
> by the client; only code reviews of the client can encourage that.

My point is not to enforce anything, not even code reviews. But by
having separate APIs for parsed and unparsed data, code review can be
made easier and more accurate.

> Yes, agreed. And a special way or ways to get various algorithms for
> attempting to interpret not-quite-perfect data, when the client thinks
> that might be useful.

I don't think we should be talking about special ways (plural) or
"not-quite-perfect" data. At this point in the design process, we
have *parsed* and *unparsed* data. Heuristic algorithms for
recovering from unparsable input can be layered on top of these two
sets of APIs, when we have *real* use cases for them. For example, I
don't think your use case of prepending a mailing list's topic or
serial number to an unparseable subject is realistic; in all lists I
know of such a message would be held for moderation, or even discarded
outright as spam.

And again:

> Right. And it is the more detailed structure that I was referring to...

But why? There is no need to discuss it at this point, and bringing
it up is confusing as all get-out.

> How a particular email server interprets the "stuff before the @" is
> pretty much up to it... so as long as it does something appropriate, it
> can interpret all or a fraction of it as a mailbox name, or could it
> intuit a mailbox name from the body content if it wants, or even from a
> special header. So yeah, particular interpretations of the address is
> non-RFC stuff.

Right. To riff on the RFC vs. not theme ["Barry, pick up the bass
line, need more bottom here!"], I think we should pick a list of RFCs
we "promise" to implement as "defining" email; if we reserve any
structures as "too obscure for us to parse," we should say so (and
reference chapter and verse of the Holy RFC). On the other hand, of
course as we discover common use cases for which precise
specifications can be given, we should be flexible and implement them.
But there should be no rush.

Which RFCs?

First of all, the STD 11 series (RFCs 733, 822, 2822, 5322). Here we
have to worry about the standard's recommended format vs. the obsolete
format because of the Postel principle. AFAIK, there is no reason not
to insist on *producing* strictly RFC 5322 conformant messages, but I
think we should implement both strict and lax parsers. The lax parser
is for "daily use", the strict parser for validation.

Second, the basic MIME structure RFCs: 2045-2049, 2231. (Some of
these have been at least partially superseded by now, I think.)

The mailing list header RFCs: 2369 and 2919.

Not RFCs, per se, but an auxiliary module should provide the
registered IANA data for the above RFCs.

Strictly speaking outside of the email module, but we make use of URLs
(RFC 3986 -- superseded?) and mimetypes data (this overlaps
substantially with the "registered IANA data". We need to coordinate
with the responsible maintainers for those.

Ditto coordinating with modules that we share a lot of structure with,
the "not email but very similar" like HTTP (RFC 2616), and netnews
(NNTP = 3397 and RFC 1036).

Which extensions?

Er, don't you think the above is enough for now?<wink>

> Just to point out that good data can be obtained from bad email
> messages, I think, and that that is a use case.

But we already know that, and the basic idea of how to treat bad data
(send it to a locked room without any supper). No need to rehash
that, AFAICS from your use case.

> The only context for parsing MIME parts that I'm aware of is that when
> determining the end of a nested MIME part,

Indeed, but this is Postel principle stuff, not about parsing correct
syntax. First we need to decide what to do with correct syntax, then
come up with belt and suspenders algorithms for broken mail.

> The only other cross-part context that I am aware of is Content-ID
> references. That doesn't affect parsing, but rather semantic
> interpretation, after parsing, validation, and decoding is complete.

I wasn't thinking of those, but that's a good point. Those will need
to be kept in a mapping at a higher level of the representation,
probably top-level, I guess.

_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

Re: fixing the current email module

2009-10-10T17:39:36Z

Glenn Linderman writes:
> On approximately 10/10/2009 8:40 AM, came the following characters from
> the keyboard of Stephen J. Turnbull:

> > So why are we discussing this? We don't even know what our mainline
> > APIs are going to look like, why are we discussing forcibly operating
> > on broken input?
>
> Use case generation. If the only way to access header values is to
> successfully, fully, decode them, then some uses may be rendered
> impossible, or at least difficult, even by choice of APIs.

Since invertibility is a requirement, "successfully fully decoding" a
header field is not a prerequisite to accessing it.

The question of "what should we do about broken mail" at this point
has three components:

(1) To what level do we (ie, the email module) promise to parse
conforming wire format into useful objects?

(2) For nonconforming input, when is it OK to raise an error and
return to the calling client rather than handle it ourselves?

(3) What is the API for accessing and/or mutating unparsed data, and
requesting a reparse?

I don't think we should go any farther than that.

> > "Re" is a Latin abbreviation; there is no appropriate translation. ;-)
> >
>
> Nonetheless, I have seen both Re: and Fwd: translated to other languages
> (besides Latin or geek) :)

Sure. This is an aspect of question (1): is this the responsibility
of the email module?

> > Maybe they are, but the email module doesn't know or care about what
> > they do. Let's stick within what the email module is supposed to
> > handle
>
> Yep, this is just use case exploration.

But since by definition this is broken input, discussing what
applications are going to want to do with it is inappropriate, IMO.
We don't care if the app is going to prefix, suffix, or crucifix it.
We need to specify

(a) what object will hold the raw data we couldn't handle
(b) how a calling client can retrieve the raw data
(c) how the client can replace (or more generally mutate) that data
(d) how the client can request a reparse from us if it attempted to
repair the breakage at a low level rather than parse it

Manipulations of text or bytes are in principle not the responsibility
of the email module IMO; that will be done *by* the client *using* raw
Python, not methods provided by email. I don't see how discussion of
*what* manipulations can be done with one hand up our nose is anything
but useless bikeshedding.

If we decide that the email module can usefully provide sufficiently
general facilities that would be convenient and hard to implement by
general client programmers (eg, the Mailman Developers collective
wisdom about foreign equivalents for "re" and "fwd" is surely greater
than that of the average American programmer), we will do it by
calling low-level methods to get and put the data, and raw Python to
manipulate it as text or bytes.
_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com

Re: fixing the current email module

2009-10-10T15:58:38Z

On approximately 10/10/2009 2:20 PM, came the following characters from
the keyboard of R. David Murray:

> On Fri, 9 Oct 2009 at 11:59, Glenn Linderman wrote:
>> On approximately 10/9/2009 5:05 AM, came the following characters
>> from the keyboard of Barry Warsaw:
>>> On Oct 8, 2009, at 6:39 PM, Glenn Linderman wrote:
>>> > 1) wire format. Either what came in, in the parser case, or what
>>> would > be generated.
>>> > 2) internal headers from the MIME part
>>> > 3) decoded BLOB. This means that quopri and base64 are decoded,
>>> no more > and no less. This is bytes. No headers, only payload.
>>> For > Content-Transfer-Encoding: binary, this is mostly a noop.
>>> > 4) text/* parts should also be obtainable as str()/unicode(),
>>> payload > only. This is where charset decoding is done.
>>> > > I think your talk in the next paragraph about hooks and other
>>> object > types being produced is a generalization of 4, not 3, and
>>> generally no > additional decoding needs to be done, just
>>> conversion to the right > object type (or file, or file-like object).
>>> I mostly agree with that. I've always called #4 the "decoded
>>> payload" and
>>> #3 I've usually called the "raw payload". Maybe we can bikeshed on
>>> better
>>> terms to help inform us about the API's method/attribute names.
>>
>> It would be good though to have standardized terms for easier
>> communication. Maybe as they are chosen, they could be added to that
>> Wiki RDM set up?
>
> I didn't set it up, Barry did. I just started adding stuff ;)

OK. I seem to have an account there, so made some edits.

>> My only problem with "raw" and "decoded" payload, is that there are 3
>> payload formats, not 2, so there needs to be a 3rd term,
>> corresponding to #1, #3, and #4, above. #2 is somewhat orthogonal
>> from the payload.
>>
>> To me, "raw" conjures up #1, not #3.
>
> I think I understand why Barry uses it for #3: it's the 'raw data' that
> went in to get transfer-encoded in the first place. But clearly the
> term is ambiguous.

I found it so.

> I have set up two more documents on the wiki. One is UseCases[1], and
> I've
> tried to copy into it all of the use cases that have been mentioned in
> this discussion, plus a few more. Edits welcome.

I hadn't seen UTF-16/-32/-BE/-LE mentioned in this discussion, but the
MIME RFCs do mention use cases that require them, so I added it to
RFC822 handling, but it might be better in HTTP handling? Or maybe
elsewhere?

> The other is a Glossary[2]. I think most of it accurately reflects the
> consensus here, but in it I'm proposing to use the term
> 'transfer-decoded'
> for #3, and 'transfer-encoded' as an alternative to 'wire-format' just
> for symmetry. Comments and suggestions welcome.

I like the distinction you made that 'wire format' is "in the wild", not
known to be RFC compliant, and 'transfer-encoded' be the generated type,
and compliant. I would think that if we get data as far as
'transfer-decoded', that we've (mostly) proven that the received 'wire
format' is compliant, or can be made compliant. (I switched conformant
to compliant, not finding the former at dictionary.com, and not liking
conformable which I found there, as it seems to imply able to be changed
to conform, in my head, although not in the definition).

> Any other terms of art we should record?
>
> --David
>
> [1] http://wiki.python.org/moin/Email%20SIG/UseCases
> [2] http://wiki.python.org/moin/Email%20SIG/Glossary
>

--
Glenn -- http://nevcal.com/
===========================
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking

_______________________________________________
Email-SIG mailing list
Email-SIG@...
Your options: http://mail.python.org/mailman/options/email-sig/lists%40nabble.com