Sending attachments

[Richard Stallman]

I searched several weeks of incoming mail and disposition=inline never
appears.

Viewing text inline is very useful, but that doesn't mean sending
disposition=inline is useful.  Both rmailmm and Etach display some
text parts inline, and they don't need to see disposition=inline to do
it.

[Reiner Steib]

On Thu, Jul 09 2009, Richard Stallman wrote:

> I searched several weeks of incoming mail and disposition=inline
> never appears.

There are many examples in emacs-devel:

/var/spool/news/gmane/emacs/devel# grep '^Content-Disposition:.*inline.*filename' *|wc -l
40

Most MUAs (Mail User Agents) probably don't offer to any choice to
the user.

> Viewing text inline is very useful, but that doesn't mean sending
> disposition=inline is useful.  

"disposition" is more or less a recommendation for the receiving agent
how to display it.

> Both rmailmm and Etach display some text parts inline, and they
> don't need to see disposition=inline to do it.

Neither does Gnus (and probably MH-E, too).  Viewing inline
vs. attachment can be toggled interactively or specified through
`mm-inline-override-types'.

Bye, Reiner.
--
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

[Reiner Steib]

On Thu, Jul 09 2009, Richard Stallman wrote:

> For most files, the type can be determined from the file name and
> contents.  

That is what mml.el does.

> The simple interface should be able to do everything automatically.
> Maybe just accepting mml's default type would do the right job.

For most types, it is okay.  But e.g. for "README" text/plain +
disposition=inline is much more suitable than application/octet-stream
+ disposition=attachment.

Bye, Reiner.
--
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

[Richard Stallman]

    There are many examples in emacs-devel:

    /var/spool/news/gmane/emacs/devel# grep '^Content-Disposition:.*inline.*filename' *|wc -l
    40

If I search my incoming mail over a long enough period,
I suppose I would see them.

    "disposition" is more or less a recommendation for the receiving agent
    how to display it.

    > Both rmailmm and Etach display some text parts inline, and they
    > don't need to see disposition=inline to do it.

    Neither does Gnus (and probably MH-E, too).  Viewing inline
    vs. attachment can be toggled interactively or specified through
    `mm-inline-override-types'.

In theory, that kind of recommendation might be useful.
But in practice, since the readers decide heuristically
whether to display inline, users mostly have no reason
to go out of their way to offer a recommendation.
It is better normally not to ask.

[Stephen J. Turnbull]

Richard Stallman writes:

 >     Neither does Gnus (and probably MH-E, too).  Viewing inline
 >     vs. attachment can be toggled interactively or specified through
 >     `mm-inline-override-types'.
 >
 > In theory, that kind of recommendation might be useful.
 > But in practice, since the readers decide heuristically

True, but shouldn't we focus on users of smarter readers, and on our
own use cases?  Most Emacs users don't use such dumb readers.  They
use Gnus or another advanced implementation which will normally
respect the recommendation.

 > whether to display inline, users mostly have no reason
 > to go out of their way to offer a recommendation.
 > It is better normally not to ask.

That may be true in many users' practice.  However, I find myself
offering a recommendation about 10-20% of the time, because I disagree
with the default.  For example, some projects include images.  I
generally do not want a new version of an image displayed in the
buffer, I want it saved to the project tree.  But when I'm sending
mail to a local users' group of the last meeting with pix of
somebody's system, I often do want it inlined.  I've been asked how I
do that by non-Emacs users on several occasions, so I know that there
are readers out there that do respect this header.

Patches are another example.  Sometimes patches are intended to be
reviewed, but discouraging cut and paste from mail buffers (of
non-Emacs MUAs) is a good idea since they often munge the text of both
patches and

Sure, the user can (if they're lucky or wilful enough to be using a
good MUA like Gnus) toggle the display or save an inlined attachment
to a file if they want.  But it takes me ~0.2 seconds (1 keystroke) to
accept the default 80% of the time when it really doesn't matter, ~1
second (1 keystroke) to accept the default when it matters, and ~2
seconds (2 keystrokes with completion) to change it.

I think that's a small cost to advertise this feature, and give the
expert user the option by default.

I realize that there are users with disabilities etc such that the
cost is much higher.  However, I suggest they may be better served by
a feature that evaluates the quality of the default offered, and short
circuits the question when the program is "pretty sure" the default is
correct.  Eg, I don't think anybody really would want tar.gz displayed
inline by default.  Ditto large PDFs.

It would also be possible to specify a customizable alist of MIME
types with values 'attachment or 'inline, allowing the user to specify
ones that should automatically be given a particular disposition.
Something like that must already be part of MML, although if it's
customizable I didn't find it.

This may be way more complex than you would like, of course.

Thing is, that complexity is going to pop up at every turn as you
develop these features.

[Jason Rumney-4]

Richard Stallman wrote:
> But in practice, since the readers decide heuristically
> whether to display inline, users mostly have no reason
> to go out of their way to offer a recommendation.
> It is better normally not to ask.
>  

A good interface might  be to normally only prompt for the filename, but
if a prefix argument is given then prompt for disposition, content-type
and description also.  Then users would have a simple interface, but can
still access the full functionality if they want.

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@...>
> Date: Fri, 10 Jul 2009 18:02:28 +0900
> Cc: Reiner Steib <Reiner.Steib@...>, emacs-devel@...
>
> Thing is, that complexity is going to pop up at every turn as you
> develop these features.

True.  Which is why I thought the intent was to create something
exceedingly simple, e.g.:

  . a single command to attach a file, perhaps with C-u that allows
    finer control on MIME headers (see Jason's proposal not long ago)

  . a single command to save one attachment to a disk file, perhaps
    with C-u that would run a program on the data rather than saving
    it

Anything more complex than that is really going to resemble existing
code in mml and elsewhere, and is not worth another from-scratch
effort, IMO.

[Jason Rumney-4]

Eli Zaretskii wrote:
> Anything more complex than that is really going to resemble existing
> code in mml and elsewhere, and is not worth another from-scratch
> effort, IMO.
>  

Even the simple case involves a lot of code in the background to encode
and provide the correct headers, so it is not worth another from-scratch
effort. My suggestion was intended to be about putting a simple
interface in front of mml, since the main objection to using that for
mail-mode seems to be that the interface is too complex, and if we can
overcome that, we can avoid adding yet another mime library into Emacs.

The other things to address are hiding the mml code it inserts (I'd
suggest a file icon image for graphical displays, or buttonized
[Attached File] for text mode, but RMS has expressed a preference for
seeing the actual base64 or quoted-printable content that would be sent,
so another option is to immediately call the function that transforms
mml to actual mime instead of calling it in the send hook).

[Stephen J. Turnbull]

Eli Zaretskii writes:
 > > From: "Stephen J. Turnbull" <stephen@...>
 > > Date: Fri, 10 Jul 2009 18:02:28 +0900
 > > Cc: Reiner Steib <Reiner.Steib@...>, emacs-devel@...
 > >
 > > Thing is, that complexity is going to pop up at every turn as you
 > > develop these features.
 >
 > True.  Which is why I thought the intent was to create something
 > exceedingly simple, e.g.:

That simplicity drove me nuts as a Mule maintainer, because it didn't
try to defend against users who went beyond the parameters sendmail.el
was ready to deal with.  Eg, I would tell somebody to send a bug
report using M-x report-xemacs-bug.  That uses sendmail.el because
Gnus and VM are both in the package distribution of XEmacs.  They'd
put non-ASCII in the header because it was a phrase they were having
problems getting Mule to handle correctly, Mule would encode it to
(raw, not RFC 2047) KOI8 or something, and it would take down the bug
list because (at that time) Mailman assumed ASCII in the headers of
mail coming off the wire, and threw an exception if it didn't get it.
(Now we just get mojibake, but we've fixed our bug reporter to say "if
possible we would really appreciate if you would kill this buffer,
customize `mail-composition-agent' and start your report over" because
really people do want to attach files, sometimes with non-ASCII names,
with their reports.)

OK, so now you fix mail-mode to deal with this.  You have two options:
error on non-ASCII in the headers, or RFC 2047 encode.  This is
non-trivial if you want to be RFC conformant, because you SHOULD use
quoted printable encoding for languages like French that use ASCII
supersets, and SHOULD use BASE64 for languages like Hebrew that don't.
(In practice this is usually done by looking at the fraction of ASCII
characters, rather than by determining the language.)  See RFC 2119
for the official definitions of requirement levels of capitalized
terms like SHOULD and MUST; IMHO a SHOULD is pretty strong.

Good, good, we're making progress (at the expense of a couple hundred
lines of code unless you grab rfc2047.el from Gnus -- which is what
Ken'ichi's 10-line .emacs hook does), whose usage assume several other
libraries in the usual Gnus-y way, and is 35KB or so by itself).

OK, so now we add an attachment feature.  No problem, right?  We
already handle RFC 2047, so our headers are OK, right?  Wrong.  RFC
2047 *specifically* forbids use of the encoding it defines to specify
file names in MIME headers.  This is a MUST NOT; a failure to conform
means you have a non-conformant MUA.  This is a boolean standard, your
MUA flunks, it's not "a little bit non-conformant".

OK. so now we have a choice.  You can error on an attempt to use a
non-ASCII filename, or you can implement RFC 2231.  More complexity,
rfc2231.el is 10KB.  Or you could munge the file name in some way and
instruct the recipient to change it back, or ask the sender to change
it.

Sure, you could stop short of a full implementation of all the RFCs
related to these "minimal features" you want, but where?  Figuring out
what to to include and what not to include, and introducing code to
check for and refuse unsupported requests from the user is
non-trivial, itself, and IMO should be based on an even better
knowledge of the RFCs than simply implementing the whole RFC.

You could say, oh, we don't have to do that.  And you're right ... but
IMO a non-RFC-conforming product really isn't something you want to
distribute as part of Emacs.

Is it really worth dealing with all this when as far as Miles and I
can tell, the sendmail.el UI (including hooks and keyboard macros) is
a 100% compatible subset of the Message mode UI?  Wouldn't it be a
better idea for one of the few serious Mail mode users to just try
Message mode for a few days, and fix any problems that show up?
Reading the RFCs is not easy.

Richard's point about complexity and modularity is a good one, of
course, but Message mode isn't that complex and is reasonably
well-factored.  The complexity comes in on the reader side of Gnus,
and I admit that's an impenetrable thicket of code.  But then, it's
much much easier to write something that just produces conformant
messages in RFC 822 format than it is to have a multibackend reader
that not only deals with netnews and a host of mail formats, but also
the hideous stuff that is produced by Microsoft and Yahoo! mailers,
not to mention spammers.

[Chong Yidong]

Richard Stallman <rms@...> writes:

>     > message.el forces the loading of many other packages (I sent the list
>     > before).  So its effective complexity includes all of them.  It is
>     > tied in with Gnus, and can't be maintained on its own.
>
>     Message mode is actively maintained by the Gnus maintainers, so this
>     point is irrelevant.
>
> It is not a good thing that message mode can only be maintained by the
> Gnus maintainers.

Proper maintenance requires a familiarity with the relevant RFCs; the
Gnus maintainers have this expertise, and we should leverage this to the
greatest extent possible.

Thus, even if we keep mail-mode, due to vested interests, I am opposed
to adding a "simple" (and possibly standards-noncompliant) mime library.
We should instead make use of mml.el, because the Gnus maintainers have
already expended effort making it standards-compliant.

[Stefan Monnier]

>>>> Context-sensitive commands are good but in message-mode they seem to
>>>> require that the "--text follows this line--" separator line exists.
> [...]
>>> You should set the variable `mail-header-separator' to ""; message-mode
>>> also pays attention to this variable.
>> Still, message-mode needs to be fixed so this is not necessary.
> Please elaborate.

message-mode should accept both the empty line and "--text follows this
line--" as a header-terminator.


        Stefan

[Stefan Monnier]

>> As far as I'm concerned, mail-mode should be moved to the
>> `obsolete' directory.
> I hope that's a joke.

It's most definitely not.  AFAIK, message-mode is a proper superset of
mail-mode, so mail-mode is a prime candidate for obsolescence.

> Especially since there's no mail-mode.el to move.

I see you've figured out why I said mail-mode and not mail-mode.el.


        Stefan

[Stefan Monnier]

>     As far as I'm concerned, mail-mode should be moved to the
>     `obsolete' directory.
> Please don't do that!

Why not?  More specifically, what do you think would break if we did
something along the lines of:

   (define-obsolete-function-alias 'mail-mode 'message-mode)


-- Stefan

[Stefan Monnier]

> I can adapt and install Handa's code.  Before I do,
> does anyone see a problem with the change he proposes?

Feel free to install it.  Note that it brings mail-mode closer to
message-mode, relying on the same MIME code, so I'm very favorable
to it.  I'd go even further and implement all the remaining missing
features.  It can be done with even less code (just a defalias).


        Stefan

[Richard Stallman]

     > In theory, that kind of recommendation might be useful.
     > But in practice, since the readers decide heuristically

    True, but shouldn't we focus on users of smarter readers, and on our
    own use cases?

The readers that decide heuristically are the smarter readers, at
least in my opinion.  I would not want to use one that always obeys
the specified disposition.

    That may be true in many users' practice.  However, I find myself
    offering a recommendation about 10-20% of the time, because I disagree
    with the default.

I am not opposed to having a more complex interface like the current one
of mml-attach-file.  I just want a simpler one to be the usual one.
Using C-u to make it ask the other questions seems fine.

[Richard Stallman]

    The other things to address are hiding the mml code it inserts (I'd
    suggest a file icon image for graphical displays, or buttonized
    [Attached File] for text mode, but RMS has expressed a preference for
    seeing the actual base64 or quoted-printable content that would be sent,
    so another option is to immediately call the function that transforms
    mml to actual mime instead of calling it in the send hook).

I think it would be clearer that way, but I don't think it's important
enough to make a big deal about.

This can be done with a simple extension to mml.
mml-attach-file could optionally convert to mime when it finishes.

mml-attach-file could also detect that the message has already been
converted to mime, in which case it could unconvert using mime-to-mml
before attaching the file.

This, together with the suggestion to ask all the questions only with
C-u, would make it do what I have in mind.

[Stefan Monnier]

> My gray hair says there ain't no such thing.  If it's more powerful
> and sophisticated, there's some price to pay.

So your argument is hypothetical, rather than based on concrete
problems.  To me that counts as "one vote to make mail-mode obsolete".


        Stefan

[Stefan Monnier]

>> message.el forces the loading of many other packages (I sent the list
>> before).  So its effective complexity includes all of them.  It is
>> tied in with Gnus, and can't be maintained on its own.

Then the conclusion should be that we should work with the Gnus
maintainers to improve the modularity of the code relevant to
message-mode.  AFAIK they are in favor of such an effort (the intention
of message-mode has always been to supercede mail-mode and stay
independent from Gnus, IIUC, which is why it has its own manual).

As a matter of fact I think message-mode is already fairly well
modularized, when you consider the fact that it's developed by people
who have no real need to care about other MUAs than Gnus.

>     Message mode is actively maintained by the Gnus maintainers, so this
>     point is irrelevant.
> It is not a good thing that message mode can only be maintained by the
> Gnus maintainers.

FWIW, I'm not a Gnus maintainer, but I have made changes to message-mode
and to the MIME code of Gnus, so I don't think it's fair to say that it
can only be maintained by the Gnus maintainers.


        Stefan

[Stefan Monnier]

> The real issue that I raised is sending attachments with Mail mode.

Actually, not really.  I think "the real issue" if sending attachments
(and non-ASCII text, including in headers) in "the default
mail-composition interface".  *Very* few users actually choose
mail-mode.

> Installing Etach will deal with that.  This actually does not involve
> a change in sendmail.el, since Etach is modular.

A simpler change is to make message-mode the default UI.
This thread seems to show that noone knows of any actual difference that
users would notice (other than being able to attach files and stuff, of
course).

So I think the first step should be to change mail-user-agent to
default to message-user-agent.
The next step will then be to make Rmail use message-mode as well
(which may require some changes to message-mode, of course).


        Stefan

[Alfred M. Szmidt]

   So I think the first step should be to change mail-user-agent to
   default to message-user-agent.  The next step will then be to make
   Rmail use message-mode as well (which may require some changes to
   message-mode, of course).

PLEASE DO NOT DO THIS!  People using mail-mode have already expressed
that they prefer it over message-mode, please respect that!