Pipe to DBMAIL-SMTP

[brainjuice]

Hello all,
I have researched a fair amount via the dbmail wiki, googling and searching the mailing list archives and have yet to come across clear direction and am seeking your collecive wisdom.  I am considering the use of dbmail for a project where I have email that needs to be processed and inserted into mysql.  I am thinking it would be very helpful to incorporate dbmail into the mix to properly parse and insert the email into mysql.  Ultimately what I am trying to do seems very similar to what I am certain others have done to port over existing mail from a remote IMAP server into a local dbmail instance.

At present, I am accessing/processing multiple email boxes, one at a time, via IMAP in a custom built script.  As such, I have access to the raw email message.  I was hoping to be able to fetch the email via IMAP and pipe the email directly to dbmail-smtp but I am uncertain what format dbmail-smtp is expecting.  Can I simply open a pipe to it and start shoving email across the pipe?  Though I wish it was that simple, I am guessing that is not the case.  Do you know what communications format dbmail-smtp is expecting when it is being communicated with via a un*x pipe?  Is is expecting a fully formed SMTP interchange over the pipe connection?

Alternately, if I can not pipe directly to dbmail, do you have a suggested path of least resistance?  I have thought about writing the email out from my custom script in maildir or similar format and using some middleman software to read the emails from the filesystem and send them on to dbmail.  Presumably mailbox2dbmail would serve this purpose.  This method is less desirable to me because of the the intermediary involvement of the filesystem but I am willing to consider it.  I suppose it does have some upside as I could use dbmail-lmtp as a daemon for receiving the files.

Of greatest importance, I prefer the email to be imported as-is.  Meaning, I do not want the headers disturbed or appended to by the intervening processes.  Where the process looks like the following: original IMAP account --> Custom Script accessing via IMAP --> (intervening steps) --> dbmail.  Maybe, I should just be downloading the email from the remote IMAP server and saving/inserting the messages via IMAP to the dbmail users mailbox.  Experience with any method suggested above or informed discussion (pro or con) of other solutions is greatly appreciated.

Thank you in advance for your time and consideration.

[Josh Marshall-2]

I use and recommend:

http://home.arcor.de/armin.diehl/imapcopy/imapcopy.html

For copying emails from one IMAP server to another, regardless of what
the server it is. Note that it's best to do this at a time when you
don't expect emails to be accessed, as it's not a two-way sync software.
It just copies from one spot to another. But it does the job well.

Regards,
Josh.

_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

[Jesse Norell-2]

Hello,

  A few more comments/responses to your questions:

  There are numerous options for copying, which you should find in the
documentation you've been searching.  Imap->imap is a common way, though
you do have to know (or temporarily change) the account passwords.  You
could do imap->dbmail-smtp (eg. I think fetchmail can do that, and
probably other tools).  If you have a large amount of mail to move, it
would probably be faster to do so directly from the filesystem level of
the remote mail server, if that's accessible (using something like
mailbox2dbmail you mentioned).

The poorly named dbmail-smtp (more recenctly renamed dbmail-inject) does
not talk smtp, it takes a single message as stdin and stores it in the
database.  Check the manpage for the format, I think it's just mbox (if
not, you just remove the "From " line 1 from mbox format to send it a
raw message).

  If you wanted something you can talk smtp to, use dbmail-lmtp (it of
course talks lmtp, but does what you need for delivery).  Performance is
better than spawning lots of processes/connections with dbmail-smtp, but
I don't think you can specify a mailbox to store to, everything goes to
INBOX (though I may be mistaken there).  You could probably have a
custom migration script fiddle with sieve scripts on the backend to work
around that, if it's worth the effort (imap->imap would be less work).


--
Jesse Norell
Kentec Communications, Inc.
jesse@...
_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

[Paul J Stevens]

brainjuice wrote:
> I was hoping to be able to fetch the email via IMAP and pipe the
> email directly to dbmail-smtp but I am uncertain what format dbmail-smtp is
> expecting.  Can I simply open a pipe to it and start shoving email across
> the pipe?  Though I wish it was that simple, I am guessing that is not the
> case.  Do you know what communications format dbmail-smtp is expecting when
> it is being communicated with via a un*x pipe?  Is is expecting a fully
> formed SMTP interchange over the pipe connection?

dbmail-smtp expects just a single raw rfc822 or mime-compliant message.
It is not designed for pipelining more messages in a single session. You
want to use dbmail-lmtpd for that, which expects to hear and talks LMTP
on the wire. You'd have to use a network socket, though in 2.3.x you
could also use a pipe (which I use to run functional tests against the
daemons) to hook things up.

> Alternately, if I can not pipe directly to dbmail, do you have a suggested
> path of least resistance?  I have thought about writing the email out from
> my custom script in maildir or similar format and using some middleman
> software to read the emails from the filesystem and send them on to dbmail.
> Presumably mailbox2dbmail would serve this purpose.  This method is less
> desirable to me because of the the intermediary involvement of the
> filesystem but I am willing to consider it.  I suppose it does have some
> upside as I could use dbmail-lmtp as a daemon for receiving the files.

If you want to connect an imap store to dbmail insertion, fetchmail
could do the honors of dispatching imap retrieved mail. Using the pipe
interface (dbmail-smtp, aka dbmail-deliver) or is the easy way.
Insertion can be done unconditional but will spawn a delivery process
for each message.

You could also have fetchmail deliver into dbmail over lmtp. That would
  most likely be faster, but insertion then becomes conditional.
Messages may bounce if there's no valid recipient for the message at
hand. So it depends.

> Of greatest importance, I prefer the email to be imported as-is.  Meaning, I
> do not want the headers disturbed or appended to by the intervening
> processes.

Do your own tests. Dbmail doesn't append or delete any headers but for
one exception: Lmtp delivery adds the RFC required From_ envelope
header. But gmime has been observed to change ordering and
tab-formatting, and possibly even encoding. Nothing to break message
integrity though.

> Where the process looks like the following: original IMAP
> account --> Custom Script accessing via IMAP --> (intervening steps) -->
> dbmail.  Maybe, I should just be downloading the email from the remote IMAP
> server and saving/inserting the messages via IMAP to the dbmail users
> mailbox.  Experience with any method suggested above or informed discussion
> (pro or con) of other solutions is greatly appreciated.
>
> Thank you in advance for your time and consideration.


--
   ________________________________________________________________
   Paul Stevens                                      paul at nfg.nl
   NET FACILITIES GROUP                     GPG/PGP: 1024D/11F8CD31
   The Netherlands________________________________http://www.nfg.nl
_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail