Having thought some more, I now have different ideas.
Not totally different - maybe just more refined.
The idea of a queuing engine is still central. It should do queueing of mail items and little else.
A given queuing engine has a directory tree where it stores mail messages and control data (described below). A single process handles the whole queue but (optionally) does little of the real work.
Messages enter the queue via SMTP or LMTP. The xMTP server is fairly minimal. It only handles simple authentication and doesn't do any interesting check on addresses or content. It just bundles it all up and sticks it in the queue. This server should, therefore, not be exposed to the "Open Internet", or at least it should be configured to require authentication before accepting anything.
The xMTP service can be contacted on a TCP port or a Unix domain socket. The queue manager may fork a separate process to handle each connection. If this were done some simple IPC (e.g. pipe) would need to be set up so that the queue manager can get the names of files containing new messages.
The queue manager disposes of messages using (once again) SMTP or LMTP. Each recipient address for a message gets tagged with a destination which is initially "New". The destination maps through a config file to an xMTP service. This maybe identified by a program (to be conversed with over a socketpair), a unix domain socket, or a TCP/IP socket (possibly on another host). Each destination can specify a number of servers and each server might have other configuration such as authentication mechanism and service type (e.g. SMTP or LMTP).
A service may accept or reject (temporarily or permanently) a message, or may request that it be delivered elsewhere. This last is done using a slight extension to LMTP. As well as "551 user not local, please try user@domain", we allow "551 @destination:user@domain" to allow the server to indication a new service to send the message to. The RCPT TO command can also result in a multi-line 551 response which lists an arbitrary large number of addresses to forward the mail to.
So, how would it all work?
Incoming mail would be handled by a separate Internet-facing SMTP server which understands TLS and SMTP/AUTH and SPF and performs a number of other ad-hoc checks on the incoming mail. It will open an xMTP connection to an address-verify service and verify each incoming address. It will also open a connection to the queueing service. Those addresses which pass, and the mail message, will be forwarded to the queuing engine. The mail message will have a "Received:" and possible "Received-SPF:" header added to the top.
If the xMTP server determines that the incoming mail is a submission (From a locally authenticated source), extra header modification may be performed in-line with local policy, and non-local addresses will be accepted.
/usr/sbin/sendmail will simply open a unix-domain socket to the queueing system, pass SCM_CREDENTIALS, and then send over the relevant information.... or maybe it should be to the incoming smtp server.
The "New" destination would normally be connected to a service which processes the recipient addresses and determines whether they are suitable for local deliver, remote delivery, or something else such as delivery to some special-purpose service.
As such it would reject all addresses and so never get to see the message body.
The queueing engine would requeue for the alternate service and try again.
If a rejection without redirection was received, the address would be queued for the special "Error" destination which may result in the a bounce being generated.
A service might be MX-SMTP which performs MX lookup on domain in addresses and starts SMTP transaction to send mail to its destination. This would be a LMTP service so that it can report on the status of each address after transmission.
A service might a local delivery service which simply copies the mail item into the mailbox for each addressee.
Each service can be configured with an error-report services and time range and with a retry time-range. If a temporary failure is reported, the address is requeued for later and possible the message is delivered (with status ignored) to the error-report service.
The queue of messages would be stored in a directory tree. One subdirectorytree contains control messages, on for each item. Others contain the actual message which is never modified by the queue system.
The control messages list the filename of the actual message, the From and To addresses, and possibly other metadata. Each To address has a sequence number. When a To address is changed (e.g. by 551) a new line is added to the file with the same sequence number. Thus to find the set of current addresses, the file needs to be parsed from start to finish, and the last address with any sequence number is kept. Sequence numbers may be dotted numbers, so that if address 2 is replaced by two addresses they will be 2.1 and 2.2.
For each address there may also be a next-attempt time.
The queue manager reads through all messages and creates a list of the file names and the time when the file should next be processed. When that time comes, the queue manager will (optionally) for a child process which will read through the control message, determine which destination to try next and which addresses to pass, and will make the connection and try it out. Any results are appended to the control file. When the child finishes, the message will be re-parsed by the parent and either it will be deleted if it is finished with, or queued for a later time if that is appropriate.
With a system like this, it should be easy to distribute much of a mailsystem across several machines. It is also easy to distribute it across several authority domains (uids) so that compromises with be very limited in effect.
Providing that it isn't too hard to write a new LMTP server, it should be easy to add interesting functionality simply by writing and configuring a new services.
Some functionality that might be wanted included vacation messages, mailing lists, heuristic-mailaddress-matching.
It might also be useful to run a mailqueue engine for a single user. This would be appropriate for situation where a computer might be used by multiple users who have different mail accounts that need to be authenticated to. The per-user queue would be configured to send mail to a specific "smarthost" with suitable authentication. The queue manager would need to be run whenever the user logs in, or maybe all the time in the background. Currently this sort of functionality (queueing per-user mail for delivery when the internet is next connected) is included in various Mail User Agents. It would be nice if it could be a separate system.
