TLS capable SMTP store-and-forward server -- does it have to store private keys?

Question

I know that email is inherently insecure anyway, but considering that pretty much everyone that I send email to is using a large, somewhat-trusted cloud provider, (hint, starts with G) I would expect that most of my email is sent through relatively secure MTAs that either require or prefer to use TLS.

From that assumption, I would draw the conclusion that if I wanted to host my own email server and if I cared about security, then it MUST support TLS and have a certificate signed by a trusted authority.

Ok, that's all fine, I can set that up myself. But what if I want to self-host this server? I know I can easily pay a cloud provider to host it for me, but I have an irrational desire to escape the cloud altogether and build my own infrastructure.

However, that opens up a host (no pun intended) of other problems. Email delivery from residential IPs is spotty at best. Internet access and electricity don't have 100% uptime in residential areas. Maybe I mess up my server and have to redeploy it. Or maybe I have to take it down to upgrade it.

So, maybe I want to rent a VM from a cloud provider and run an SMTP server, but at the same time I don't want my private key to leave my own machine.

I am a very capable programmer, so I'm considering actually attempting this. I'm just unsure if it is physically possible given the TLS protocol.

I was poring over this file, https://github.com/digitalbazaar/forge/blob/master/js/tls.js, a javascript implementation of TLS, trying to figure out how it could be done. I found this network state diagram:

* =======================FULL HANDSHAKE======================
* Client                                               Server
*
* ClientHello                  -------->
*                                                 ServerHello
*                                                Certificate*
*                                          ServerKeyExchange*
*                                         CertificateRequest*
*                              <--------      ServerHelloDone
* Certificate*
* ClientKeyExchange
* CertificateVerify*
* [ChangeCipherSpec]
* Finished                     -------->
*                                          [ChangeCipherSpec]
*                              <--------             Finished
* Application Data             <------->     Application Data

I was also skimming over this article: http://www.fehcom.de/qmail/smtptls.html

It looks like the steps marked with an asterisk are optional according to the protocol. Generally, as I understand it, the client and server are supposed to share public keys, individually and separately compute a shared secret, negotiate a cypher, and then begin communication through that mutually understood cypher using the shared secret as a seed on both sides.

If I was going to do this, I'm assuming I would have to create a slightly jacked up implementation of TLS, let's call it "YesTLS".

YesTLS is stupid, but very outgoing. It will attempt to please everyone that talks to it, even though it has no idea what they are saying. I would configure it with my public keys and any other required data, and then it would respond to TCP connections on certain ports that I am listening on in my application. The idea is that if the remote client doesn't actually ask for a unique digital signature to verify that they are talking to the "keyholder" they think that they are, then my server need not hold the key. It can simply store the encrypted packets as it receives them. Then, my server which actually has they keys can request those recorded connections one by one when it's ready, and re-enact them using the real private keys, thus receiving the encrypted messages.

This way I can limit the storage of plaintext emails.

Would this actually be possible with TLS? I want to get a quick sanity check on this before I start trying to make it happen.

mail.google.com         yes-TLS-smtp-server        trusted-smtp-server

                                 X <---  config with pubkey   <---
                                 ~
---> request TLS connection ---> X
X <---   send public key   <-----
--->     send public key    ---> X
X <---    ok, i'm ready     <---
--->    encrypted stream    ---> X
                                 ~
                                 X <---     get new emails    <---
                                   ---> send google public key --> X
                                   ---> and other session info --> X
                                 X <---     ok, i'm ready     <---
                                   --->    encrypted stream   ---> X
                                                            should be decryptable?

Answer 1

The purpose of the certificate is to prove the identity of the endpoint in the connection and to deter man in the middle attacks. To prove that you own the certificate you need to have the private key, because the certificate itself is public and thus known even to an attacker. Usually the private key is stored on disk or the necessary cryptographic operations are done in a hardware module (HSM, smartcard). But there is also an implementation to do these operations on a remote system where you might have more control, see cloudflare keyless ssl.

Apart from that TLS will in theory provide secure transport but don't expect too much from the way it is currently used between the mail servers. Lots of mail servers ignore certificate errors like self-signed or non-matching hostname. Also in order to make sure that the mail gets delivered to the correct server you would need to secure the MX records too, i.e. the records have to be protected by the still rarely used DNSSec and the delivering MTA has to use DNSSec to get the MX information. Thus all you get from the current use of TLS is a protection against passive sniffing but not against an active man-in-the-middle. This does not mean that you should not try to properly implement TLS on your site but only you should be aware how much security you really get from it.

Answer 2

No. The client will require the server to prove that it holds the private key before it will complete the TLS handshake. Without the private key, you'll never get a connection. Per the Wikipedia

The TLS_DH_anon and TLS_ECDH_anon key agreement methods do not authenticate the server or the user and hence are rarely used because those are vulnerable to Man-in-the-middle attack.

No reasonable client will allow those ciphers.