9

According to this ArsTechnica article, using a self-signed certificate on a mail server is a bad idea:

When we set up a Web server in our previous series, getting SSL/TLS working was a recommended but optional step. However, it is most definitely not optional for this guide: you must have a valid SSL/TLS certificate for your mail server. "Valid" here means "issued by a recognized Certificate Authority (CA)," not something self-signed. Using self-signed certificates to identify your mail server to other mail servers is a great way to get other mail servers to refuse to deliver your messages. Then your mail server would be lonely because none of the other servers will want to talk to it, and that would be sad. You don't want your server to be sad, do you?

Is this actually true? Will other mail servers really refuse to interact with a mail server that uses a self-signed SSL certificate? If so, which ones? Or is this true of almost all common mail servers?

9

You're fine with a self-signed certificate.

In my experience over several years, I never had any problems with a self-signed certificate on my mail server. I eventually switched to one signed by StartSSL and there was no discernible difference in interoperability.

See also this post on ServerFault.

The bottom line is, STARTTLS is about encryption, not authentication. While it can be locked down to perform authentication, the default of most (if not all) SMTP servers is opportunistic encryption. SMTP is plaintext by default, and needed encryption more than it needed authentication, thus the emphasis.

It is possible to lock down connections - for example, I've seen an Ironport configured to require a valid certificate signed by a known CA it liked (in this case, it liked Entrust but not Comodo, so we ended up changing the cert because it was easier than debugging somebody else's Ironport). But that's for locking down specific domains which want to ensure partnered encryption, I've never seen someone doing that with opportunistic encryption.

  • 1
    To have secure encryption using the symmetric cryptography used by TLS you either have to use pre-shared keys or a way to identify the server before sharing newly created keys, otherwise you might share the keys with an attacker instead. Self-signed certificate does not provide such identification and thus the connection is open to active man-in-the-middle attacks. – Steffen Ullrich Sep 7 '14 at 0:48
4

Encryption with SSL consists of the following steps:

  1. Identify the peer using a certificate.
  2. Create and exchange keys for the following communication.
  3. Exchange data, encrypted with the previously exchanged keys.

You are usually employing encryption because you think that anybody might listen to the connection. A small step from this passive listening is active manipulation of the connection which SSL can detect too. But for this it needs to properly identify your peer, otherwise anybody in the middle could claim to be the right server.

With a self-signed certificate proper identification is not possible, because anybody could create such a certificate with the same credentials as you did. This means that nobody should trust a self-signed certificate by itself. It should only trust a self-signed certificate if it could verify it using another way. This other way could be a manual configuration at the server side (usually not the case unless the sender knows the recipient directly) or the verification of the certificate through DANE, where the owner of the domain provides their certificate or fingerprint inside a trusted DNS record (needs DNSSec).

What servers actually do when they detect an untrusted certificate differs: some will accept the certificate and write verify=NO in the Received header of the mail while others might refuse to connect with SSL because they consider the connection insecure and downgrade to plain text or not connect at all. It depends on the settings of the sending server.

Currently probably most mail servers will successfully connect to other servers, even if they can't verify their certificate. But, more and more are using "real" (verifiable) certificates. If you compare the reports from Facebook from May 2014 to August 2014 you can see the use of verifiable certificates move from 30% up to 95% in just 4 month. This means, that the use of real certificates will probably required by more servers in the near future.

This means, that using a certificate signed by a well known CA is definitely better than using a self-signed certificate. But even that might not provide the security you are trying to achieve. Any of the 100th of CA trusted by the operating system can issue such a certificate, so an attacker might try to get a certificate for your domain too. But this is at least much harder then just create another self-signed certificate, because the attacker needs to either hack the CA or control your mail domain somehow to capture mails the CA use inside the identification process.

Another attack of an active man-in-the-middle man is to re-route the mail by attacking DNS: To route a mail from the sending mail server to the server responsible for your domain, the sending server needs to to a DNS lookup for the MX record of your domain, which contains the names of your mail servers. An active attacker near the sending server could attack this DNS lookup and provide a fake MX record which points to its own servers. TLS will not help here anymore. Instead you have to use DNSSec to secure your DNS records and the sender has to verify this record.

Apart from that TLS does not provide end-to-end security for mail, just hop-by-hop. Any mail server in between has access to the plain mail and thus needs to be fully trusted. To get end-to-end encryption you need to use S/MIME or PGP. And because of this some server admins don't care about proper TLS at all, because TLS will not offer real (end-to-end) security for the mails, even if properly implemented.

0

A certificate's authenticity is verified using the trusted party's public key. A certificate spoofer can enter any data they want in any cert they make but can not sign the cert with the trusted party's private (signing) key as they do not have it. When the trusted party's pub or if you like verification key is used on the wrong private (signing) key the signature will be seen as forged.

This so far is no different for either a CA signed or self signed certificate.

Issues arise however in the distribution of the public key. This is where certificate 'authorities' excel as they can, seemingly, more securely distribute their public keys to a larger number of people simply by securely sending it to a relatively small number of key application providers willing to install it in their app, eg: 'browser vendors', mail server providers etc.. that know for sure that it's them sending it. Thus their public keys, used to verify their own signatures, are 'for sure' theirs and can securely be used to check if they in fact signed something. Namely a certificate.

If a self signer can securely distribute their public key, that is, distribute it in a manner that it's users know for sure it's theirs then their signature, and thus their self signed certificate, is as good if not better, considering their idenity is better assured through the key distribution than CA certificate requestor id verification, as that of any CA.

In this light the question becomes how securely and cost (or time) effectively can your pub key be distributed to and installed on the other mail servers and if whatever's driving you to do so, be it better security, CA $upport aversion or some other is worth it.

0

This is not true anymore. Some email servers started to verify the chain of trust of offered certificate and refuse the connection if it does not pass. This can prevent sending and receiving emails.

In our case, we are using mailgun and there is a small switch in their configuration interface which by default requires the receiving server to have a proper certificate.

  • A lot has changed in the realm of crypto in the previous five years. You have Let's encrypt which offers free certificates for anyone - five years ago certificates carried a very real cost in many cases. – vidarlo Oct 19 '17 at 18:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.