I'm having a hard time finding the practical benefits of transmitting a self-signed certificate over simply a public key. I understand that a self-signed certificate proves the integrity of the public key and user ID (i.e. proof the signer is able to decrypt messages encrypted by the public key), but what is the practical security benefit of this knowledge?
From what I understand, both sending a self-signed certificate and sending simply the public key are susceptible to man-in-the-middle attacks:
- Alice and Bob want to communicate privately.
- Mallory is between Alice and Bob.
- Mallory intercepts the public keys or self-signed certificates.
- Mallory creates a new keypair, and self-signs in the case of self-signed certificates.
- Mallory relays the new public key or self-signed certificates to Alice and Bob respectively.
- Mallory is now transparently between Alice and Bob.
- Alice and Bob mistakenly believe their communicate is private.
Since Mallory can just generate new keypairs and self-sign, a man-in-the-middle attack can be executed quite as straight forward as if solely public keys were exchanged.
I've also considered a denial-of-service attack, where Mallory can corrupt the user ID or public key so that Alice/Bob can not decrypt messages and thus not communicate. A self-signed certificate would allow Alice/Bob to tell that there is a corruption as soon as the certificate arrives, whereas a public key by itself would only be discovered as corrupted once the response is received.
However, Mallory gains no benefit in performing this convoluted denial-of-service: Alice and Bob are not revealing secrets as they are not able to communicate, and simply blocking network packets is a far simpler way of denying service to Alice and Bob.
So in what practical ways is transmitting a self-signed certificate more secure than transmitting a public key by itself?