Public keys without the certificates

Question

There are two clients and a server. Each client generates an RSA key pair and uploads public key to the server. The server associates the public key with the user identifier that is specified during upload.

Then client A needs to share the data with client B. It asks the server for the public key of the client B. The fingerprint of the public key is presented to the user for the visual verification. It is the responsibility of the user A to decide if she trusts the public key of the user B and vice versa.

After both users trusted the public keys of each other, they can encrypt data for each other. Client A encrypts data with the public key of the client B, uploads it to the server. Client B downloads the data, and decrypts it.

Clients communicate with the server using TLS implemented by the operating system. The server authenticates clients using OAuth2 with MAC token.

Although the bigger question is, of course, if this system from this high perspective already has some security problems, the question of this post is about the certificates.

In a situation like that, when the higher goal is to provide end-to-end encryption, and one of the decisions is to make the users verify each other’s public keys, is there any benefit of having self-signed certificates in addition to just the public keys?

It is said, for example, that the self-signed certificate guarantees that somebody who generated it also has the corresponding private key. When no certificates are involved, is there a security risk of client A encrypting something with what is thinks is a public key of the client B, but not actually is a public key?

Answer 1

In general, self-signed certificates offer no security benefit over raw public keys (there can be convoluted situations where the self-signature provides a "proof of possession" of the corresponding private key, but this very rarely matters). Self-signed certificates can offer a usability benefit, though, in that it allows the use of certificate-based software.

For instance, in Windows systems, private keys are commonly referenced through the corresponding certificate: the user has some certificates in a "certificate store" and each such certificate may reference the corresponding private key. If you want to use the CMS format for signatures, with the .NET class System.Security.Cryptography.Pkcs.SignedCms, then you must sign with "a certificate", so something which looks like a genuine X.509 certificate must be used.

This mostly amounts to reusing the certificate encoding format (X.509) for public key storage. Conveniently, such a format includes human-readable names and validity dates; OS-provided GUI can also be used to visualize the certificate "thumbprint", i.e. a hash of the certificate (which can be used as a kind of hash of the public key). Less conveniently, the X.509 format includes a non-optional field for a signature, so there must be a sequence of bytes which at least superficially looks like a signature: the habit of using self-signatures comes from that. However, you could also store a sequence of random bytes of approximately the right size, it would also work.

Answer 2

Why are you trying to verify the public key after you obtained it from your trusted server over a TLS connection?

You are mixing things. Certificates are used to bind a human-readable name/organization to their corresponding public key. Usually a publicly trusted entity (trusted by all potential clients) “vouches” that Mr. Smith has public key XYZ and that’s it.

In your case, once you asked your server (i.e your source of truth) for the public key and once you get it, you need to trust it and use it to encrypt. You can even securely store it locally or cache it. Remember this is a “public” key, so it is ok to be public.

Self-signed certificate from a trusted server would tell the potential clients that the organization running the server vouches to any potential client that Mr Smith has public key XYZ. So those who really trust the organization can trust their assertions and vouching.

From a programmatic sense (i.e presentation OSI layer and below), you can achieve your goal without dealing with certificates, unless if you must do that for better human-readability on the application layer.

Keep in mind: Running an enterprise-level public key encryption scheme is not just about encryption. You will need a PKI and cryptographic key mechanisms to rotate, suspend, revoke, etc keys and certificates (public or private - never self-signed) might be required.