There are two clients and a server. Each client generates an RSA key pair and uploads public key to the server. The server associates the public key with the user identifier that is specified during upload.
Then client A needs to share the data with client B. It asks the server for the public key of the client B. The fingerprint of the public key is presented to the user for the visual verification. It is the responsibility of the user A to decide if she trusts the public key of the user B and vice versa.
After both users trusted the public keys of each other, they can encrypt data for each other. Client A encrypts data with the public key of the client B, uploads it to the server. Client B downloads the data, and decrypts it.
Clients communicate with the server using TLS implemented by the operating system. The server authenticates clients using OAuth2 with MAC token.
Although the bigger question is, of course, if this system from this high perspective already has some security problems, the question of this post is about the certificates.
In a situation like that, when the higher goal is to provide end-to-end encryption, and one of the decisions is to make the users verify each other’s public keys, is there any benefit of having self-signed certificates in addition to just the public keys?
It is said, for example, that the self-signed certificate guarantees that somebody who generated it also has the corresponding private key. When no certificates are involved, is there a security risk of client A encrypting something with what is thinks is a public key of the client B, but not actually is a public key?