I'm trying to work out a somewhat anonymous secure messaging system, and was hoping someone could validate my overall approach. Keep in mind, this all takes place over HTTPS.
While the transport supports client certificate authentication at the protocol level, we don't want to have to issue certificates, and we don't want to have sign them. The mod_ssl/Apache options are none, optional, require, and optional_no_ca - unfortunately there's nothing like require_no_ca.
- Client generates public/private key pair - either 4096 bit RSA keys, or an EC key (using either brainpoolP512r1 or NIST P-521 curves).
- The client requests a challenge from the server, sending it it's public key.
- The server generates a challenge - a largish amount of random data (say a byte 8x the block size of the cipher). The server encrypts the response with the client's public key.
- The client takes the response, decrypts it, generates a SHA-512 hash, and passes it back to the server.
- The server verifies that the hash matches, and if so, the client is authenticated.
We don't need logins or passwords, just client certificates. If access control to the server is desired, client certs could be whitelisted, or more traditional username/password authentication could be used.
- Clients exchange public keys. This has all the old problems of PKI - revocation, etc. that I really don't want to get into. So this happens manually amongst clients out of band.
- Client A sends the server a handshake request with an ECDH public key and the public key of the sender which is encrypted with the public key of Client B. The server doesn't have an record of the sender.
- Client B connects to the server, and requests any messages. The server sends the message to Client B. The client decrypts it, and checks the sender against the list of public keys it's already. If it's got the key, it generates an ECDH key pair, and sends a response to the server with it the ECDH keypair's public key, which is encrypted with Client A's public key.
- Client A gets the handshake response, and from their Client A&B can start communicating using their shared secret, through the server. The shared secret will be used to encrypt the message(s) via AES-256 (GCM, no padding)
- Further messages will generate new (and thus ephemeral) ECDH key pairs, piggybacked on top of previous messages, so the AES key for the client<->client communications have perfect forward secrecy.
The client content is all encrypted so that the server cannot see messages. It basically acts as a store and forward transport for the client messages. The cert based authentication allows us to ship messages to individual clients on request.
I'd love any and all feedback.