I'm looking at implementing my own security mechanism rather than using SSL due to SSL being 'hopelessly broken' and also because it is an interesting exercise :)
This is for a client-server internet application.
Caveat - I am not a security expert - just been reading up on security.
I will also be building the client application therefore I will have control over the implementation both sides.
The main problem I see with SSL is due to the swapping of public keys at the handshake stage, and a man in the middle intercepting this. SSL tries to avoid this problem using message signing and certificates etc to validate the server's public key to the client. But an attacker with a forged certificate can still get around this.
Due to this problem of safe public key exchange, I'm thinking this can be avoided by simply not having to swap the server public key, by having 'known' server public key built into every client application. Say a 3072 bit RSA key.
- The implementation for each communication session would thus be client generates a public / private RSA key pair
- client encrypts its new public key with server's public key and sends it to server
- server decrypts message to discover client's public key and then uses this to encrypt any responses to the client
- client continues to send traffic to server encrypted with server's public key
- each message sent from the client also contains in the header:
- the client's username encrypted with the server's public key
- a hash of the un-encrypted client message - using as input to generate this hash, another hash of [client username + realm + client password]
- Server can prove identity of client by first decrypting message using its private key, and then calculating hash of the unencrypted message using the hash of [client username + realm + password] which it has stored in its database.
- Man in the middle cannot monitor traffic as it is encrypted both ways
- Man in the middle cannot pose as client because he cannot generate a valid message hash
- Man in the middle cannot pose as server because since he cannot decrypt initial message, he cannot discover client's public key in which to encrypt responses to the client
- The client's password itself is protected against brute force scrutiny as the hash which used the clients password as input, is against the un-encrypted message, so attacker would first have to decrypt the message before beginning a brute force attack against this hash
Seeing as I'm not a security expert I am hoping if someone who know's more about security can spot any holes in this approach, or confirm if it is good. And also if a 3072 bit RSA key can stand the test of time without being broken.