How to roll my own security mechanism - avoid SSL

Question

I'm looking at implementing my own security mechanism rather than using SSL due to SSL being 'hopelessly broken' and also because it is an interesting exercise :)
This is for a client-server internet application.

Caveat - I am not a security expert - just been reading up on security.
I will also be building the client application therefore I will have control over the implementation both sides.

SSL Problem

The main problem I see with SSL is due to the swapping of public keys at the handshake stage, and a man in the middle intercepting this. SSL tries to avoid this problem using message signing and certificates etc to validate the server's public key to the client. But an attacker with a forged certificate can still get around this.

Workaround

Due to this problem of safe public key exchange, I'm thinking this can be avoided by simply not having to swap the server public key, by having 'known' server public key built into every client application. Say a 3072 bit RSA key.

Implementation

1. The implementation for each communication session would thus be client generates a public / private RSA key pair
2. client encrypts its new public key with server's public key and sends it to server
3. server decrypts message to discover client's public key and then uses this to encrypt any responses to the client
4. client continues to send traffic to server encrypted with server's public key
5. each message sent from the client also contains in the header:
   1. the client's username encrypted with the server's public key
   2. a hash of the un-encrypted client message - using as input to generate this hash, another hash of [client username + realm + client password]

Observations

• Server can prove identity of client by first decrypting message using its private key, and then calculating hash of the unencrypted message using the hash of [client username + realm + password] which it has stored in its database.
• Man in the middle cannot monitor traffic as it is encrypted both ways
• Man in the middle cannot pose as client because he cannot generate a valid message hash
• Man in the middle cannot pose as server because since he cannot decrypt initial message, he cannot discover client's public key in which to encrypt responses to the client
• The client's password itself is protected against brute force scrutiny as the hash which used the clients password as input, is against the un-encrypted message, so attacker would first have to decrypt the message before beginning a brute force attack against this hash

Seeing as I'm not a security expert I am hoping if someone who know's more about security can spot any holes in this approach, or confirm if it is good. And also if a 3072 bit RSA key can stand the test of time without being broken.

Thanks!

Answer 1

You're confusing SSL/TLS with its most common usage pattern, which is TLS in conjunction with X.509 certificates and a Public Key Infrastructure (PKI, RFC 5280).

While it's really important to secure a TLS connection by authenticating its server site (to prevent active MITM attacks), the TLS specification doesn't actually mandate to do it using X.509 certificates, even less using a PKI.

As @jathanism said, you can choose not to rely on a hierarchical PKI, by using a self-signed X.509 certificate. You will have to establish trust in the server certificate by some other means.

In addition, TLS can also be used with OpenPGP certificates, Kerberos or Pre-Shared Keys.

TLS itself has had very few known bugs as far as I know, e.g. there was the renegotiation bug (CVE-2009-3555), which has been fixed (more recently, there were also certain issues with certain block ciphers, which can be fixed by choosing better cipher suites, or upgrading to newer versions of TLS).

The issues you're talking about (and often discussed in the press) tend to be related to PKIs, i.e. how to establish trust in the remote party. How this should be done isn't dictated by the TLS specification itself.

If you want to try to find a better way, by all means do some research, but I'd suggest focussing on this part and leaving SSL/TLS and the cryptography side of things alone. You'll probably find that the biggest problems regarding PKIs and their alternatives are administrative, not so much technical ones.

The other aspect of authenticating the remote party is to check you're talking to the intended party (i.e. host name verification).

Answer 2

If you have control of both sides, use self-signed SSL certificates. Done. You're not trusting a root Certificate Authority, and therefore your crypto becomes almost impossible to crack without internal knowledge of the architecture.

Answer 3

I have decided to go with SSL as in the end it ended up being the simplest approach - the other idea just wasn't working well on the WCF side.

I have written this up on code-project to help others out having a similar problem.

http://www.codeproject.com/KB/WCF/WCFJsonRestHttpSecureRole.aspx

I'd still need to figure out how to trust only my own public key and no CA keys - looking at the as3cryptolib it might be possible with the CAStore property on the TLSConfig - http://code.google.com/p/as3crypto/source/browse/trunk/as3crypto/src/com/hurlant/crypto/tls/TLSConfig.as

Answer 4

It's not SSL the mechanism that is "hopelessly broken" but rather SSL v3.0 (and SSL v2.0) the specification versions.

After SSL v3.0, the specification was renamed to TLS. TLS 1.0 is still considered secure. I would recommend you use TLS 1.2.

And, yeah, as others have said, don't even think about rolling your own. Trying to roll their own by somebody who didn't know enough cryptography is why early WiFi encryption could be cracked in a matter of seconds or minutes. They started with a weak algorithm, and then through implementation problems further weakened it.