A secure way to encrypt a connection between 2 clients securing against both passive and active adversaries

Question

Recently I got interested in encryption and the methods behind them. So for a fun / school project I am building a program in python to encrypt data between two clients (chat and files).

I'm just really not sure how to truly make it secure against both passive and active attacks. Right now my idea is to use a server to establish a handshake between the clients in the following manner:

1. Once a client starts, it connects to server to let it know its alive.
2. Client bob lets the server know it wants to connect to client Alice
3. Server sends message to client Alice that bob wants to set up a secure connection
4. Alice tells the server she accepts the connection
5. The server generates a random number and encrypts it using the client public key The client has private RSA key hard coded (doesn't change per connection / client)
6. Server sends the random number to both Alice and Bob
7. Alice and bob decrypt the random number and hash it, both send it back to the server
8. Server checks if both hashes are the same (none are tempered with)
9. Server sends an OK or error to both parties
10. Alice and bob hash the number a number of times equal to the random number (so if the server sends 1234 they hash it 1234 times.
11. Alice and Bob setup a common key for this connection using Diffie-Hellman key exchange and encrypt the exchange with AES.
12. Once a common key is decided, use this key or a hash of this key for AES encryption (this key does change per connection and all the data is sent over this channel)

The only problem I see with this protocol if adversaries can reverse engineer my code to get the private key, and can use active attacks at both Alice and Bob's connection OR if the server is ever compromised.

I will use py2exe to compile the code, that way it should be hard to get the key but I'm not sure just how hard it would be.

Sorry for the long question but it just feels like a really horrible idea to hard code the key into the client but I'm not sure where to go from here.

Thoughts on this idea / improvements / How can I do it better?

Sorry for any typos, English isn't my first language.

Answer 1

You can't have a private key hard coded in software that is distributed.

The two options I see are:

1. Have each client generate a unique key-pair that they use for communicating with the server. Bob would send his public key to the server when he initially connects. The server will then encrypt messages to Bob with his public key. The first communication between the server and Alice will be the server asking Alice for her public key so that it can be used for future communications. The server can then pass a symmetric key encrypted with their public keys to each of Bob and Alice to use for future communications.
2. Give the server a key-pair. When Bob first connects with the server, Bob creates a symmetric key that he encrypts with the server's public key. The server and Bob use this for encrypting further communications. When the server first contacts Alice it passes her its public key. Alice responds by creating a symmetric key and sending it to the server, encrypted with the server's public key. The server can then communicate with Alice using this key.

Option 1 above uses what I'll call the SSH model where clients generate unique key-pairs for each server. Option 2 uses what I'll call the SSL model where servers have the key-pair (though SSL communicates the public key in a signed cert).

If you do the above you'll have very good security. That said, it won't be perfect. The problem is that you'll still be vulnerable to MiTM attacks.

In order to work safely, SSH requires you to validate the fingerprint of the server's public key the first time you connect with it. The protocol doesn't explain how you validate this. Most users just say say OK without validating, but there's a chance for a MITM there.

SSL automatically validates the public key from the server by validating the cert chain. This only works because there's a keystore with trusted root certs pre-installed with your OS, browser, or other client. Without that validation, the public key in the server's cert can be forged without detection.

I don't think that you're going to be able to do much better unless you use an existing security infrastructure. For example, you can have your server run SSL and have the client's validate the certificate passed in the SSL handshake.

I don't see this as a problem as you're going to be as secure as SSH is when the client doesn't properly validate the server's public key. IMO, that's pretty good.

This is a very cool problem. Enjoy!

PS: No need to apologize about your English. It's as good as a native speaker's.

Answer 2

Depending on where you're at, I'd say this is a great high school project. The ipsec + dnssec + pki + tls/ssl route is very much a protocol based facet of this exercise. Maybe you'd like to bring up the physical security aspect of things as well. For example you bring up the location of the endpoints as a mitigating factor. After all, fips level skiffs for equipment aren't there for nothing.

Answer 3

Fundamental to the question, is if the endpoint is not secured, keys, whatever they may be can be obtained, which is why for endpoint secure transactions, the terminal itself is under the aegis of the proprietor - eg. atms, cash points, etc.

Answer 4

Nice project but as usual you really should not reinvent the wheel. Writing such protocols for production use is a bad idea in almost every case. I recommend reading Bruce Schneier's Cryptography Engineering book. It is not a light read but would definitely clear out all your questions. First of all I think burning keys into executables is a terrible idea. It should be a key that protects your sensitive data not the method (see Kerckhoff's principle). Also makes key management a PITA. Why do you involve the server in the key negotiation, is it a requirement? If I am not mistaken, you use that "1234" key for peer authentication. You could simply have it signed by the clients and verified at server side using client keys. You wrote "Once a common key is decided, use this key or a hash of this key". Again, the key strength should provide security, not the method. Methods can be reproduced, keys cannot.

All in all, I think you should use a well known protocol rather than inventing your own. If it is only a project to learn crypto protocols, firts I recommend looking into other working ones.

Answer 5

The answer (at least for my work) is IPsec host to site or host to host with PKI authentication and protection of the private keys in hardware.