How to 'pair' a client/server application securely?

Question

Please forgive me if my question is a rephrasing of a common problem, but I am not sure what to search for!

My use-case is a client-server program. The server runs on a user's PC and the client runs on their mobile device(s). The clients need to be paired to the server in a secure way (e.g. enter a PIN or password once-off), and once the pairing is completed then we need to ensure that the client can securely verify the server, before transmitting confidential data from time to time.

Earlier in my research I thought it would be possible to use RSA keys, but this approach seemed too novel and complex to implement securely.

My current idea is to implement a scheme like this:

1. At initial pairing, Diffie-Hellman key exchange between client and server
2. Server displays a random PIN, and client enters it for verification (preventing mitm). Only a couple of wrong PIN attempts are permitted before the pairing attempt is dropped.
3. Since the server and client now agree on a secret key, we can simply encrypt all communication with this key. Although the server may have multiple known clients (say up to ten clients), the server could simply check any incoming connection with all ten of these keys. And since the client uses the same secret key, if the server is an impostor, decryption will not be possible.

Now my question, is this a reasonable scheme or is there a clear standard protocol that can achieve similar aims?

Answer 1

Reinventing the wheel is a bad idea in software development in general, but especially bad when security is concerned. Before (or instead of) designing your own authentication protocol, you should look at existing ones like TLS or Kerberos, understand how they mitigate possible attacks and perhaps even chose an existing protocol to use in your application. You will avoid many common pitfalls, save on development time and your app will have better compatibility, should you want to have it. TLS specifically seems like a good fit to your use case.

By the way, could you explain how displaying a random pin would prevent MitM attacks? Wouldn't the attacker see the very same pin as well?

Answer 2

It can be secure, but the devil is in the details. There is a number of things which could be done incorrectly leaving the protocol wide open to attacks.

1. Too small Diffie-Hellman prime. If you do hardcode a prime in your protocol, then the prime needs to be larger than it would be if generated dynamically.
2. PIN verification prone to mitm-attacks. An active mitm-attack against Diffie-Hellman is trivial. That's why you need the PIN in the first place. You have to remember though, that the purpose of the PIN is not to verify the PIN but rather to verify the key. If for example you just send the PIN as is through the encrypted channel established using Diffie-Hellman, an adversary can simply mitm the connection and forward the PIN received from one side to the other side.
3. Using a single key for too much data. The more data is being send encrypted with the same key, the easier it will be for an adversary to attack said key. Periodic re-keying would be a good idea. Don't do re-keying in the clear. Do the re-keying inside the channel protected by the previous key, but at the same time design the re-keying protocol such that it would plausibly have been secure if it had not been inside the protected channel. That is defense in depth.
4. If you encrypt data in transit without also protect the integrity using some sort of message authentication code, you will be vulnerable to a wide range of attacks.

The above is by no means an exhaustive list of possible weaknesses. If you can find an existing well-known protocol, which satisfies your needs, I would recommend using it.

The only thing that comes to my mind from your description is the Bluetooth pairing protocol. Unfortunately I don't know enough about the Bluetooth protocol to recommend for or against its use.