So I was thinking about client-server communication and how to keep this private, especially from man-in-the-middle attacks. I came up with a scheme that I'm sure exists already, but I could not find sufficiently concrete search results to learn more about it.

It started by my thinking that when a client connects to the server, they would like to establish a secure communication channel, so the client could generate a private/public key pair and send the public key to the server. The problem is getting the public key to the server securely (otherwise a man in the middle could just decrypt all messages using the public key it grabbed from the first message, attach its own public key to the message, use the corresponding private key to decrypt the response and, after storing that, re-encrypt with the client's actual public key).

So I thought of a possible solution: the client first receives a public key from the server, that it will use to encrypt its own public key. The MitM can intercept all messages, but won't be able to decrypt any of them - only the actual server can decrypt the client's request and the response can only be decrypted by the actual client. Once the server knows the public key of the client they securely store that and use it to encrypt any further communication.

I have a few related questions.

  1. What is this type of security / handshaking called?
  2. I have heard the word nonce but am I right that this is about preventing replay attacks and not the MitM problem I'm trying to solve?
  3. The initial key that the server sends to the client, should that be fixed and possibly published somewhere, so clients can skip the "give me your public key step" on subsequent sessions? Or should it be a freshly generated key for each client at each new session request?
  4. Conversely, should the client's public key change with every session or can client re-use the same key once it's been securely communicated to the server?
  5. What are the risks to this, other than the obvious failure by one of the involved parties to keep private keys private?

I am sure that this is fairly common and probably already how a lot of authentication schemes work, so rather than answering all these questions separately, I would be happy if someone could provide me with the proper terminology (I tried some combinations of keywords like "client/server", "asymmetric encryption", "establishing session"; but they are too general to find anything useful). And even though this is a theoretical SE, I guess that like with all good security one should not roll their own implementation, so any example code of a proper client/server application (e.g. in PHP) using well-established libraries would be welcome.

  • 1
    the client first receives a public key from the server, Why wouldn't it be possible for a MITM to intercept that initial public key and replace it?
    – Arminius
    May 11, 2018 at 23:32
  • You don't decrypt with the public key. You encrypt with the public key and decrypt with the private key. The danger of the public key being intercepted is that someone can impersonate the person sending data to you, not that they can decrypt the data coming from the legitimate client or the server. May 11, 2018 at 23:42
  • As Arminius said, your second scenario suffers from the same MitM vuln that you correctly identified in the first. This is why we have CAs; the client won't trust the certificate unless it is correct and signed by a trusted CA. A MitM attacker usually cannot obtain such a valid certificate. May 12, 2018 at 4:51

1 Answer 1

  1. The specific phrase you're looking for is "key exchange". https://en.wikipedia.org/wiki/Key_exchange

There are a few schemes(many deprecated) for accomplishing this securely, but if you're writing your own program(and thus aren't as worried about "who is this person actually", and just about securely exchanging the keys), Diffie-Hellman is a good place to start at. https://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exchange

Diffie-Hellman is specifically designed to make exchanging keys secure even if observed. If you are wanting to make key exchange secure against MitM attacks, that is where having certs signed by a trusted CA comes in. A trusted CA will not(or is not supposed to) sign the same identity for multiple parties, so an attacker in the middle intercepting the legitimate public key before it is given to the intended converser will not be able to sub in their own public key, since it won't be properly digitally signed.

  1. Nonces are indeed there in order to make sure each message is only being replied to once, and usually within a set time. They don't prevent an attacker from intercepting them, forwarding them on, intercepting the reply, forwarding that, etc.

  2. Forward Secrecy is considered the best practice now. https://en.wikipedia.org/wiki/Forward_secrecy With systems implementing forward secrecy, new keys are generated each session, so that previous session information cannot be compromised using a mitm interception of the new keys. Note: you're not generating new long-term private keys each session, you're generating new and random 'secret keys' (i.e. the key used only during that session to encrypt traffic).

  3. The user-specific private keys which are used to generate per-session secure keys are kept, but those keys are not the keys being used to directly encrypt and decrypt traffic.

  4. Apart from private key compromise, nothing can prevent a mitm from tampering with traffic they can touch. Proper CA signatures can prevent the certificate chain from being valid if they do tamper with something, but that won't actually protect the data; just its trustworthiness.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .