So I was thinking about client-server communication and how to keep this private, especially from man-in-the-middle attacks. I came up with a scheme that I'm sure exists already, but I could not find sufficiently concrete search results to learn more about it.
It started by my thinking that when a client connects to the server, they would like to establish a secure communication channel, so the client could generate a private/public key pair and send the public key to the server. The problem is getting the public key to the server securely (otherwise a man in the middle could just decrypt all messages using the public key it grabbed from the first message, attach its own public key to the message, use the corresponding private key to decrypt the response and, after storing that, re-encrypt with the client's actual public key).
So I thought of a possible solution: the client first receives a public key from the server, that it will use to encrypt its own public key. The MitM can intercept all messages, but won't be able to decrypt any of them - only the actual server can decrypt the client's request and the response can only be decrypted by the actual client. Once the server knows the public key of the client they securely store that and use it to encrypt any further communication.
I have a few related questions.
- What is this type of security / handshaking called?
- I have heard the word nonce but am I right that this is about preventing replay attacks and not the MitM problem I'm trying to solve?
- The initial key that the server sends to the client, should that be fixed and possibly published somewhere, so clients can skip the "give me your public key step" on subsequent sessions? Or should it be a freshly generated key for each client at each new session request?
- Conversely, should the client's public key change with every session or can client re-use the same key once it's been securely communicated to the server?
- What are the risks to this, other than the obvious failure by one of the involved parties to keep private keys private?
I am sure that this is fairly common and probably already how a lot of authentication schemes work, so rather than answering all these questions separately, I would be happy if someone could provide me with the proper terminology (I tried some combinations of keywords like "client/server", "asymmetric encryption", "establishing session"; but they are too general to find anything useful). And even though this is a theoretical SE, I guess that like with all good security one should not roll their own implementation, so any example code of a proper client/server application (e.g. in PHP) using well-established libraries would be welcome.