1

I am working on a secure messaging platform and am having trouble figuring out how to protect against MITM attacks.

My current setup can be summed up as follows: The server keeps a private key and makes the public key public which can then be installed into clients. When sending data to the server, it must be ciphered with the public key which will then be decrypted on the server and re-encrypted with the private key before being forwarded to the recipient.

ASCII sketch of normal situation:

Sender --Encrypted, can be intercepted but not deciphered-->Server--Encryped with the private key, if intercepted can be broken since the public key is public--> Recipient

From that information alone we should see that the Sender->Server part is secure and a man in the middle wouldn't do anything (Trying something will corrupt the protocol and the connection will be terminated). However, the Server->Recipient part is not secure against MITM attacks.. at all!

To make matters worse, when data is sent, it is also sent back to the sender to confirm that it was sent, thus compromising both clients to MITM attacks.

So here is my question, upon the initial connection, how can the server prove that it is indeed the real server?

Sample scenarios:

Evil Server must not be able to pretend to be the Real Server Client-->Evil Server-->Real Server

Real Server must be able to prove that he is, in fact, the Real Server. Client-->Real Server

Notes: This is done over TCP. I am looking for a solution that won't require heavy infrastructure but I can allow using a couple of external servers to check (However these too, will need a way of proving their authenticity).

Improve this question
7
  • 2
    ... This sounds a lot like how HTTPS works... Why don't you want to use that or are you talking about installing a certificate on the client during an application installation so that a MITM becomes nigh-impossible because keys never get exchanged?
    – Robert Mennell
    Commented May 13, 2016 at 23:11
  • 1
    Why can't you use HTTPS? Any home-grown solution is going to be inferior.
    – Neil Smithline
    Commented May 14, 2016 at 1:41
  • @NeilSmithline 1. More insight on how those technologies actually work by rebuilding something similar. 2. I can't be bothered to find the perfect library for my project due to how complex the backend is.
    – Slava Knyazev
    Commented May 14, 2016 at 1:48
  • 1
    Are you suggesting that it would be less complex to implement your own network protocol than use SSL? That seems incorrect to me. While I think it is fine to play with these technologies to learn things, using a hand-crafted solution is a risky decision.
    – Neil Smithline
    Commented May 14, 2016 at 1:51
  • 1
    @NeilSmithline Not necessarily less complex but lighter and the implementation of URSA (RSA for node) and the standard crypto library seems far easier than learning to use the https library. Another thing is that https uses certificate authorities which I don't really like the concept of and don't need them while I don't think I would be able to take it out without compromising the security even more. I feel like for this specific case, re-inventing the wheel in more practical. I might move to TLS if I come to see it that way, but for now I'll reinvent the wheel.
    – Slava Knyazev
    Commented May 14, 2016 at 3:19

1 Answer 1

Reset to default
3

Begin each session by having the client generate a symmetric encryption key, encrypt it with the public key of the server, and send it to the server. Both server and client can then encrypt all further communication during the session with that symmetric key.

An impostor server would not have the private key of the server, so it can not decrypt the session-key and thus can not communicate with the client.

By the way, you are currently reinventing TLS with certificate pinning. You should generally avoid inventing your own cryptographic systems. Use what's already there and already tested for vulnerabilities by countless people.

Improve this answer
7
  • "encrypt it with the public key of the server" --- how would you obtain it? That's what OP is concerned about.
    – zerkms
    Commented May 13, 2016 at 23:12
  • "An impostor server would not have the private key of the server" --- how does the client prove authenticity of the real application server public key?
    – zerkms
    Commented May 13, 2016 at 23:14
  • @zerkms According to the question, the clients already have the public key: "The server keeps a private key and makes the public key public which can then be installed into clients." If the clients would not have the public key and need to acquire it from an untrusted source, the solution would be to sign the public key by a trusted certificate authority (which is, again, something TLS has already invented).
    – Philipp
    Commented May 13, 2016 at 23:21
  • Yes! This is exactly what I needed. I was looking more for a way to avoid MITM alltogether but with this method, it simply doesn't matter. Working on the idea a bit, which encryption would be best to use in this case? Ideally something that has libraries for most modern languages.
    – Slava Knyazev
    Commented May 13, 2016 at 23:30
  • @SlavaKnyazev As I said, use TLS with certificate pinning. Any programming language which is widely used should have a free implementation.
    – Philipp
    Commented May 13, 2016 at 23:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .