Let's say I want to create a protocol to securely communicate between two endpoints - each of the two endpoints know the other's public key and all data exchanged is encryted using one's public key and decrypted using the recipient's private key. My question is aside from man in the middle attack which can be avoid by using a PKI(?), what other kinds of attack is this protocol vulnerable to? And how is it compared to SSL as we have removed the initial handshake step?

  • It will be much much slower compared to SSL.
    – Shurmajee
    Commented May 12, 2013 at 19:12

3 Answers 3


This protocol is vulnerable to a replay attack. In common notation

A -> I_B: {m}_PK(B)
I_A -> B: {m}_PK(B)
I_A -> B: {m}_PK(B)

If m was a message saying loan me $10 and B was a little naive ...

This protocol is also vulnerable to message reordering attacks e.g.

A -> I_B: {m}_PK(B)
A -> I_B: {m'}_PK(B)
I_A -> B: {m'}_PK(B)
I_A -> B: {m}_PK(B)

If we assume RSA, then if there is no padding then such a protocol would be weak to homomorphic encryption attacks.

Various other attacks are obvious but uninteresting (such as denial-of-service by the intruder blocking all messages, dictionary attacks letting the attacker verify a guess at the message, dictionary attacks letting the attacker detect repeat messages, message truncation attacks etc.)

  • Thanks, but then how is it different with SSL?
    – Ryan Saka
    Commented May 8, 2013 at 22:16
  • 2
    SSL uses public key cryptography to exchange a secret session key and uses random numbers from both sides to prevent a replay attack. All further communication is encrypted with the session key. The communications under the session key are encrypted with a block cipher in some useful mode (usually GCM or CBC). This means that the decryption of each message depends on the previous message (or in counter mode the number of previous messages). Thus replay attacks give gibberish and dictionary attacks lack the proper key.
    – jhoyla
    Commented May 8, 2013 at 22:36
  • let's say I decide to implement the same security on per-message basis in my protocol (with the use of timestamp, sequence number etc..), but instead of using a shared secret session key, I opt to use the recipient's public key to encrypt the message - by this I eliminate the need to do initial handshaking, if so then is my protocol as secure as SSL?
    – Ryan Saka
    Commented May 8, 2013 at 23:03
  • 1
    I can't see an obvious attack, but that doesn't mean there isn't one. The main reason people don't do this is public key cryptography is horribly slow. However it definitely won't be as secure as SSL, because SSL has been studied intensively and has fixed implementations that have been rigorously bug-checked. There are hundreds of subtle ways to leak information, timing attacks, power attacks (both active and passive), etc.
    – jhoyla
    Commented May 9, 2013 at 1:45
  • I see, thanks. I know that it definitely has flaws, I'm just curious what are they might be though. The only one I could think of was that if the private keys are exposed then all the previous communication between one endpoint with all the other would be cracked, SSL avoids this problem by using a unique key for every session.
    – Ryan Saka
    Commented May 9, 2013 at 12:08

A PKI is a system to allow safe distribution of public key. The certificates issued by the PKI allow the various systems to know each other's public keys with some strong level of guarantee that the keys are genuine. In your case, you already assume that each of your two endpoints knows the public key of the other endpoint, i.e. that the problem that which PKI strives to address has already been solved. No need for a PKI in your case, then.

This mutual knowledge of public keys can then be escalated into a two-way protected channel for data, but that's not an easy thing to do; you cannot evacuate it with a simple "we encrypt the data with the other system's public key"; as others have pointed out, this will suffer from replay attacks (at least). There are many details to worry about. A protocol where all these details have been painfully dealt with, along a course of two decades of breakage and fixes, is, indeed, SSL. So far, most people who thought they could do better have failed, sometimes hilariously, often spectacularly. SSL includes all kinds of features which keep track of individual data chunks ("records" in SSL parlance) and reliably detect external alterations, including reordered, dropped or duplicated packets; it also ensures verified termination (when the connection is closed by one machine, the other machine has some guarantee that this was indeed triggered by the other machine, not by an attacker inserting a fake RST packet).

Note that in SSL, the client and server send their public keys to each other as part of certificate chains. This supports the PKI model in which a client (respectively a server) learns a server public key (respectively a client public key) through certificate validation. However, nothing forces client and server to process the certificates in a pure X.509 PKI way. What matters is that the client (resp. the server) uses the correct public key of the server (resp. the client). If the client already knows the server key, then the client can simply use it in SSL and disregard whatever blob the server sent as certificate. Thus, SSL is not intrinsically tied to X.509 or to any other type of PKI.


In addition to the replay and ordering issues identified by jhoyla, there is no entity or message origin authentication in this protocol.

Any party with the public key of either peer can send arbitrary messages to the peer, and the peer has no way of authenticating their origin. Indeed, they could even fake a conversation between two peers, especially if they can block outbound messages from the participants or the details of the sequence number scheme you implemented allowed the attacker's messages to 'get in first'.

As it's a public key, you can fairly safely assume your attacker has access to it.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .