Let's say I want to create a protocol to securely communicate between two endpoints - each of the two endpoints know the other's public key and all data exchanged is encryted using one's public key and decrypted using the recipient's private key. My question is aside from man in the middle attack which can be avoid by using a PKI(?), what other kinds of attack is this protocol vulnerable to? And how is it compared to SSL as we have removed the initial handshake step?
This protocol is vulnerable to a replay attack. In common notation
This protocol is also vulnerable to message reordering attacks e.g.
If we assume RSA, then if there is no padding then such a protocol would be weak to homomorphic encryption attacks.
Various other attacks are obvious but uninteresting (such as denial-of-service by the intruder blocking all messages, dictionary attacks letting the attacker verify a guess at the message, dictionary attacks letting the attacker detect repeat messages, message truncation attacks etc.)
A PKI is a system to allow safe distribution of public key. The certificates issued by the PKI allow the various systems to know each other's public keys with some strong level of guarantee that the keys are genuine. In your case, you already assume that each of your two endpoints knows the public key of the other endpoint, i.e. that the problem that which PKI strives to address has already been solved. No need for a PKI in your case, then.
This mutual knowledge of public keys can then be escalated into a two-way protected channel for data, but that's not an easy thing to do; you cannot evacuate it with a simple "we encrypt the data with the other system's public key"; as others have pointed out, this will suffer from replay attacks (at least). There are many details to worry about. A protocol where all these details have been painfully dealt with, along a course of two decades of breakage and fixes, is, indeed, SSL. So far, most people who thought they could do better have failed, sometimes hilariously, often spectacularly. SSL includes all kinds of features which keep track of individual data chunks ("records" in SSL parlance) and reliably detect external alterations, including reordered, dropped or duplicated packets; it also ensures verified termination (when the connection is closed by one machine, the other machine has some guarantee that this was indeed triggered by the other machine, not by an attacker inserting a fake RST packet).
Note that in SSL, the client and server send their public keys to each other as part of certificate chains. This supports the PKI model in which a client (respectively a server) learns a server public key (respectively a client public key) through certificate validation. However, nothing forces client and server to process the certificates in a pure X.509 PKI way. What matters is that the client (resp. the server) uses the correct public key of the server (resp. the client). If the client already knows the server key, then the client can simply use it in SSL and disregard whatever blob the server sent as certificate. Thus, SSL is not intrinsically tied to X.509 or to any other type of PKI.
In addition to the replay and ordering issues identified by jhoyla, there is no entity or message origin authentication in this protocol.
Any party with the public key of either peer can send arbitrary messages to the peer, and the peer has no way of authenticating their origin. Indeed, they could even fake a conversation between two peers, especially if they can block outbound messages from the participants or the details of the sequence number scheme you implemented allowed the attacker's messages to 'get in first'.
As it's a public key, you can fairly safely assume your attacker has access to it.