I've been reading though various RFCs and couldn't find a definite answer to my question: can a negotiated TLS extension skip some of the TLS Handshake messages and still be compliant with the TLS specification? My goal is to develop a new version of TLS, while preferably, staying backwards-compatible.
Here, I will be specifically talking about
TLS 1.2, defined in RFC 5246. Below is a message flow for the full handshake (taken directly from RFC 5246):
Client Server ------ ------ ClientHello --------> ServerHello Certificate* ServerKeyExchange* CertificateRequest* <-------- ServerHelloDone Certificate* ClientKeyExchange CertificateVerify* [ChangeCipherSpec] Finished --------> [ChangeCipherSpec] <-------- Finished Application Data <-------> Application Data * Indicates optional or situation-dependent messages that are not always sent.
Now, I do know that it's perfect legal for a TLS extension to modify the structure of some message or add a new message, but I'm not sure if one of the messages, not defined as optional/situation-dependent can be omitted.
Let me give you a concrete example. Let's say I create a new extension called
XYZ. The client and the server negotiate that extension in the their extended hello messages. Would it be legal for the
XYZ extension to mandate the server not to send the
ServerHelloDone message? As far as I understood, this is not legal.
RFC 5245 Section 126.96.36.199 states that:
it would be technically possible to use extensions to change major aspects of the design of TLS; for example the design of cipher suite negotiation. This is not recommended; it would be more appropriate to define a new version of TLS -- particularly since the TLS handshake algorithms have specific protection against version rollback attacks based on the version number, and the possibility of version rollback should be a significant consideration in any major design change
I would assume, however, that those major aspects do no include omitting messages not marked as optional/situation-dependent in the spec.