TLS Extentions: Omitting TLS Handshake Messages

Question

I've been reading though various RFCs and couldn't find a definite answer to my question: can a negotiated TLS extension skip some of the TLS Handshake messages and still be compliant with the TLS specification? My goal is to develop a new version of TLS, while preferably, staying backwards-compatible.

Here, I will be specifically talking about TLS 1.2, defined in RFC 5246. Below is a message flow for the full handshake (taken directly from RFC 5246):

  Client                                          Server
  ------                                          ------      

  ClientHello                  -------->
                                                  ServerHello
                                                 Certificate*
                                           ServerKeyExchange*
                                          CertificateRequest*
                               <--------      ServerHelloDone
  Certificate*
  ClientKeyExchange
  CertificateVerify*
  [ChangeCipherSpec]
  Finished                     -------->
                                           [ChangeCipherSpec]
                               <--------             Finished
  Application Data             <------->     Application Data

 
* Indicates optional or situation-dependent messages that are not always sent.

Now, I do know that it's perfect legal for a TLS extension to modify the structure of some message or add a new message, but I'm not sure if one of the messages, not defined as optional/situation-dependent can be omitted.

Let me give you a concrete example. Let's say I create a new extension called XYZ. The client and the server negotiate that extension in the their extended hello messages. Would it be legal for the XYZ extension to mandate the server not to send the ServerHelloDone message? As far as I understood, this is not legal.

RFC 5245 Section 4.4.1.4 states that:

it would be technically possible to use extensions to change major 

  aspects of the design of TLS; for example the design of cipher 
  suite negotiation.  This is not recommended; it would be more 
  appropriate to define a new version of TLS -- particularly since 
  the TLS handshake algorithms have specific protection against 
  version rollback attacks based on the version number, and the 
  possibility of version rollback should be a significant 
  consideration in any major design change

I would assume, however, that those major aspects do no include omitting messages not marked as optional/situation-dependent in the spec.

Answer 1

I've sent this same question to the TLS mailing list, and got an answer that it is, in fact, legal for an extension to specify to omit messages not marked as optional/situation-dependent in the spec.

You can find the discussion here.

Answer 2

I don't see any reason why you couldn't fork from normal TLS handshake state, and do whatever you wanted based on an extension. Especially because it was negotiated and the client/server would know how to talk to each other. However, whether you are compliant with TLS 1.2 depends on the changes you make. Something like changing the handshake would take you out of compliance.

The intent of extensions, I believe, is to allow the protocol to plug security holes, and provide more security options. It's clear from RFC 5246 that there are optional and required messages in the Handshake Protocol. Omitting any required messages would violate the specification. There is also a specific order to the Handshake messages that if changed would violate the specification:

Section 7.4

The handshake protocol messages are presented below in the order they MUST be sent;

I don't think that you would be able to create a new protocol that is compliant with TLS 1.2 in the manner that you want.