DHE_RSA Pubkey Length in TLS 1.2?

Question

Summary:

A TLS client appears to be failing to negotiate when the server hands over a 127-byte pubkey in the DHE_RSA Server Key Exchange message, but succeeding when it hands over a 128-byte pubkey. What's the deal with pubkey length, and specifically, is this legitimate behavior on the server's part?

Gory details:

I have a client (software unknown) that's experiencing intermittent failures connecting to my TLS server (F5, TLS 1.2). Successful and failed connections all settle upon Cipher Suite TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x009f) in the Server Hello. All connection attempts reach Server Hello Done and failed connections are FINned by the client immediately following that.

Looking at packet captures, the failures all seem to coincide with a "Pubkey Length" of 127 in the Server Key Exchange handshake:

On the other hand, every successful negotiation with this client has involved a "Pubkey Length" of 128 in the Server Key Exchange handshake:

Whenever the server sends the 127-byte version, the client responds with a FIN/ACK instead of the Client Key Exchange. The conclusion we've come to is that the 127-byte pubkey length is not acceptable to the client. I've read RFC 5246 to see if I can get an idea for what's allowed and they're not helpful (to me, at least!):

struct {
    opaque dh_p<1..2^16-1>;
    opaque dh_g<1..2^16-1>;
    opaque dh_Ys<1..2^16-1>;
} ServerDHParams;     /* Ephemeral DH parameters */

 struct {
     select (KeyExchangeAlgorithm) {
         case dhe_rsa:
             ServerDHParams params;
             digitally-signed struct {
                 opaque client_random[32];
                 opaque server_random[32];
                 ServerDHParams params;
              } signed_params;
    };
} ServerKeyExchange;

So the specific questions are:

1. What's the deal with Pubkey Length in DHE_RSA Server Key Exchange? Specifically, does 127 vs. 128 represent something like zero-truncation to save length, or is it truly providing a 127-byte key?

2. More specifically, is 127 a valid Pubkey Length that the client should be able to handle, or could this represent an issue with the server that needs addressing?

Clarification:

The server in question is an F5 load balancer running an up-to-date (11.6) software, so it's not only off-the-shelf but off-a-reasonably-high-shelf when it comes to TLS provenance. That's not to say it can't be the issue, just to say that it's not like I'm rolling my own Lua-over-Django-with-third-Rails TLS server in the backend :)

What OpenSSL thinks

I left a script using openssl s_client to hit the server every minute over the weekend, and captured full packets, so that I could see what OpenSSL thinks of this particular server configuration. And simply put, with a 127-byte DHE "Pubkey" length, OpenSSL considers the "Server Temp Key" 1024-bit.

Here's the Server Key Exchange as displayed in Wireshark:

And here's the OpenSSL output for that connection:

$ tail -25 20151107234244.txt
---
No client certificate CA names sent
Server Temp Key: DH, 1024 bits
---
SSL handshake has read 2014 bytes and written 291 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : DHE-RSA-AES256-GCM-SHA384
    Session-ID: 8C48CAD47FC01AF350CF618E21AE9C7E8764BB7C7243D8D6204F95634523EF2F
    Session-ID-ctx:
    Master-Key: FB80D1070A3BB2435C2D4E50D6633DA3DE4FDCFA5C8A922E17EC6FB0EDC41E259F55DFC33345B51F9A90568B36CFBB7C
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1446957764
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---
$

So is it 1024 or 1016 bit? Does the 'p' length indicate key strength instead of the Pubkey Length (ssl.handshake.ys_len for those following along in Wireshark)? Where is "Server Key Exchange" for Diffie-Hellman defined? Someone found enough documentation to write a Wireshark parser, if I could look at the doco they used maybe I'd better understand the expectations around these fields.

Answer 1

In all your examples, all DH keys are of size "1024 bits", exactly. In DH, there are the parameters (modulus p, generator g), the private key (x) and the public key (y). The relation is y = g^x mod p.

The "key length" is the size of the prime p. In all your cases, p lies between 2¹⁰²³ and 2¹⁰²⁴, so it is a "1024-bit integer" and this is the size of the DH key pair. Now, the "public key" itself is the integer y which lies between 1 and p-1; in particular, nothing forces y to be larger than 2¹⁰²³, and this is not a problem. The "pubkey length" field is the size, in bytes, of the encoded value y (in big-endian unsigned notation). Depending on the exact value of p, it is expected that between 1/256th and 1/128th of all DH public keys will need only 127 bytes for their encoding. In your last example, the value y is encoded over 127 bytes, the first one being 06, which implies that y is technically a 1011-bit integer; the DH key length is still 1024 bits since that is the size of p.

Thus, the "pubkey length" relates to the public key length as in "length of the encoding of y", but NOT as in "length that tells whether the key can be broken by powerful attackers".

Apparently, some system in your setup (the client, or some router/firewall in between the client and server -- maybe the FIN you observe on the server is not sent by the client itself) is confused and fails to distinguish the two notions of "pubkey length". Configuring your F5 to use a larger modulus for DH may work around the problem, and could be considered as a good idea anyway (1024-bit DH is not yet breakable, but theoretically you would like to use 1024-bit DH no more than you would accept to use 1024-bit RSA).

Answer 2

I just wanted to sum up the current state on this issue:

• A F5-engineer filed a bug at Microsoft regarding this issue that can be found here. It seems MS has accepted it and is working on a solution, so it really seems to be an SCHANNEL-issue.
• Currently the only solution seems to be to disable DHE-ciphers.

Answer 3

Heads up - OpenSSL 1.1.0 has a change I submitted to address this (see the PR here), and it's also been merged into the 1.0.2 branch as of Dec 14, 2016, so it will be in any future 1.0.2 releases. It would be trivial to backport to 1.0.1 as well, if you need that.

Answer 4

The length of the public key is dependent on the value of the prime field. I'd guess that indeed the key size is smaller than 128 bytes / 1024 bits. That means that there is something wrong with the software on the server.

In principle any sized prime can be used for Diffie-Hellman; in the end the value will be used as a number. Using a 127 byte / 1016 bit key however does have some problems:

• not all software is able to handle any particular key size, sometimes software requires a multiple of 8 bytes for instance;
• 1024 bits is already a very small size for DH, you would nowadays hope for larger sizes such as 2048 bits.

I'd however first contact the organization that programmed the server. Even if a 1016 bit key size is not directly a problem, it may be an indication of more serious problems.