A TLS client appears to be failing to negotiate when the server hands over a 127-byte pubkey in the DHE_RSA Server Key Exchange message, but succeeding when it hands over a 128-byte pubkey. What's the deal with pubkey length, and specifically, is this legitimate behavior on the server's part?

Gory details:

I have a client (software unknown) that's experiencing intermittent failures connecting to my TLS server (F5, TLS 1.2). Successful and failed connections all settle upon Cipher Suite TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x009f) in the Server Hello. All connection attempts reach Server Hello Done and failed connections are FINned by the client immediately following that.

Looking at packet captures, the failures all seem to coincide with a "Pubkey Length" of 127 in the Server Key Exchange handshake:

Wireshark capture with 127-byte DHE_RSA pubkey

On the other hand, every successful negotiation with this client has involved a "Pubkey Length" of 128 in the Server Key Exchange handshake:

Wireshark capture with 128-byte DHE_RSA pubkey

Whenever the server sends the 127-byte version, the client responds with a FIN/ACK instead of the Client Key Exchange. The conclusion we've come to is that the 127-byte pubkey length is not acceptable to the client. I've read RFC 5246 to see if I can get an idea for what's allowed and they're not helpful (to me, at least!):

struct {
    opaque dh_p<1..2^16-1>;
    opaque dh_g<1..2^16-1>;
    opaque dh_Ys<1..2^16-1>;
} ServerDHParams;     /* Ephemeral DH parameters */

 struct {
     select (KeyExchangeAlgorithm) {
         case dhe_rsa:
             ServerDHParams params;
             digitally-signed struct {
                 opaque client_random[32];
                 opaque server_random[32];
                 ServerDHParams params;
              } signed_params;
} ServerKeyExchange;

So the specific questions are:

  1. What's the deal with Pubkey Length in DHE_RSA Server Key Exchange? Specifically, does 127 vs. 128 represent something like zero-truncation to save length, or is it truly providing a 127-byte key?

  2. More specifically, is 127 a valid Pubkey Length that the client should be able to handle, or could this represent an issue with the server that needs addressing?


The server in question is an F5 load balancer running an up-to-date (11.6) software, so it's not only off-the-shelf but off-a-reasonably-high-shelf when it comes to TLS provenance. That's not to say it can't be the issue, just to say that it's not like I'm rolling my own Lua-over-Django-with-third-Rails TLS server in the backend :)

What OpenSSL thinks

I left a script using openssl s_client to hit the server every minute over the weekend, and captured full packets, so that I could see what OpenSSL thinks of this particular server configuration. And simply put, with a 127-byte DHE "Pubkey" length, OpenSSL considers the "Server Temp Key" 1024-bit.

Here's the Server Key Exchange as displayed in Wireshark:

OpenSSL-triggered Server Key Exchange handshake

And here's the OpenSSL output for that connection:

$ tail -25 20151107234244.txt
No client certificate CA names sent
Server Temp Key: DH, 1024 bits
SSL handshake has read 2014 bytes and written 291 bytes
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
    Protocol  : TLSv1.2
    Cipher    : DHE-RSA-AES256-GCM-SHA384
    Session-ID: 8C48CAD47FC01AF350CF618E21AE9C7E8764BB7C7243D8D6204F95634523EF2F
    Master-Key: FB80D1070A3BB2435C2D4E50D6633DA3DE4FDCFA5C8A922E17EC6FB0EDC41E259F55DFC33345B51F9A90568B36CFBB7C
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1446957764
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)

So is it 1024 or 1016 bit? Does the 'p' length indicate key strength instead of the Pubkey Length (ssl.handshake.ys_len for those following along in Wireshark)? Where is "Server Key Exchange" for Diffie-Hellman defined? Someone found enough documentation to write a Wireshark parser, if I could look at the doco they used maybe I'd better understand the expectations around these fields.

  • ServerKeyExchange is defined in rfc5246 at the link you already posted (and its predecessors). DHE (different from static or anon DH at the protocol level) is case dhe_dss: case dhe_rsa: ServerDHParams params; digitally-signed [irrelevant] and ServerDHParams is opaque dh_p<1..2^16-1>; opaque dh_g<1..2^16-1>; opaque dh_Ys<1..2^16-1>;. The variable length vectors (4.3 in the same document) implies the encodings don't require padding zeros, although it doesn't say so explicitly AFAICS. Nov 9, 2015 at 21:18
  • If both client(s) and F5 are capable, you might enable/use ECDHE instead of DHE. The public-key value for ECDH is not a potentially-variable-size number like DH but a point in SECG/X9.62 format, which actually has several options (compressed and uncompressed), but those options are explicitly negotiated/controllable by the EC-point-formats extension in Hello, and each option is fixed size for a given curve (size). Nov 9, 2015 at 21:24
  • @dave_thompson_085 RFC 5246 describes ServerDHParams as "three fields, each between 1 and 65535 bytes long". That doesn't match what I'm seeing Wireshark parse, which is three pairs of var_length and then var (going for length). That's why I wonder if there's a more explicit definition out there. Not as important now that Thomas Pornin has explained the fields.
    – gowenfawr
    Nov 10, 2015 at 14:00
  • 1
    Every 'variable-length vector' field begins with a length prefix as explained in 4.3. Wireshark chooses to display and label this separately, I'd guess because it handles a huge variety of protocols and formats and datatypes and this is a simple way to be consistent. Nov 10, 2015 at 15:33
  • 1
    I wanted to add a note that SChannel on Windows 7 and 8 suffers from this bug when connecting to servers using DHE_RSA key exchange. Every once in the while the response from a server is one byte shorter than normal (traced to the ServerDHParams struct in the ServerKeyExchange message), and InitializeSecurityContext() returns the error SEC_E_BUFFER_TOO_SMALL .
    – wbond
    Dec 9, 2015 at 18:44

4 Answers 4


In all your examples, all DH keys are of size "1024 bits", exactly. In DH, there are the parameters (modulus p, generator g), the private key (x) and the public key (y). The relation is y = gx mod p.

The "key length" is the size of the prime p. In all your cases, p lies between 21023 and 21024, so it is a "1024-bit integer" and this is the size of the DH key pair. Now, the "public key" itself is the integer y which lies between 1 and p-1; in particular, nothing forces y to be larger than 21023, and this is not a problem. The "pubkey length" field is the size, in bytes, of the encoded value y (in big-endian unsigned notation). Depending on the exact value of p, it is expected that between 1/256th and 1/128th of all DH public keys will need only 127 bytes for their encoding. In your last example, the value y is encoded over 127 bytes, the first one being 06, which implies that y is technically a 1011-bit integer; the DH key length is still 1024 bits since that is the size of p.

Thus, the "pubkey length" relates to the public key length as in "length of the encoding of y", but NOT as in "length that tells whether the key can be broken by powerful attackers".

Apparently, some system in your setup (the client, or some router/firewall in between the client and server -- maybe the FIN you observe on the server is not sent by the client itself) is confused and fails to distinguish the two notions of "pubkey length". Configuring your F5 to use a larger modulus for DH may work around the problem, and could be considered as a good idea anyway (1024-bit DH is not yet breakable, but theoretically you would like to use 1024-bit DH no more than you would accept to use 1024-bit RSA).

  • Excellent, thank you. This gives me enough information to go back to the client and discuss their behavior, as well as to improve my server config.
    – gowenfawr
    Nov 9, 2015 at 14:54
  • @Thomas: Thanks. Could you explain the background for "between 1/256th and 1/128th"? (Because my intuitive/off-the-cuff-answer to the question "How many keys where the MSByte is 0x00 and the second-to-MSByte is not 0x00 are there?" would have been: "Exactly 1/256 times 255/256 times the total amount.") Nov 9, 2015 at 16:27
  • @StackzOfZtuff: value y is lower than p so its MSB cannot exceed the MSB of p. This reduces the range of values. If p is very close to 2^1023, then its MSB will be 0x80, and the MSB of y will be almost always between 0x00 and 0x7F. Nov 9, 2015 at 17:30

I just wanted to sum up the current state on this issue:

  • A F5-engineer filed a bug at Microsoft regarding this issue that can be found here. It seems MS has accepted it and is working on a solution, so it really seems to be an SCHANNEL-issue.
  • Currently the only solution seems to be to disable DHE-ciphers.

Heads up - OpenSSL 1.1.0 has a change I submitted to address this (see the PR here), and it's also been merged into the 1.0.2 branch as of Dec 14, 2016, so it will be in any future 1.0.2 releases. It would be trivial to backport to 1.0.1 as well, if you need that.

  • Great link. You might want to summarize some of the salient points, such as how it addresses the backwards compatibility issue, the fact that other libraries also compensate, and the way it addresses the problem, in case that link moves.
    – gowenfawr
    Dec 19, 2016 at 6:57

The length of the public key is dependent on the value of the prime field. I'd guess that indeed the key size is smaller than 128 bytes / 1024 bits. That means that there is something wrong with the software on the server.

In principle any sized prime can be used for Diffie-Hellman; in the end the value will be used as a number. Using a 127 byte / 1016 bit key however does have some problems:

  • not all software is able to handle any particular key size, sometimes software requires a multiple of 8 bytes for instance;
  • 1024 bits is already a very small size for DH, you would nowadays hope for larger sizes such as 2048 bits.

I'd however first contact the organization that programmed the server. Even if a 1016 bit key size is not directly a problem, it may be an indication of more serious problems.

  • Depending upon the key generation protocol the server may have randomly generated a key with leading Zeros. Usually, this is circumvented with a proper key gen step. However, "clever" developers sometimes decide to roll their own solutions. So, OP may have a server key of odd provenance. Nov 7, 2015 at 0:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.