TLS-Handshake, Server Key Exchange, Diffie-Hellman: Where does key length come from?

Question

Where does the key length come from?

In this example it is 2048 bits long:

Secure Sockets Layer
    TLSv1.2 Record Layer: Handshake Protocol: Server Key Exchange
        Content Type: Handshake (22)
        Version: TLS 1.2 (0x0303)
        Length: 783
        Handshake Protocol: Server Key Exchange
            Handshake Type: Server Key Exchange (12)
            Length: 779
            Diffie-Hellman Server Params
                p Length: 256
                p: ...
                g Length: 1
                g: 02
                Pubkey Length: 256
                Pubkey: ...
                Signature Algorithm: rsa_pkcs1_sha512 (0x0601)
                Signature Length: 256
                Signature: ...

It is apparently not a parameter of the cyper suite (here: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256) as there is no key length given for DHE.

Is this just a parameter configured on the server side? Or is it somehow negotiated? I cannot find much information except in "Bulletproof SSL and TLS" in the section about deployment best practices.

To be on the safe side, if deploying DHE, configure it with at least 2,048 bits of security.

That section makes it sound like it's server configuration...

Reason for my question:

I have got a piece of older software which cannot connect to a server via TLS. It complains that it only supports DH key lengths from 512 to 2048. And in the according network trace p Length is 512 (== 4096 bits). Thus the TLS handshake fails.

Now I need to know how I could get my legacy software (running on an embedded device) to talk to the server (preferrably without updating the software).

Answer 1

For TLS through 1.2 in most cases (see below) using DHE key exchange, the DH group (prime p and generator g of suitable subgroup of Z_p^*) is sent by the server; the client has no influence beyond specifying a list of ciphersuites that does or does not lead the server to select a ciphersuite that uses DHE (at all).

In general server software (or firmware, or middleware as applicable) varies in how it determines the DHE group to use. Some servers are hardcoded, and some are configurable; if configurable the method of configuration and options available vary wildly. However, it would be pretty aggressive for a server today to hardcode 4096-bit DH, so I'd say it's reasonably likely this server is configurable somehow.

Since you give no clue what the server is, it is impossible to give any useful answer as to that 'how'. Although if it is any of the more common ones like Apache httd, nginx, haproxy, or Tomcat (including Jboss/Wildfly/whatever it is today) you can probably find documentation and/or existing Qs here or superuser.com or serverfault.com or maybe unix.SX by searching something like "{server [version]} configure DH".

Technically there is an extension to TLS1.2 (and earlier), RFC7919 which does allow the client and server to negotiate among a standardized (specifically, IANA-registered) set of groups. But I'd be surprised if a client that can't support DH over 2048 bits supports RFC7919. You can confirm this by looking at the extensions in the ClientHello message in your capture. TLS1.3 uses only the RFC7919 groups for DHE, within its substantially new scheme for doing keyexchange, but you aren't using 1.3.

Depending on what other ciphersuites the client offers, it might be possible to configure or otherwise induce the server to choose a non-DHE ciphersuite; in that case there's no issue of the server using DHE parameters the client dislikes. Especially if both client and server support any ECDHE ciphersuite(s) with acceptable other parameters (auth and data cipher/MAC), that would provide the same forward-secrecy benefit as DHE but be more likely to interoperate successfully since for ECDHE (pre-1.3) practically everybody implemented the former Suite B curves P-256 and P-384, and maybe P-521.