OpenSSL DH modp size

Question

If OpenSSL is configured to use a DH for key exchange, e.g. TLS_DHE_RSA_WITH_AES_128_CBC_SHA, where does it take the size for modp (or even modp itself) from?

The following is a screenshot from wireshark. The server selected TLS_DHE_RSA_WITH_AES_128_CBC_SHA from the client's proposals and sends its DH parameters.

Any ideas, why OpenSSL decides to use the (too) small 96 * 8 = 768 Bit group?

Answer 1

In SSL/TLS, the group (modulus and generator) used for DHE cipher suites is entirely up to the server. OpenSSL (the library) can be configured to use a specific group; with the command-line tool (openssl s_server), this is a matter of using the -dhparam option; otherwise, the DH parameters can be included in the file containing the certificate itself.

This page contains some information, including this:

To use perfect forward secrecy cipher suites, you must set up Diffie-Hellman parameters (on the server side), or the PFS cipher suites will be silently ignored.

So the library does not use "default DH parameters"; it is up to whatever software uses the library. In the case of the command-line tool, it seems to use a 512-bit modulus, which is, to say the least, a bit short... but OK for test. In your case, one must assume that whatever is your server application which uses OpenSSL, it has configured the library with a 768-bit modulus. It is possible (but it depends on that application) that stuffing explicit DH parameters in the same file as the one containing the server's certificate suffices to force OpenSSL to uses these parameters.

Notes:

• If OpenSSL, as a server, selects an "export" cipher suite then it will force use of a 512-bit modulus for DH, but this is an edge case.
• You are not completely free to use any size you want because of interoperability issues. It seems that some SSL clients out there will not like a DH modulus longer than 1024 bits. I also surmise that a DH modulus size which is not a multiple of 32 or 64 may induce extra problems.
• With Elliptic Curve DH (the recent ECDHE cipher suite) the problem is different, because most implementations support only a handful of specific curves, not "any curve of a given size". There is an extension which allows the client to tell which curves it supports; otherwise, the server will choose depending on its configuration (in any case, anything else than P-256 will imply interoperability issues).