As TLS 1.2 puts it:
The ServerKeyExchange message is sent by the server only when the
server Certificate message (if sent) does not contain enough data
to allow the client to exchange a premaster secret.
ServerKeyExchange message containing a RSA public key would be used when the server's public key is not fit for key exchange (e.g. it is in a certificate which is marked as signature only) but client and server wish to use RSA for key exchange nonetheless.
Sending a RSA key like this is a feature which has fallen out of use, but it was used in TLS 1.0 with the
RSA_EXPORT cipher suites. At that time, there were rather stringent regulations in the USA, limiting the size of cryptographic keys that were used for encryption in systems which were exported outside of the USA (these regulations were lifted in 2000); in particular, RSA encryption was limited to 512 bits (and 40 bits for symmetric encryption). The
RSA_EXPORT cipher suites complied with these export regulations. When the server has a RSA public key, the client and the server agree to use one of the
RSA_EXPORT suites, and the server's public key is larger than 512 bits, then the server must send a
ServerKeyExchange message with a RSA key of at most 512 bits, and use its permanent RSA key (the one from its certificate) to sign the
Dynamic RSA key generation is a bit cumbersome and expensive; a given SSL server can reuse such a key with several clients, but it is still heavier than DHE. The main use case was to support feeble clients who can do RSA only; however, even if RSA public key operations are very fast, ECDHE (Diffie-Hellman with elliptic curves) is competitive and much easier for the server. TLS 1.1 clients are allowed to advertise support for the
RSA_EXPORT suites, but only for backward compatibility with SSL 3.0 and TLS 1.0 (section A.5 of TLS 1.1):
TLS 1.1 implementations MUST NOT negotiate
these cipher suites in TLS 1.1 mode. However, for backward
compatibility they may be offered in the ClientHello for use with TLS
1.0 or SSLv3-only servers. TLS 1.1 clients MUST check that the
server did not choose one of these cipher suites during the
handshake. These ciphersuites are listed below for informational
purposes and to reserve the numbers.
ServerRSAParams structure in TLS 1.1 is thus a survivance from TLS 1.0 which somehow evaded the wrath of the RFC 4346 editors, but was finally removed when RFC 5246 was produced.