If an attacker replays old handshake messages, the trickery will become apparent with the
Finished message from the attacker, because the message previously recorded was encrypted and MACed with distinct keys, and its contents won't match either:
The derivation from the master secret (resulting from the key exchange mechanism, e.g. RSA) into symmetric encryption and MAC keys involves the
server_random from the two "hello" messages; the attacker replayed one of these values (e.g. the
client_random if the attacker is replaying the client messages) but the other is from the peer, and is distinct from what was recorded.
The contents of the
Finished messages are a hash computed over all the previous handshake messages, including the two "hello" messages, and thus, again, one of the "randoms" has changed and the hash won't match.
Note: if the server requires certificate-based client authentication, and the attacker is replaying the client part of the exchange, then the replay will get detected earlier, at the
CertificateVerify message: the recorded handshake message is a signature (by the genuine client) computed over a hash of all previous handshake messages, including, again, the two "randoms". However, SSL server who ask for client certificates are relatively rare.
An attacker who begins but never finishes handshakes is doing a Denial-of-Service attack: he makes the server allocate resources and spend CPU cycles for nothing. Typically, the attacker would send a
ClientHello claiming support of only "DHE" cipher suites; then, the server must respond with a sequence of messages, including a
ServerKeyExchange message which necessarily involves a newly computed signature. The CPU cost for the attacker is minimal (only the effort of opening a TCP connection and sending a few bytes) while the server must do some cryptography.
Since that kind of DoS requires doing an actual TCP connection, the attacker's IP address is revealed; also, signatures are not that expensive: a PC can do thousands of signatures per second. Therefore, SSL-based DoS are usually done by botnets.