How exactly is this protecting against a malicious DNS routing?
Not at all.
If an evil guy has a valid certificate (e.g. from a hacking a CA) and then manages to man-in-the-middle you, then your connection is hacked.
I was expecting a challenge response using the public key of the server, but I cannot find it.
In order to do this you would have to know some things about the server first. And you don't usually have that info. But certificate pinning is one way of communicating that info:
Certificate pinning helps
Certificate pinning allows you to give a client some extra info about your certificates. Such as: "For my certs, please only accepts signatures by ACME-CA." Or "For my certs, please only accept them if they contain the following public-keys ...".
Further reading for pinning: Wikipedia: HTTP Public Key Pinning
EDIT:How are key exchanges authenticated?
From the comments section:
I am exactly asking how the server proves that he has the private part. It should be a challenge/response initiated by the client using the public key of the server (or using the symmetric key once it is established). I cannot find this in the texts
Indeed, there is challenge/response here.
This is detailed in a subsection of the TLS 1.2 RFC.
There are three options:
- F.1.1.1. Anonymous Key Exchange
No authentication at all.
- F.1.1.2. RSA Key Exchange and Authentication
The client encrypts (part of) the bulk encryption session key with the server's public key. The server's ability to decrypt is proof of possession of the private key.
No "Forward secrecy":
If the server's private key is EVER exposed, even years late, then all old session keys can be retroactively decrypted.
- F.1.1.3. Diffie-Hellman Key Exchange with Authentication
Client and server use a different key agreement protocol. The server then signs the a transcript of the negotiations with its private key. If the client can verify this signature with the server's public key, then that is proof that the server really knows the private key.
Has "Forward Secrecy":
The advantage of this procedure is "Forward secrecy". If the server's private key is ever exposed years later, then session keys can NOT be retroactively decrypted. (Because, well, they were never encrypted with the server's private key in the first place. See Diffie–Hellman key exchange for details.)