Diffie–Hellman key exchange protocol vulnerable to man in the middle attack? If yes, then what type of MITM is that? I believe that is a replay attack or a relay attack, not sure though.
It is vulnerable in one version. Just to be more clear, the three basic versions of Diffie-Hellman are:
Anonymous DH, where Alice and Bob simply send each other unsigned values;
Fixed DH (sometimes simply called DH);
Ephemeral DH (see below).
Anonymous DH is easily vulnerable to a MitM attack, as follows.
Alice generates a value X, Bob generates Y, Eve generates Z.
Alice sends g^X mod p. Eve intercepts the message and changes it with g^Z (her own random value) mod p, before sending it to Bob.
Bob now thinks Alice has sent him g^Z mod p, as there is no signature nor any other authentication mechanism.
Bob sends g^Y mod p. Eve intercepts the message, changes it with g^Z mod p and sends it to Alice.
Alice now computes (g^Z mod p)^X mod p.
Bob now computes (g^Z mod p)^Y mod p.
Eve computes (g^X mod p)^Z mod p, which is her secret with Alice, and (g^Y mod p)^Z mod p, which is her secret with Bob.
Alice encrypts "I love you" with (g^X mod p)^Z mod p.
That key has been agreed with Eve, so Eve can decrypt the message, change it to "I hate you" and re-encrypt it with her secret with Bob. In other words, Eve can act as a proxy by having agreed on a different key with Alice and Bob.
This can be avoided by using Fixed DH - where g^X and g^Y are signed by a trusted CA - or, even better, Ephemeral DH.
In Ephemeral DH (DHE) X, Y are changed every time, i.e. at each session. They are also signed, in order to prevent MitM. However, because they always change, and the signature changes with the signed message, how can the other party verify the authenticity of the signature?
The DHE solution is simple: use a certificate. If Alice can sign her own g^X with a private key and prove that the corresponding public key is associated to her, via a certificate, Bob can verify the certificate and, by doing so, verify Alice's identity and therefore the authenticity of her g^X.
Therefore, DHE works like this.
- Alice previously obtains by some trusted Certification Authority (CA) saying "Name: Alice, PublicKey: PKAlice. Signed by CA".
More formally, the certificate is Cert_Alice=(Alice, PKAlice)_CA, which means that it contains the public key of Alice PKAlice and is signed by the CA.
The same is done by Bob, which obtains a similar certificate.
Alice generates X and computes g^X mod p.
Alice then sends two messages to Bob, M1=(g^X mod p, PKAlice)_Alice (signed by Alice) and M2=Cert_Alice.
Bob verifies that M2 has been really signed by the trusted CA (by using the known PKCA to see if the signed message matches the signature), and that it really belongs to Alice.
If M2 has passed the previous tests, it means that it has been sent by Alice (unless someone has stolen Alice's private key and the certificate is still valid). At this point, Bob uses the certified PKAlice, which was inside M2, to verify the signature of M1. If everything is OK, Bob stores g^X mod p.
Bob generates Y and computes g^Y mod p.
Bob then sends two messages to Alice, M3=(g^Y mod p, PKBob)_Bob (signed by Bob) and M4=Cert_Bob.
Alice checks the validity of M4 (see 4).
If M4 has been completely verified, Alice uses the certified PKBob, which was inside M4, to verify the signature of M3. If everything is OK, Alice stores g^Y mod p.
Alice and Bob separately compute (g^Y mod p)^X mod p==(g^X mod p)^Y mod p.
At each new session, restart from 2.
Of course, using signed messages after the DH exchange would allow to immediately detect that something weird has happened, but there's nothing in anonymous DH itself that prevents MitM attacks.