How do certificates stop this man in the middle attack?

Question

Alice and Bob are trying to set up a secure conversation, but unknown to them in the middle is Eve, who can intercept and modify all packets sent between them.

The communication goes as follows:

1. Alice sends a hello to Bob, and includes some parameters (say, an ephemeral Diffie-Hellman public key) A. Eve intercepts the message and replaces A with E.

2. Bob replies with parameters B, and also sends his certificate which contains his public key and his domain. Eve intercepts this. At this point Eve and Bob derive a shared secret EB from E, B, and their associated private data (say, associated ephemeral DH private keys), which Alice does not know.

3. Eve forwards the message from (2) to Alice, leaving the certificate the same (not being a CA, she can't create her own) but changing B to E. At this point Alice and Eve derive a shared secret EA from E, A, and associated private data, which Bob does not know.

4. Alice verifies the certificate. It wasn't tampered with, so no flags are raised.

5. Alice encrypts a message using Bob's public key (from the certificate) and sends to Bob. It contains the shared secret EA.

6. Eve intercepts this message (which admittedly she can't decrypt), and discards it. She instead sends Bob a new one, encrypted with his public key, containing the shared secret EB. Bob looks at this and it matches the shared secret he was expecting, and so considers the handshake to be a success.

From now on Bob and Eve have a conversation using the shared secret EB, and Eve and Alice have a conversation using the shared secret EA. Neither Bob or Alice are aware that Eve is eavesdropping.

Supposedly, this is meant to be prevented by the certificate, but I don't understand how.

Answer 1

No actual security system works like you describe, for exactly this reason. The certificate is not used to encrypt the shared secret to verify that they have the same secret; that would be pointless (you might as well just have one side generate the secret and send it encrypted with the other side's public key). When the shared secret is generated using data not in the certificate, the role of the certificate is to sign that data; a rule of thumb is that you should never accept any input to the key generation process that didn't come from you and isn't signed by a trusted public key (with TLS the CA keys come preloaded into a browser, so they're trusted; the CA-signed certificate then establishes trust in Bob's public key).

In this case, the actual way the communication would work is:

1. Alice sends Bob A. Eve intercepts it and replaces it with E.
2. Bob sends back a certificate, B, and a signature for B that can be verified with the certificate. The certificate is for a signing key, not an encryption key (with RSA the difference is just what you designated the key for, but there are algorithms that are encryption-only or signing-only).
3. Eve intercepts that and replaces B with E. However, she can't sign E with Bob's private key, so she either changes the certificate and signs E with the new one, or leaves the certificate as-is (but doesn't create a new signature with it, because she can't).
4. Alice sees that either the certificate isn't signed by a CA, or that the signature verification fails (the signature is not a valid signature of E).
5. Alice terminates the communication, because she knows it's being tampered with.

Answer 2

Alice sends hello to Bob which contains a code A. Eve intercepts the message and instead puts a code E.

The above just won't happen. before Alice sends anything, Alice uses the CA's public key(present in her own Certificate store) to verify bob.com(Eve), the verification fails and so Alice will receive a warning saying that Bob's certificate is invalid.

If somehow Alice has Eve's certificate added in Alice's trusted certificate store, then Alice is in big trouble because, then Alice will not verify bob.com against a CA's public key.