A key exchange scheme consists of two algorithms:
- A key generation algorithms, which randomly selects a keypair;
- A key exchange algorithm, which takes as input your private key and the remote party's public key, and outputs a shared secret.
A signature scheme is a triple of algorithms:
- A key generation algorithm, which randomly selects a keypair.
- A signing algorithm, that takes a message and a private key and outputs a signature.
- A verification algorithm, which takes a message, a signature and a public key, and outputs a boolean indicating whether the combination is valid.
To perform an authenticated ephemeral key exchange, the parties must agree on a key exchange scheme and a signature scheme, and must have each other's authenticated signature public key. Then:
- Both parties generate their own ephemeral key exchange keypair;
- Both parties signs their ephemeral key exchange public key;
- Both parties send their ephemeral key exchange public key to the other, along with the signature of that key;
- Both parties check the signature on the other's ephemeral key exchange public key, and abort if it's invalid;
- Both parties now use their ephemeral key exchange private key and the other's ephemeral key exchange public key to compute the shared secret.
This can also be done by having only one of the parties sign their ephemeral key exchange public key. That's how we normally do HTTPS, for example. The other party then doesn't get any guarantees that the other one is who they claim they are.
You can choose any combination of signature and Diffie-Hellman algorithms for this. It doesn't matter if the signature scheme is RSA and the key exchange scheme is ECDH. In that case step #1 uses the ECDH key generation algorithm to generate an ECDHE keypair, and then step #2 uses the RSA signing algorithm to sign that ECDHE public key. The signature algorithm doesn't care that the message it's signing is an ECDHE public key—it's just data for one party to sign and then the other to verify.
Another thing to note is that the title of your question reveals you're confused about something:
RSA or ECDHE for x.509 certificates-what does each do?
ECDHE is not involved in the certificate. The certificate contains a public signature key, metadata describing its owner, and signatures to help the recipient authenticate that the metadata is accurate. The most popular signature algorithm used in certificates is RSA. ECDSA is another alternative. ECDH is not relevant, because it's not a signature algorithm.
With certificates, the sketch algorithm above would be modified by adding two steps at the beginning:
- Both parties send their certificate to each other.
- Both parties use their PKI to verify the other's certificate, and abort if it's invalid.
Then the procedure continues using the certificate's enclosed keys to sign and verify the key exchange.