Establishing safe connection in Java

Question

In my Java project I'm trying to create a 100% secure method of communication between the method and the client. I used to use this process:

Client: generates 4096-bit RSA keypair
Client: sends public to server
Client: generates 256-bit AES key
Client: encrypts AES key using RSA and send
Server: decrypts AES key from RSA
Server & Client now communicate using AES only

But I found out this isn't safe as it can easily be ruined by a man-in-the-middle attack. I began researching TLS and found out about security certificates. My question is this: if the client generates the RSA keypair, signs it using the certificate, and sends it to the server, what stops a MITM from doing the same thing (assuming the certificate is publicly available, which I assume it would be because the server and the client would both need it).

When I use OpenSSL to generate a certificate it always provides an RSA key alongside it. Isn't it safer to generate a new keypair for each connection, or do I actually use this particular key? What am I missing about the standard pattern for TLS?

Answer 1

If the client generates the RSA keypair, signs it using the certificate, and sends it to the server, what stops a MITM from doing the same thing?

That does not happen. If the server have a CA-issued certificate, the client will know for sure that the certificate belongs to the server. If an attacker creates a fake certificate, it won't be signed by a valid CA and will be invalid.

When the client receives the certificate, it checks the signatures on it, and who is the CA issuing it. If the CA is not trusted, the certificate is not trusted either (and the client issues the error sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target or something similar).

The attacker could connect to the server, receive the certificate, and generate its own AES keys, but that would be useless in any MitM scenario, as his keys aren't the ones the client uses, so they cannot decrypt anything.

Isn't it safer to generate a new keypair for each connection, or do I actually use this particular key? What am I missing about the standard pattern for TLS?

You are missing quite a lot. The RSA key cannot change on every connection, unless you want to issue a new certificate, send it to the CA to sign it, install it, and repeat that for every single connection.

TLS have some complexity on it, but the over-simplified ELI5 version is:

1. client connects to the server and gets the certificate

2. client sends a random key (the client key), and all ciphers he supports

3. server responds with a random key (the server key) and the cipher he will use

4. at this point, the session is established and secure

There's more details on this, but is basically that. The server certificate have the server RSA key, the issuer (the CA), the subject (the servers it is valid for), the validity (expiration date), and the client uses that information to decide if it trusts the certificate or not.

So use TLS, don't bake your own crypto. TLS is around for a lot of time, is battle-tested, have extensive documentation, libraries, and examples available, and it's easy to any programmer to inherit the project later and keep going.