Is this RSA/AES combination good?

Question

The team of developers I am part of is trying to develop a safe way to exchange sensitive data between a server and mobile devices.

We have come up with the following algorithm:

1. Device generates private RSA key, and sends the public key to server.

2. Server generates unique to user AES key and uses the RSA public key to encrypt it and send it back to device.

3. Device gets the AES key. Uses it to encrypt password and username and sends it to the server.

4. Server decrypts the username and password. If there is a match, the AES key is used for secure communication for X amount of time or until log-out. Else, the process must be restarted.

Is it safe enough? Are there ways it can be improved? What are the faults?

Edit: After reading the comments, what is a safer alternative and why?

Edit2: Ok, i get it. I won't use my own implementation, will find something already tried and proven.

Answer 1

All of the weaknesses in your protocol can be summed up as "use SSL" or even "use SSL, dammit !".

In more details:

• All the protocol is of course vulnerable to impersonation, specifically the double impersonation that is also known as Man-in-the-Middle attack.
• Similarly, if any of potential attackers that can eavesdrop on the line decides to do a modification of the data in transit, then he can, and your client and server will be none the wiser.
• Experience shows that saying "encrypt all the data with that key" and then doing it correctly is awfully complex. SSL itself took almost 15 years to do that, and many implementations are still not up to it. Padding oracles, predictable IV, MAC verification timing, verified closure, protection against resequencing and replay of packets...

As an overall assessment: don't do that.

Answer 2

No this protocol isn't safe.

As already mentioned in the comments:

Don't roll your own crypto. Chances are you're getting it wrong. Especially if you admit that you don't really know what you're doing.

First, you seem to have standard problems for which there are already standard protocols, which have nice properties and security proofs. The protocol for the data transmission should be TLS v1.2 (or newer). You should use a library (like NSS, GnuTLS or OpenSSL for this). For password authentication the best choice you have is to use SRP or TLS-SRP. However if you really can't afford SRP (which should be your first choice) you can still follow this paper.
So far for the improvements.

Now for the flaws of your protocol:

1. It's vulnerable to Man-in-the-middle attacks. If an attacker sits in-between the device and the server, he can fish the public key and replace it with his own. In response he'd know the AES key and could decrypt all the data.
2. You don't mention integrity / authenticity of the data. It sounds like you want to use ECB mode for AES (which you really don't want to use), you'd be a lot better off with AES-GCM for the bulk-encryption. You also don't mention how RSA is to be used, RSA-OAEP should be your choice (and not textbook RSA). However I assume you'd have done this with your implementation.
3. You put a really large workload on the device. The device needs to generate a new private RSA key with every connection, which is a highly computing-intense task. You should rather consider using ECDH key exchange.
4. For the verification of the passwords to work, you either need the server to store the passwords in plaintext or to hash them himself, which may enable DoS attacks on the server. And in case a man-in-the-middle attack takes place, the plain passwords will be revealed to an attacker, which may cause serious harm to your users (as they may re-use their passwords across sites...)

Answer 3

1. Device generates private RSA key, and sends the public key to server.

2. Server generates unique to user AES key and uses the RSA public key to encrypt it and send it back to device.

3. Device gets the AES key. Uses it to encrypt password and username and sends it to the server.

Anonymous key exchange.

There is no authentication. Your key exchange is encrypted, but not authenticated. This is bad.

If there is an active Man-in-the-Middle, then you have no way of detecting this. This active MitM can just pretend to the client that he is the server. And pretend to the server, that he is the client.

-> Don't do it. @SEJPM is right with his comment. Use established crypto systems. Like TLS with certificates or TLS with SRP.

Answer 4

that's the way i use to do rsa /aes combination

1. client generate a random aes key
2. use the aes key to encrypt the data you need (rsa take longer time and have length limitation, so better use aes to do encrypt)
3. use the public key to encrypt the aes key
4. pass the encrypted data and the aes key encrypted by rsa public key to server
5. when server receive, decrypt that aes key by private key and then decrypt the attached data by the decrypted aes key

Answer 5

It's pretty good but I would do this in an easier manner which I think is more secure and similar to what you thought of.

Since the communication is client-server:

1. Mobile client generates key using PRNG
2. Client sends step 1 key using server public key (only server has private to decrypt)
3. Client encrypts user/password with step 1 key and sends to server
4. If user/password are verified - server returns new key for data encryption, encrypted with step 1 key

-. Please do step 2 - 4 over TLS if possible

--. Change keys every X iterations (enforce on client and server)

---. Client must trust server certificate to prevent MITM/impersonation