If I understand your scheme correctly, it's this:
- Client generates a random AES key and initialisation vector (IV).
- Client encrypts that key with the RSA public key of your server, which it already possesses.
- Client pads the message with PKCS#5 and encrypts it with AES-CBC using the IV + key they generated.
- Client concatenates IV + encrypted key + ciphertext and sends it to the server.
- Server receives the packet and splits it back out into IV + encrypted key + ciphertext.
- Server decrypts the encrypted key using its RSA private key, but returns no error message if this fails.
- Server uses the IV and decrypted key to decrypt the ciphertext, again using AES-CBC.
- Server validates the PKCS#5 padding of the decryption and discards the message if the padding is not correct. No error is returned to the client.
If so, there are a number of problems here, mostly relating to malleability:
- An attacker can pick a tweak T and replace the IV with IV ⊕ T, resulting in the first block of decrypted plaintext becoming M0 ⊕ T. This can be used along with known plaintext in order to spoof message contents.
- An attacker can pick a tweak T and replace any ciphertext block Cn with it, resulting in Mn becoming corrupted and Mn+1 becoming Mn+1 ⊕ T. If an attacker knows that the data in Mn is not validated or not important for the purposes of the attack scenario, this can be used to spoof message contents via a known plaintext.
- Even if you send back no error messages, there may be timing differences in your code that allow an attacker to distinguish (within a reasonable probability bound) between valid and invalid padding. This could be exploited in order to recover the plaintext of packets.
I recommend that you modify your scheme to the following:
- Client generates a pair of random keys Ke and Km plus a random initialisation vector (IV).
- Client concatenates Ke and Km together to make a combined key K.
- Client encrypts K with the RSA public key to create the encrypted combined key K'.
- Client pads and encrypts the plaintext message M with PKCS#5 and AES-CBC using Ke and the IV, resulting in ciphertext C, i.e. C = E(M, IV, Ke)
- Client concatenates IV and K' and C to create an encrypted packet P.
- Client computes H(P, Km) where H is a HMAC function, e.g. HMAC-SHA256, resulting in an authenticity record A.
- Client concatenates P and A and sends this to the server.
- Server extracts P and A from the packet.
- Server splits P back into IV, K', and C.
- Server decrypts K' with its RSA private key. If this step fails, the message is discarded. If not, the server now has the combined key K.
- Server extracts Ke and Km from K.
- Server validates that H(P, Km) == A. If this step fails, the message is discarded.
- Server decrypts C using Ke and IV, i.e. M = D(C, IV, Ke), and validates the PKCS#5 padding. If the padding is invalid, the server discards the message.
The addition of an authenticity record ensures that an attacker cannot capture a message from the client, modify it, and re-send it to exploit a malleability attack or padding oracle issue. The idea is that an attacker could replace K' with their own value in order to know Km and forge messages, but in the process they'd also have to replace Ke which would result in invalid padding or completely garbled plaintext.