SSL over self-implemantation of RSA?

Question

I currently write a client application which communicates with a PHP server.

The application itself requires valid user credentials and get all of his information by doing POST requests to the PHP server. Is the first scenario safe or should I use SSL/TLS instead?

1st scenario (already implemented):

• Client application has a hardcoded RSA public key
• Application generates a random AES key and IV on each start. (Never uses the same key and IV to encrypt the username and password)
• Client application encrypts the AES key and IV with RSA public key in a string
• PHP server decrypts the string and stores the AES key and IV
• Client application encrypts the username and password when trying to login
• PHP server receives the encrypted data, decrypts the data and uses password_verify to see if login credentials match which the stored ones
• After successful login the client application needs some user data and request a nonce from the PHP server using the encrypted username and password
• The PHP server encrypts the nonce and sends it to the client application
• The client application decrypts the nonce and sends it with the POST request
• PHP server takes a look in the database and sees that the nonce is valid
• PHP server sends the AES encrypted data the user asked for
• The client application now decrypts the data

2nd scenario:

• Implementing mutual ssl authentication

Answer 1

If you want some serious security, you need to think not only about confidentiality (making the data unreadable to passive eavesdroppers) but also integrity (reliably detecting alterations from active attackers). Active attackers are not a sci-fi notion; the typical cheap active attacker is a bad guy who sits in a park or a restaurant and operates a fake "free WiFi" access point, waiting for victims to connect through him.

Integrity must be guaranteed within each individual message (so "AES encryption" is not sufficient, you also need a MAC) and between successive messages (the attacker might try to drop messages, duplicate messages, reorder messages, and mix conversations between various servers and clients).

All of this can be done, but it requires a lot of care and effort, and the final result will be, more or less, SSL/TLS: when your protocol has been sufficiently expanded and fixed to account for all the possible attacks, it will exhibits internal features similar to that of TLS, with comparable complexity. Therefore, there is little gain to expect over simply using TLS.

Moreover, the protocol which is easiest and fastest to design and implement securely (a rather daunting task) is the protocol which has already been designed and securely implemented, so you will save a lot of time and headaches by simply using TLS. In your specific case, where you have a dedicated application, you could use a basic self-signed certificate on the server side and hardcode a copy of that certificate in the client, so that X.509 validation is a simple bit-by-bit comparison (so no need to buy and renew a certificate from a commercial CA). This certificate handles server authentication; once the tunnel is set, the client can simply send the username and password within it.