Is this authentication and encryption solution secure?

Question

I need to communicate over insecure TCP sockets where SSL/TLS is not available. Does the following solution provide a secure means of authentication and encryption?

-

1. Client sends: username
2. Server generates a random token (31 bytes).
3. Server retrieves a password salt and hash (bcrypt) from the database for the given username.
4. Server encrypts the random token by the password hash.
5. Client receives: passwordSalt + encryptAes(token, passwordHash)
6. Client gets the passwordHash by using bcrypt on the password and the received password salt.
7. Client gets the token by decrypting the encrypted token with the password hash.
8. Client generates a random 'validation' salt (15 bytes).
9. Client generates a hash of the token using the validation salt and encrypts it with the token.
10. Client sends: encryptAes(validationSalt + md5(token + validationSalt), token)
11. Server gets the hash by decrypting it with the token.
12. Server compares if the received hash is equal to md5(token). If they are the same, the client has authenticated.
13. Server generates a hash of the password hash using the validation salt, pads it with 15 random bytes, and encrypts it with the token.
14. Client receives: encryptAes(randomBytes(15) + md5(passwordHash + validationSalt), token)
15. Client gets the hash by decrypting it with the token.
16. Client compares the hash with md5(passwordHash + validationSalt). If they are the same, the server has authenticated.

All data will now be encrypted with the token generated by the server.

-

The server is in Java, the clients are in C++ (UE4).

Public key encryption is currently not available.

-

Is there any flaw or weakness in this solution?

Answer 1

You basically created a shared secret authentication scheme here.

However, you did not define what your security goals are. Entity authentication, strong entity authentication, confidentiality, anonymity, ...? So it's hard to say "(yes/no), this is (not) secure". If your security goals were none of the above, this is probably secure. But I wouldn't use this thing (hope you don't mind)! :-)

You seem to assume that the authentication key is the password. However, this is not the case as knowing the hash from the database is sufficient to successfully authenticate. Try it out and play the client which only knows the password hash and you will see that it works.

The problem here is that you didn't specify what exactly the secret is. Knowing that you can make the whole process much simpler. Assume the secret is the hash from the database and you want to authenticate the user to the server:

• Server generates a session key and a random number, encrypts both with the secret and sends it to the client
• The client generates the shared secret by asking for the password
• The client decrypts the session key
• The client adds 1 to the random number, encrypts it with the session key and sends it back to the server

As you can see the security still relies on knowing the hash from the database, but setting up a shared secret and authenticating the client has become much simpler. Please note that this is still far from secure! I only wanted to show that things are often not what they seem. Fiddling around with salts doesn't make things necessarily better.

The point is: Complicated does not mean secure, and - as pointed out before - do not trust you self-made crypto ;-)

Edit: Just for the record, think about oracles, chosen protocol attacks, reflection attacks, ... get yourself used to these things and try find attacks against your own protocol. Makes fun and you will learn a lot.

Answer 2

Your solution could be affected by the following concepts:

Do-it-yourself Cryptography

DIY crypto is an anti-pattern of secure software design. Most (>99%) attempts will fail. Nevertheless I respect your try! :-) For example: Don't reveal the "passwordSalt", if you don't need to. It helps the attacker to crack the challenge. After one request an attacker knows the salt and has a challenge which could be cracked. The security relies only on a strong password. The attacker can crack the password offline. This should not be the case for online services. I rate this as highly insecure.

Perfect Forward Secrecy

The key for the encryption (serverside generated token) should be calculated on both sides (client and server) using Diffie-Hellman Key Exchange or similar. If someone recorded the traffic and gets the user password, he is able to decrypt the token and then decrypt the whole traffic, afterwards.

Steps 13-16 not needed

If the client can decrypt "passwordSalt + encryptAes(token, passwordHash)" successfully, the server had to know "passwordHash", therefore the user password.

This list is not complete, as several other weaknesses could be present.

Answer 3

Your trying to recreate a wheel, and unless your background is in cryptography it's also likely to not be a great as your intent.

Your goal is to send encrypted data from a Server to a client with the limitation that a PKI cannot exist yet. The short is we are limited to Symmetric Key's right?

Their are two rough approaches that you can consider:

Pre-Shared key:

Pre-Sharred keys are generated and set in the client and Server and used as a common key for encryption and decryption. Your encryption of data can be secured quite well this way, but you deal with the maintenance of a PSK.

Key Derivation Function:* Key Derivation Functions work under a similar principal as PSK's but they are generated on the client and server at the start of a session. The data is shared data between the two systems to generate a secret value which can be used to encrypt and decry pt systems.

Both Java and C,C++ have libraries which can build these functions.