Client Server TCP connection - sending a password

Question

I'm new to cybersecurity and am writing a TCP server with the Java Socket API. There is a client and server class. The server class is what would be run on a server, it listens on a specified port and accepts socket connections. If the client part of the program attempts to connect, it must send the correct encrypted password to the server. The server has a list of correct passwords, hashed with SHA-256.

I have an RSA public-private key system set up, where the server keeps the private and sends the public key to the client. The client hashes the raw password with SHA-256 then uses the public key to encrypt it before sending it across the socket.

What I am worried about (and I'm sure there are other issues) is that an attacker will use the public key to try a bunch of passwords with the server. If I generate AES symmetric keys using some special string hardcoded into the client and server side, would it make sense to encrypt the public key itself with the symmetric key before sending that over the wire, and using those keys for additional communication?

I am not concerned about anything other than the password that is sent from client to server right after the socket is accepted.

Answer 1

While there is no good reason to not use TLS to safeguard this, your approach seems okay and here is why:

The computational complexity of hashing sha 256 and encrypting it with (asymmetrical or hybrid) algorithms is fairly high; there is (currently) no feasible way to brute force that if the passwords are reasonably strong.

Additionally, if you are worried about brute force, you can ban an IP for x hours if the password is entered wrong y times in a row - like fail2ban does - and/or add additional time between logins like GNU/Linux does.

That being said: there is a perfectly good way to authenticate users securely and that is TLS.

To address your trying to encrypt the public key with symmetric encryption: there is no gain in that. If you use reasonably strong key pairs, it should be computationally infeasible to generate the private key from the public key - asymmetric crypto systems are designed in such a way that public keys are meant to be public (hence the name!).

Adding a layer of AES wouldn’t harm that in any way, but introduce additional problems: the client can be reversed to get the key and render your attempt useless - and can be debugged to retrieve the symmetric and/or public key from memory. Also, you add more complexity to your software, increasing the likelihood of failure and error without a benefit.