Securing data as it travels from a client to a server is a top priority in authentication systems. You're on the right track using RSA encryption, but the key to strong security lies in how you handle your encryption keys.
To make it more secure and avoid hard-coding keys in your client and server, consider working with a Certificate Authority. This trusted third party specializes in generating and managing digital certificates. The server possesses a certificate issued by the CA, which it shows to the client during connection. The client, in turn, checks the server's certificate and uses the public key for encrypting data. This way, you can improve security without dealing with direct key transfers.
Another option to steer clear of hard-coded keys is by using a Key Management System. The KMS securely stores and manages cryptographic keys. When a client connects to a server, it can request a key from the KMS. The KMS then creates a key and securely sends it to the client. The client can use this key for data encryption. This approach simplifies key management while enhancing security.
To make your authentication system even more robust, remember to employ strong passwords or passphrases, consider implementing Two-Factor Authentication for added security, ensure your system uses secure session management like HTTPS, keep a close watch for any suspicious activities in your system, and keep your software up-to-date to guard against potential vulnerabilities.