If you are using public key authentication for SSH, no one can log in to the server without having the corresponding private key. This is as secure, and usually more secure, than password authentication. The encryption OpenSSH provides is state of the art; there is no known way to break it.
The only remaining attack would be a man-in-the-middle attack. You can copy over the host key from the server to your client, just like you copied a key to make public key authentication possible. This will completely mitigate man-in-the-middle attacks and the client will warn you if an attempt is detected.
See also What is the difference between authorized_keys and known_hosts file for SSH?