If you are using public key authentication for SSH, no one can log in to the server without having the corresponding private key. This is as secure, and usually more secure, than password authentication. The encryption OpenSSH provides is state of the art; there is no known way to break it. You can further improve security on the Tor side by using authorized hidden services. This will make the domain inaccessible to all but your client. Note that this only works with v2 hidden services, not the latest v3.
The only remaining attack would be a man-in-the-middle attack. You can copy over the host key from the server to your client, just like you copied a key to make public key authentication possible. This will completely mitigate man-in-the-middle attacks and the client will warn you if an attempt is detected.
See also What is the difference between authorized_keys and known_hosts file for SSH?
