A lot of what I discussed in this answer about alternate SSH ports is applicable to this question.
Since your goal is to be as invisible as possible*, I think your best bet is secure port knocking with a tool like fwknop (as noted in this crypto.SE question) or perhaps a custom one-time password scheme for sufficiently complicated port knocks (to help protect against replay attacks).
Another idea is to port knock through Tor; there may be latency within Tor, but port knocking doesn't consume much bandwidth and can be configured to tolerate more latency. Once you've knocked on the .onion address, the end of your knock could be an encrypted (or plaintext, depending on your sensitivity) form of your client public IP address, which would then be able to access the server's public IP address for a short time window.
You'll also want a system like Fail2ban, which can recognize failed logins and ban the IP that attempted them (by default, Fail2ban blocks an IP for ten minutes after ten failed logins within ten minutes, but this is all configurable). This will help protect you against attacks coming from systems that share your public IP address (e.g. by being behind the same NAT or by claiming your IP when you disconnect).
I'd still recommend using Tor for your everyday access and only fail over to the public internet when Tor becomes too slow (which hopefully isn't that often; much of Tor's slow speed comes from being starved for exit nodes, but a hidden service doesn't need an exit node).
*Since Tor runs atop the internet, it still needs some ports exposed to the public internet. You won't be completely invisible and any adversary should be able to easily determine that your server runs Tor. You might be able to do some kind of reverse port-knocking (upon detecting a port scan, drop all packets to and from that IP), but some port scans are hard to detect (and an attacker might start with the port in question).