If you are the only one accessing the machine, I find it helps to keep your ssh logs clean by picking a non-standard port for ssh.
Yes, this is trivial to bypass if the attacker uses a botnet to do a simple port scan, so adds no security against a serious attacker (though it stops you from being the lowest hanging fruit). Still makes sense to have strong ssh passphrases/keys and other best practices (e.g., disable root logins). But on my VPS, I found it cut down failed login attacks that I would see in the logs from random IP addresses from thousands of attempts a day to zero in the past three months.
Basically just pick a port that isn't being used for other purposes (and I tend to pick ports that aren't used for anything by checking
/etc/services to make sure you aren't using a well-known service -- and I also pick ports in the system port range less than 1024), for example 501. Then just do port forwarding in your router to forward incoming TCP requests on port 501 to port 22 on the local machine.
You then connect with
ssh -p 501 $your_routers_ip from the outside world. If you setup a
.ssh/config file you can put the port in there:
Hostname example.com # or ip address
and then can just do
ssh example and it will automatically go to the right port/right user name (and disable X11/Agent forwarding).