I suppose I should have mentioned that both computers are running Linux. The main concern is to prevent "crimes of opportunity". I don't want to leave the front door to my network wide open to someone who 1) knows the correct username/ip address/port and 2) knows where the key is / what the password is (keylogger perhaps).

I like the idea of 2FA. I tried before setting up Duo, but with no success. I'll give these a try!

I like the 2FA idea, but this answer gets at my main concerns. Currently my normal user has sudo privileges I use it to administer my entire home network (raspberry pi cluster, multiple computers, etc...) so I really don't want someone to have access to my whole home network through this computer. Thanks!

@Nobody I just checked again and you are right. I can ssh into another user's home directory, but many of the files and directories are protected so I can't read them.