My question is this: If a good password is used, and all other security practices held equal, does a server administered by a single person gain much practically by switching to SSH keys?
The problem with this question is that it's built on the premise of people using good passwords—something that people, in actual practice, do not do. You're asking us to draw a "practical" conclusion from an theoretical premise that doesn't hold up in practice.
In both cases, I think the biggest risk is that an attacker could obtain a user's secret and use it to authenticate as that user. With password-based authentication they might:
- Target your users with a phishing attack. As 2016 has shown, this can be very successful.
- Steal your password database and crack the entries to find weak passwords.
- Target your users to steal their passwords on other services, on the hopes that some of them will reuse those passwords on your system.
With key-based authentication (as usually practiced, with the keys stored on the users' PCs with weak or no passphrase), the ways an attacker can obtain your users' keys is by hacking into their computer that stores the private key. This could be achieved by targeting your users with malware, which is another realistic attack.
If you're really serious about SSH authentication security, you have to look not just at using private keys, but also at how to store them securely so that attackers can't easily steal them from your users. Smartcards are the classic solution here; a dedicated hardware device that generates a keypair internally and promises never to reveal the private key; the SSH client must forward the authentication request for the smartcard to sign.
A more recent variant is to use a smartphone app to generate and store the SSH keys on the phone, and an SSH client that forwards auth requests to the phone, so it can authenticate the user without revealing the private key. I have no connection to the company, but I've been using the free edition of Kryptonite recently and though it's a young tool it's shaping up pretty well.