I always had a problem with SSH keys when I was a QSA. Mainly because people basically build them once and then ignore them.
As a penetration tester there are many ways to gain access to systems. The easiest to me was always to gain access to a system admin system with ssh keys, or use a browser or other local exploit (phishing attack) to extract the keys & email or web/submit them out if I couldn't gain access directly to the system. Or just buy old hardware their it staff is selling with annual upgrades; I'm sure they aren't wiped and one of them has a key left on it. There are lots of ways to get access to the keys.
Once I had the ssh keys I then had access to everything. The best accounts to find were admins who were no longer with the company. With no central management then those accounts were most likely still in the system.
Ask yourself this. What are you doing to manage the ssh keys and accounts on systems? If you have local system accounts with local sudo and using ssh keys to access, what do you do when someone changes roles in the company or you hire a new person or someone leaves. Or worse, when you have someone malicious who knows they are leaving who pre-builds accounts and ssh keys before they leave. How do you monitor and control for that?
SSH keys usually leave your systems as dozens or hundreds or however many you have of individual systems.
Yes passwords are a pain, but many admins these days; especially those who are trained and given the right tools build strong passwords.
Also, with ssh keys, even with a passphrase; how are you controlling that? There is no way to centrally enforce passphrase changing, or even a strong passphrase. I've seen lots of ssh key passphrases with the password as "a" or "password"...
If you really want security you should look at a radius based token solution; but either way you should look at having your accounts centrally managed and controlled. winbind or even just ldap can be used for user account verification as well as proper user group authorization so you can properly control access to systems.
It's not an easy answer; but the question you need to ask is are you wanting this for convenience or security?