It seems that they are mutually exclusive, as disabling one gives me the other, and vice versa. Two-factor auth for my ssh servers sounds really nice, so is there any way to accomplish this?
With recent Fedora and RHEL 6 releases, you can use
Update: Now on RHEL / CentOS 7, and any system with a recent version of OpenSSH, you can use:
It's also possible to use the Match directive to exclude IPs or Users.
You can have both public-key and password authentication on the same server. If public-key authentication fails, it will go to password authentication.
As to requiring both, that's seems silly and counterproductive, and checking
Your ssh private key should have a secure passphrase. So if an attacker obtains your private key, they still can't do anything without first obtaining your passphrase. If they've compromised that passphrase (most likely with a keylogger; or from brute forcing an extremely weak passphrase) they can trivially also grab/brute force any memorized password.
If you really want, you could possibly setup something with say ForceCommand (e.g., only allow public-key authentication and then direct user to a shell that prompts for a password). I don't recommend this.
A better alternative if you want to limit exposure, is to have a firewall setup to limit IPs that can reach the ssh port; possibly with an additional VPN running on a server somewhere if you may need to tunnel from another computer at some point. You could also use something like knockd to open a hole in a firewall after a particular port-knocking pattern, though recognize that anyone eavesdropping on traffic could replay the knocking pattern to open up a port.
I looked into this a little more and came up with the following.
You could use PAM for two factor authenticaion, but in doing so you won't be using SSH keys you will be using a different two factors.
For example, You could use google with their two factor authentication and use pam to authenticate, as described at
(cross posting SO answer with updated solution to these days)
If you read through the manual page for
This method should work all the current Linux systems with recent openssh (openssh-6, openssh-7).
The only exception I know about is RHEL 6 (openssh-5.3), which requires setting different option with same values (as described in the other answer):