The trade off between how much hassle it is to maintain a two-factor or Public Key authentication and the degree of security each provides depends on how/who is using it. If the usage is among a limited number of server system managers who are familiar with security so that they won't pollute either process by allowing access to their access (PC, laptop, mobile device) to be compromised, then either of these may prove secure.
The various reports, articles and blogs I have read indicate that the most often means of defeating either two-factor authentication or public key encryption is gaining access to insecure client access - a PC left on or without password protection that is used to gain root access (or other areas). Passwords may be stored on the device in a password manager or are automatically put in by a web browser or jotted down on paper in a desk.
Another means of compromise is when passwords and keys are not changed after an employee/associate leaves.
Most brute force, distributed brute force attacks and other outside attacks are effectively thwarted by either two-factor or Public key authentication methods. That is improved if reasonably tough passwords are used and password encrypted public key access is required (a form of two-factor authentication itself).
Other methods may be used, such as using a non-standard SSH port, although there is valid argument against that method. Another method is use of rotating/random non-standard ports or use of Port-knocking in which three or more ports must be sequentially hit/knocked before access is granted to a selected SSH or other ports. The port-knock ports may also be rotated periodically.
The latter methods are not usually justified because it is far more likely that security breaches will occur from within an organization that has implemented two-factor or/and public key authentication methods. If the nature of the work/servers requires the highest levels of security, the additional layers of security may make sense.
I re-watched the movie "The Imitation Game" 2014 last night. It is about the cracking of the Nazi Enigma encryption machine that required determining among some hundreds of millions of possible manual selections. The code was broken by Brit Alen Turing's electromechanical computing machine only after they came upon repeated terms in communications that could be guessed "Heil Hitler!" and other terms in daily weather reports. That goes a long way to describe the security of Linux servers and other secured systems - they are only as secure as the practices of people who use them.