Is there any downside of configuring password-less SSH authentication?
Although it is very handy and secure, if someone gets access to my system with my account and there is password-less authentication set-up to many servers, isn't it a potential security problem?
There are three different scenarios here:
SSH password authentication. Here, the SSH server accepts a password. If the password's compromised, the system's compromised, regardless of where the attacker is.
Key authentication without security on the key file. Here, the system will only allow access if you present the appropriate key file. Shifts the problem, and gives you something less to remember, but means that you need to connect from or through one of your known hosts.
Key authentication with the keys secured with a passphrase. Here you enter the passphrase at the beginning of the session to unlock the key, and that's used for all subsequent authentication. An attacker needs both the key and your passphrase, but you do need to unlock it every time.
The security of 1 vs. 2 is debatable, they're both fundamentally 1-factor authentication (what you know and what you have, respectively), however, option 1 could theoretically be brute-forced on an internet-facing service. It secures against opportunistic attacks, but would stand up as much against something targeted. 3, on the other hand, is 2-factor, in that you have to have both parts of the equation to do anything. It increases the inconvenience (need to come from known host, and need to enter passphrase), but the security benefits are also significant.
