(started as a comment - but getting a bit verbose....)
If its only on your own PC then it is not well protected - if you lose your PC you lose your server access. Protection isn't just about preventing others gaining access, its about ensuring authorized people continue to have access.
If you allow both password and keypair logins then your server is significantly more vulnerable - 'root' is a known username with privileges and hence frequently used in brute force attacks. Mitigations are to require membership of a specific group (doesn't protect 'root' but useful for 'bin', 'dev', 'apache' etc) port knocking and fail2ban.
Maybe it appears in your backups too. Maybe you have RAID to protect against data loss in hardware failures. Maybe you have an effective process for disposing of retired/broken storage. Maybe you have UPS to protect against corruption in power outages. Maybe you have IDS running on the device where the key is stored. Maybe you have IPS running in front of the device. Maybe this device is a locked down appliance only used as a terminal to your server. Maybe you have processes and technology in place to audit all the access on your server. Maybe your server uses strong password hashes and the server backups are managed securely.....
We can't say if you will be "safe" because this is a relative term - and we don't know what your definition of safe is nor the value of the assets being protected.
If it were my server and the gross value was less than a months income, I would be comfortable with using keypair logons along with fail2ban - but I'd never allow ssh logons as root remotely. The only scenarios where I couldn't log in using a different uid and su would be where the system is so broken that the sshd is unlikely to be running.