What I normally do myself is configure log on to the root account, configure the server with sudo (adding myself to the sudoers) and then not permit root login anymore through SSH.
I don't see a reason why you should not be able to work like this, but should you have a valid case yourself, using a key and a password is normally good protection (it's two-factor authentication since it's something you know and something you have).
If you disable the root login you also decrease the attack surface of your machine. One other argument generally given is that sudo gives you an amount of accountability, the root account is generally a system account and you can't, if something or someone logs on using root, know who is actually executing those commands. When using sudo you still have that accountability since the person first has to logon onto his own account and then sudo to root. This might not be applicable to you if you have few accounts (and in most cases not a seperate remote log system.