You're correct about how the passphrase works. And, given your situation, I'd advise you to not use an SSH agent. You should also be aware that if someone with root access installs a keylogger on your machine, your passphrase may be compromised. To reduce the risk, you can keep the key on e.g. a USB stick, instead of on the local harddrive. For extra security, get an encrypted USB stick such as the Ironkey - they work with Linux as well as with Windows and Mac, and can be set to lock themselves after a certain period of time so that even if you leave it in the machine, it won't be usable without the password.
But in addition to protecting your key, there are some things you can do on the target system to make the key less useful to an attacker.
- Have a separate key for each system you connect from.
- Restrict the key on the target server, so that it can only be used from that one machine. You do this by editing the
authorized_keysfile on the server, by insertingfrom="10.1.2.3"at the start of the line containing this key. (There's more information about how to restrict SSH keys in this ServerFault question.) - Make sure you have logging on the target server, and check if someone actually does log in there. The logs should go to a remote server, as an attacker might be able to alter locally stored logs.
- Disallow root logins and passwordless sudo. Make sure that sudo actions are logged.
In addition to these specific points, I'd also say that one should always consider the possibility that an SSH key can get compromised. The way to deal with it is to not rely completely on the SSH key to protect your system - you also need to look at the complete system, logging, checking for rootkits, noticing if/when system files get modified, etc.