Things used to authenticate are typically "something you know" (a password), "something you have" (a physical key, an ID), or "something you are" (fingerprint, handprint, voice, etc)
The idea behind two-factor authentication is to use two of those domains rather than one.
No matter how complicated your one-factor password is, if I get it, I'm in. If it requires me to enter a code sent to a cellphone, then it requires possession of that phone (or intercept capability of its SMS messages, which are beyond my personal threat model) AS WELL as that password, it's anywhere from slightly more difficult (for a co-worker, roommate, etc) to practically impossible (somebody > 1 km away). If it requires the password + a token password, then I need to possess the token, same idea, etc...
As a bonus two-factor offers some protection against bad passwords.
As a user, I can't compromise myself by a single mistake (leaking my password, losing my phone or token). As an administrator, I have a much higher degree of confidence that user is who they claim to be.
Replay attacks, btw, can be prevented by single-factor one-time password schemes (S/KEY), which I don't see too frequently these days.
Also, regarding your "only sites I trust" -- just because you trust something doesn't mean you shouldn't protect yourself.