If an SSH key is protected by a passphrase, then it's just encrypted with that passphrase. There isn't an additional authentication component. That is, you can decrypt the key client-side (and store the decrypted key), and the server won't ever know. In fact, many users use "key agents" which allow you to enter your decryption passphrase only once, and the key material is remembered during your workstation login session.
You can, however, require on the SSH server both a passphrase and SSH key, where the passphrase is your server's normal user password. If your SSH key is encrypted (and you're not using a key agent), then this is actually two passwords required at login; one for your key and one for your server. This is probably closer to the spirit of two-factor authentication.
There are many who would say, though, that key material on your computer is still "something you know", since it's infinitely copyable. Instead, "something you have" would be a hardware token that can't be duplicated. It is possible to implement SSH keys using hardware tokens, assuming the token supports RSA keys. This may take some doing to set up, though.