A second factor is defined as independent of the first factor. That means your system should stay secure, even if one of the factors is compromised (and you are aware of the compromise).
For example, a door badge and a fingerprint are independent of each other, and just having the door badge or the fingerprint is not enough to gain access. This is often called "multi-step authentication" instead of "multi-factor authentication".
Now imagine your scenario: You have a private key, encrypted with a strong passphrase. Are those two factors? No, because the private key can also exist without passphrase. An attacker that compromises the private key can thus log into your system, even without knowing that passphrase. In fact, the server is completely unaware if your private key is protected by a passphrase or not.In fact, the server is completely unaware if your private key is protected by a passphrase or not.
If you want true multi-factor authentication, there are SSH modules that do exactly that. That being said, a private key encrypted with a strong password is often enough.
Note: The original question talks about "an SSH key with a Passphrase to login on a server", which I interpreted as a private key, encrypted with a passphrase.