Since it is only something you have, that makes it single factor authentication right?
Factors refer to the three different modes of authentication (although there are more - geographic, time of day, etc, but they're less typically referenced)
The 3 main factors are
1) Something you have
2) Something you are
3) Something you know
You are correct that a smart card is single factor on its own - anyone with it can authenticate to the system.
It becomes multi-factor when it's paired with an element from 2 or 3. If I have a password and a smartcard, I have two factor. If I have a password and a smartcard and a fingerprint scan I have three factor.
As an aside - If I have two smart cards and a password I still have two factor. You can't use two passwords, smart cards, or two fingerprints to make two factor.