I'm evaluating smart card readers to store a number of private keys, but don't know where to start.
What criteria is important regarding smart cards and preventing the extraction or unauthorized used of the private key?
There is a security standard for smartcards under the Common Criteria scheme: the Smart Card Protection Profile. A protection profile defines the security properties that are expected from a device or system.
The smart card PP is defined for EAL4+. To put it succintly, the EAL defines what aspects of the product's design are evaluated and to what extent (design documents, code review, developer and independent testing, maintenance practice, delivery, etc.). A CC certification also involves penetration testing, and the + in smartcards' EAL4+ is an extra amount of attacker potential in the penetration testing.
You can find a list of certified products on the Common Criteria website as well as on various national websites (certifications up to EAL4 are recognized internationally). Products in that list might be evaluated against a different protection profile that covers a different target: you'll find smartcard components there, and smartcard-based devices such as passports, health cards, cards with a defined software interface such as JavaCard, etc.
Unless you're a government or bank purchaser, you don't have to get a certified product. Certification has a cost, which is reflected in the device's price. Certification isn't a silver bullet: it only says “this looks good”, not “this is absolutely guaranteed to be secure”. In principle, a CC certification gives you independent assurance, because it is conducted by a government-approved third party.
If you consider buying a certified product, be sure to read the fine print. Certifications reports usually come with assumptions that must be met for the certification to be valid; check that these assumptions make sense for you. Also, certification is often a way to shift liability: by having their product certified, the vendor demonstrates that they have used best practices, so you are unlikely to be able to claim damages if their product turns out to be vulnerable.
Even if you don't get a certified product, the protection profile contains useful information: it provides a definition of security objectives, a threat analysis and other elements of security analysis.
Keep in mind that a security infrastructure is only as strong as its weakest component. A super-secure smartcard won't do you any good if it's stolen and the PIN is the user's birth year, or if using the card is so much hassle that people set up ways to bypass it.