The weak link in most smartcard applications is PIN entry. Extracting private keys directly from the card is nearly impossible. With some acid package destruction and electron microscope work, a skilled team, and enough time, money, and luck you can in theory extract keys but it involves not only physical access but a scenario where the card will be physically destroyed.
However if the host system is compromised and the user's input of the PIN is keylogged and later replayed the smartcard will gladly work just as well for the attacker as it does for the user. If you are concerned about the host system being compromised you should look at direct PIN entry. Using a smartcard reader with a keypad the communication is directly between the keypad and the reader. The PIN never travels to host and thus is unavailable for intercept. While in theory the reader itself could be flawed I have never seen an exploit involving direct PIN entry and I would assume it would probably be hardware specific. There are methods to harden PIN entry without a dedicated keypad using programmable smartcards and OTP instead of PIN only but not sure if that is an alternative you are interested in.
Also keep in mind a lot also depends on the use case. Is the card being used for digital signatures? What is the damage resulting from an attacker obtaining a fraudulent signature vs stealing the actual private key. In some scenarios that is a significant mitigation in other scenarios it is a meaningless distinction. The latter scenario is much tougher to defend against because protecting the keys is easier than protecting unauthorized use of the keys.