It is possible to have an encryption scheme where all the cryptographic routine run on a smartcard. The smartcard contains the encryption and decryption key and the cryptographic operations are run on the smart card, rather than the main OS.
All that the main OS are able to do is send commands to the smartcard and receive encrypted or decrypted data, but it never hold the keys itself. This is rarely used in practice for full disk encryption because of poor performance (all data to be read/written must pass through the smartcard interface twice). For file encryption, many OpenPGP smartcard have this functionality.
The reason this is rarely used is because it's of limited advantage. If an attacker gains full administrative access to the system, the attacker can still encrypt or decrypt anything, even without knowing the private key. In most cases, it's the data that you want to protect, not the key itself; so an attacker being able to encrypt/decrypt arbitrary data is just as bad as leakage of the keys.