I am trying to understand the internal storage scheme used by HSMs to store the private keys. My question is: are keys stored in plaintext inside an HSM? Are there any variations?
Every HSM will be a unique implementation.
I know that the IBM PCIe Cryptographic Coprocessor stores master keys inside the HSM, which are used to decrypt the keys that are kept encrypted in an external database. The older version used 3DES to encrypt the keys; I don't know if V2 still uses 3DES or if they're using AES now. Tampering with the HSM (or even violating its environmental envelope of temperature and humidity) securely destroys the internal keys and certificates, as well as rendering the HSM itself physically inoperable.
Inexpensive HSMs, such as a PIN pad you'd find at the grocery store, maintain their key state in battery-backed RAM. Note that because PIN pads are out on countertops where they can be stolen or tampered with, they aren't trusted with a "master" key that's the same for all PIN pads. Instead, they follow a key generation protocol called Derived Unique Key Per Transaction (DUKPT). DUKPT specifies that each pad has a unique key injected at a secure facility before being shipped to the stores. The PIN pad's unique key is derived from the system's base master key and the PIN pad's serial number, and DUKPT implements a "future key" generation scheme, which continually rotates forward with every use, destroying the old keys as it goes. Attempting to violate the case will result in the CPU quickly scrambling the RAM and then cutting power to it, wiping the future key state.
I'm sure other HSMs have different variations on the theme, but they all behave in a similarly self-protective fashion.