When an application calls an HSM using PKCS#11/KSP/etc. what is actually exchanged between the two? For example, if a CA needs to sign a CRL, is the entire CRL actually sent to the HSM for signing?
Signing a CRL is like signing any other data and involves several steps:
Read the data you want to sign.
Compute a hash from the data you want to sign. This hash has the same length no matter initial data length and different data will result in different hash.
Encrypt the hash with the private key. The result of this operation is the actual signature.
This signature can then be appended to the signed data (added to the CRL to get the final, signed certificate to follow your example) to obtain the signed data.
As you see, the private key is only involved in the last step. It would be a waste of resources to send the whole data to the HSM, so the hash is calculated on the application side and only the hash is sent to the HSM to obtain the signature.
Look at functions that are part of the PKCS#11 API. When you create a CRL, the
C_Sign function is needed, which is passed a hash value.
Most PKCS#11 tokens also provide hash capabilities, sou if you really need it you can also do that on the token instead of in software, but this is very rarely done.