HSM's are used in systems that have either:

 - A FIPS compliance requirement higher than level 1 or,
 - The company has determined that risk involved requires a higher level of control.

The suggestion for use in CA's are common, typically, when a corp run's there own internal CA and needs to protect the root CA Private Key, and when RAs need to generate, store, and handle asymmetric key pairs.

Other scenarios might be for a companies Privileged Access Security System. These system's use the concept of a "vault" which is unlocked via a key or multiple keys. These keys are usually only required on startup of the vault.

Your options are to:

 - Store them on the server. This is not very secure even when you limit access to the server.
 - Store them on Disk/USB. This requires that disk be mounted every time the system is restarted (trouble if you do regular restarts).
 - Store them on a HSM.

In secure systems, this allows key to be generated without a human needing access to it, stored in a system that is FIPS Level 2+ compliant, and only accessed when a system starts. 

In general, HSM's are not an end-user product, nor a common IT product for key management. They are a purpose built product to meet a higher level of compliance for secure systems.