A HSM is like a big smart card. It will store cryptographic keys and perform cryptographic operations on behalf of some external system; however, it is designed to never allow extraction of the private keys that it contains, even if the attacker has physical access to the machine.
The actual security gain of a HSM is not as big as is usually assumed. An attacker hacking into the external system which is allowed to ask for cryptographic operations from the HSM can then just do the same. If the HSM stores a signature key, then the attacker will be able to sign at will. However, since the HSM will not allow extraction of the private key, it will prevent the attacker from retreating to his den with the key. This implies that the attacker will be able to use the private key only as long as he maintains control of the external system; or maybe the attacker physically grabbed the HSM and ran away with it, but, there again, it won't be discreet. In a way, HSM are there to defeat some kinds of advanced persistent threats: if a key compromise occurs, it will end soon, or else you will know it.
Some HSM are programmable, and you can push some of the application logic into the HSM, under protection of the hardware shielding. This can conceptually allow for much better security, but it requires some specific development which is not easy (the innards of a HSM are not a typical Unix-like operating system with all its features; it is a lot more barebone).