A HSM does not provide much more than protecting the private key. This is already a strong protection since it means that the private key cannot be simply and unnoticed stolen (and thus misused): If the attacker wants to steal the private key he would not only need to have physical access to the HSM once (which is already a strong protection) but for all transactions since the HSM cannot be cloned.
Additionally HSM are usually protected by a secret (i.e. PIN, passphrase...) which depending on the configuration need to be provided once for all transactions on the initial activation of the HSM (i.e. after plugging it in, after system reboot...) or for each transaction. The latter is more secure but not practical in all cases, i.e. it might be practical if the HSM is used in a certificate authority to sign new certificates but it is probably impractical if the HSM is used to protect the private key of a web server.
... Could some application in the OS submit a bitcoin transaction to the HSM to be signed...
The HSM itself has no idea what it is signing. The intelligence if some data should be signed in the first place (i.e. if some cryptocurrency transaction is valid) need to be provided by the user of the HSM. In most cases the logic to implement such functionality is too complex to be implemented on the fairly small computer the HSM (or smart card) is - if it is clear how such automatic checks should be done in the first place.
If you use for example a HSM to automatically (no PIN required) sign cryptocurreny transactions and you use it on a complex system where it is likely that bugs or misconfiguration allow access to attackers, then you should probably rethink your system architecture. It might be more useful to use the HSM instead within a separate and minimal system which only implements the necessary checks what should be signed and provides a minimal and secure API to the more complex system.
Maybe in that case use a TPM? with or without the HSM? but dont TPMs have vulnerabilities?
A TPM is only special kind of HSM with about the same limitations. TPM can have vulnerabilities as much as HSM or smart cards could have. But all these together are very minimal systems which are more easy to audit for vulnerabilities than larger systems. This does not mean though that bugs never happen, it is only much less likely.