I don't know the HSM Let's Encrypt is using. But all major vendors have different mechanisms for doing backup and recovery to an encrypted medium.
One vendor-specific example:
If you setup nCipher/Thales HSMs from scratch a so-called "world key" is generated which is stored encrypted on normal disk and with which all keys used for crypto operations within the HSM are encrypted.
The world key itself is encrypted with a key stored distributed on a so-called "admin card set" (Shamir's Secret Sharing). So with the files and a configurable k-of-n subset of the admin card set you can restore the world key to a new blank HSM and thus let the HSM decrypt all the key stores also restored to the HSM. They call it "add HSM to security world".
Of course you have to implement your own organizational and technical security controls around that to implement a really secure backup and recovery process.
Having said this you still have to trust the HSM vendor that there is no back-door in the firmware to extract the keys in an undocumented way.