I am a bit unclear as to the workflow, so let me know if I'm off.
I believe the situation is that you are communicating with a service. You make a request to the service and it responds, signing the reply with an HMAC. You then verify that the HMAC is correct to ensure both that the request originated from the service (authenticity) and that it hasn't been modified by an intermediary (integrity).
For an HMAC to be effective, the key must be secret. If the key is not secret, then an attacker can perform modify the message or pretend to be the service such that the recipient cannot tell that tell that they are receiving modified messages or not talking with their desired partner. And if you're willing to give up on both integrity and authenticity, there's no point in using the HMAC at all. So, the HMAC key must be secret to be effective.
Exactly what protections are appropriate to keep the key secret is application dependent. For example, storing it in a database table may lead to it getting exposed if your application has a SQL Injection vulnerability. That seems risky to me if you have an architecture that could be vulnerable to SQLi (I would consider any app that takes user input and uses it as part of a SQL query as potentially vulnerable to SQLi). Storing the key in a disk file could be vulnerable to path traversal attack.
In the end, no system will be perfect. You'll have to tune the amount of security to the sensitivity of your application. One trick is to combine multiple security measures by, for example, storing an encrypted HMAC key in the database and storing the key to decrypt it somewhere else (maybe a file outside of your web root). Then an attacker would need to get both the encrypted HMAC key and its decryption key to execute a successful attack. The further you can separate the HMAC key and its decryption key, the more secure you're app will be (and the less convenient it may become to implement and operate).