There are two different, but related problems here. One is the security of the key; the other is the security of the data decrypted by the key.
An HSM and "a separate server" are essentially the same thing from the point of view of your app server. It's sending the encrypted data "elsewhere" for decryption, but the cleartext data will end up back in your app server. The app server will always need enough protection to protect against an attacker inside it, siphoning off your secrets. So you can ignore the security of the cleartext data in making this choice, because the cleartext problem is the same regardless of which option you choose.
Let's consider HSMs first. It's usually true that protecting the key is more important than protecting any one single piece of data (unless leaking a single piece of data could result in a catastrophic loss); and that's where an HSM is a very good tool for the job. The drawbacks of an HSM approach are that they're quite expensive to purchase, maintain, and operate. They're usually priced on transaction throughput, so the more data you need to decrypt at a time, the more expensive the HSM. Any accounting for the expense also needs to include the complexity needed to interact with it - you'll probably spend a lot of time learning how to use it, how to perform key ceremonies, integrating it into your existing systems, etc. But you'll have the best protected keys money can offer.
On the other hand, if you build your own server to perform the decryption of the data, you'll have to spend time hardening it against attack, issuing certificates to authorized app servers, performing audits, etc. And be aware that an admin on the machine will always have enough access to get to the key. (This is not true of an HSM that requires a number of shares of a master key in order to access the keys, meaning a number of trusted people would have to collude to violate the security.) So building your own server will cost less than buying an HSM, but it isn't cheap; and the key is at more risk than an HSM. It's especially problematic if you don't know how to properly secure it.
Given all that, it becomes a matter of money vs risk. (Welcome to Security!) How critical is it to your organization's continued existence that you secure those keys? What security classification have you given to the data stored in those emails? Will it store data in scope for PCI, HIPAA, SOX, GLBA, or some other regulated industry? Does your data contain industrial secrets so valuable that a foreign government might bring all their resources to hack into it? Does it protect so much money that you would go bankrupt if a hacker siphoned it off? Will stealing from the database put your organization out of business? Next, price out buying an HSM vs building and securing your own decryption server.
If the answers to the above add up to "it's not really worth all this", then you have to consider more affordable means of securing it.
