Why do you need a token at all?
What's wrong with https://list.example.com/unsubscribe?address={email_address} on its own?

To prevent user ennumeration attacks that might reveal if somebody is subscribed, make the server's response more vague (see that link).

I see no concern in "malicious" unsubscribe actions (unless this is some kind of critical system, in which case: maybe you shouldn't be using email? Otherwise, consider 2FA). I think the easiest accidental unsubscribe comes from forwarded emails, in which case the presence of an auth token doesn't actually help (but 2FA does).

(If you're more interested in the cryptographic soundness of that concept, I suggest asking Cryptography Stack Exchange instead.)

First, it's good to spell out the problem that you're trying to solve. Then you can determine if an authentication token is necessary and what it needs for proper security.

To prevent user ennumeration attacks that might reveal if somebody is subscribed, make the server's response more vague (see that link).

I otherwise have no concerns about "malicious" unsubscribe actions (unless this is some kind of critical system, in which case: maybe you shouldn't be using email?). I think the easiest accidental unsubscribe comes from forwarded emails, which an auth token doesn't actually alleviate (consider 2FA instead). There is, of course, no harm in including an auth token anyway.

Another risk is that of list bombing, which an auth token would not help. Consider rate limiting, captchas, and/or 2FA to limit list bombs. While confirmed opt-in (COI) limits list bombs to a single message, most lists bombs are COI, so you need more protections or else your system will be abused and anti-spam services will block your mail.

Private or invitation-only lists/newsletters additionally need moderator approval for signups, which easily inserts itself into a COI process.

In the comments, it was established that this is primarily for COI links in order to prove that a subscription request is indeed desired by the subscribing email address (and approved by moderators when necessary). While I believe the HMAC solution is sound here, I'd personally just generate a random code instead. Store it (and the request timestamp) with the user's info in your database and remove the code when the user is verified. To limit clutter and potential abuse, regularly purge unconfirmed users whose timestamps are too old (48h and 7d seem appropriate times depending on your audience). 

(If you're more interested in the cryptographic soundness of that concept, I suggest asking Cryptography Stack Exchange instead.)