In the modern day, I would recommend logical wiping to a consistent value and verifying, even for an encrypted disk with the key stored only in one place. Given the possibility of bad blocks being silently recovered on a spinning HD, SSDs scheduling data for delete later on, it just seems safer.
In contrast, I'd say that for an un-encrypted disk you should assume that even with multiple logical overwrites that there could be some bits leaking.
If the standard you are looking for is: "provably recoverable is unacceptable," then erasing the only places the key is stored seems fine. If the standard is "probably recoverable is unacceptable," then you're still probably fine with your described plan (my personal judgment). If your standard is "possibly recoverable is unacceptable," then go the extra mile.