The x86 architecture (both in 32 and 64 bits) isn't really designed for a full isolation between (threads|processes|VMs). There are many low-level hardware pieces that —designed for efficiency— may leak some amount of information between them.
This isn't anything new. See for instance the 2005 paper Cache-timing attacks on AES by Daniel J. Bernstein.
This is also what happened with Meltdown and Spectre. Speculative execution allowed one CPU thread to figure out things that supposedly it couldn't.
Note also that these attacks are statistical by nature. It is a big deal that you only had a 0.03% error data when reading kernel data (meltdown) but it doesn't mean that it would succeed reading sensitive information, or that it was in memory at all!
Is it a risk? Yes, it could end up being one. And vendors (hardware manufacturers, Operative Systems, crypto library developers…) try to mitigate them as much as they can. Should you be too concerned about it? Probably not, but it will depend
A) How sensitive your workloads are
B) How evil are the other VMs
C) How much risk you are willing to take
D) The amount of funding you have available.
If you are the NSA, surely you can buy completely separate physical systems, but perhaps not if you are a home user. It's also different that your other VMs are (hopefully) clean than used to automatically run any malicious executable while continuously signing SSL certs with a CA key!