I've got a VPS running on Ubuntu, accessed through SSH, passwordless and with a public key.
Can Spectre and Meltdown be used to compromise it in order to get access remotely eg by reading the private key stored on the server?
Your user SSH private key is never sent to the server, so there's no risk of that being compromised by either Spectre or Meltdown on the server. Only the public key is present on the server. On the other hand, the server has a private key that is used to identify itself to the client. (This is what gets you messages about server identity when connecting for the first time.) Everything below is about stealing this host key.
Both Spectre and Meltdown require that an attacker have code execution on your host. Since you've specified a VPS, there's 2 layers of threat here: attackers on your same physical host, and attackers within your VPS.
On your physical host, if your VPS provider has patched their hypervisor, then there should not be a risk from the specific techniques documented in Spectre and Meltdown. If unpatched, the owner of another VPS could use Meltdown to read the full memory of the host, and then extract your SSH server private key (host key).
Within your VPS, both Spectre and Meltdown require that the attacker have code execution in your VPS. If a malicious attacker has access to execute code on your VPS, Spectre and Meltdown should not be the thing you are worried about. There is so much else going on, and they are already on your host. (Which is not to say you shouldn't patch; you should, but it's not the top concern I would have at that point.)
Even if an attacker could successfully break into your VPS, they will not have access to your private key. Public/private SSH keys are based on an asymmetric key encryption scheme where the public key can be shared with anyone. If you are using an SSH agent and forwarding an agent socket to your VPS, an attacker who manages to compromise your VPS could access your ssh-agent whilst you are connected (but could not steal your private key).