Maybe it depends on the keyserver. Does that pose privacy issues?
You cannot tell whether the keyserver operator logs access to his keyserver or not. Most common keyserver software is open source and can easily be changed (if logging features aren't built-in anyway).
However, it is not possible to trace which key ID / user interacted with the keyserver (the clients do not authenticate in any fashion), but keyserver operators could log their user's IP addresses.
As for adding a keyserver to a keyring, you have to store his public certificate in your own keyring, and as a public certificate is fully shareable, there is no way to trace each copy of a public certificate.
So the answer is clearly no.
As all this stuff is public and the only thing you do is to store (or not to store) this in a speficic personal directorie, you'll be able to drop them whenever you want.
The only issue may be if you sign them as a trusted third party (authority). In this case you may have to revoke your self signing.