Breaking any key from a set by random guessing is indeed 'easier' than breaking a specific key, but not enough to make a difference.
How many known keys there are depends on who does the knowing, but if consider available from public servers on the Internet, several projects have collected more or less all SSL/TLS and/or SSH keys on public Internet servers for various kinds of investigations; see https://www.factorable.net for one good example. This takes some work and a bit of time, but is quite possible. They reported 12 million in 2012; adding a generous allowance for growth and rounding up, call it about 32 million (2^25). This is not large at all for modern database systems and computers, although e.g. storing it on paper cards in an old-style library catalog would be costly.
As per the link in Crypt32's answer, the chance that a randomly generated RSA 4096 key matches a particular target is about 1 in 2^4070; with 2^25 targets, it improves to about 1 in 2^4045. Both of these are astronomically unlikely -- literally so; if you used all the energy in the universe the chance of hitting any target is still much less than the chance of your house being hit by dozens of independent asteroids within one second.
Even for RSA 2048, which is much more common today, you improve from 1/2^1005 to 1/2^980 -- still not enough to matter.
ECC, on the other hand, is much smaller because it isn't subject to math-structure attacks like RSA. For the commonly used ~256-bit curves (secp256r1, secp256k1, X25519/Ed25519) the universe does contain enough energy to break some reasonable number of keys, though the Earth doesn't. And for Bitcoin addresses, which run the secp256k1 key through a (net) 160-bit hash, it is quite possible someone will find a duplicate within not too many millenia, though probably not within your lifetime (or mine).