Let’s say I use my Yubikey OTP to login to multiple servers, and it one becomes compromised. I go to login the that compromised server, and enter my OTP. Obviously, once that’s password is verified it can’t be reused, but whose to say the server has to verify it? The malware could than go on to replicate itself by attempting to login to another server with that Yubikey OTP and so on and so one throughout more servers. Is this possible, or am I missing something?
Yes, if the compromised server does not validate the OTP for the purpose you expect, it can use it to attempt to connect to other of the services that work with the same Yubikey. This would be a man in the middle attack with the particularity that the "man in the middle" happens to be one of your servers (the compromised server). The Yubikey OTP scheme itself does not protect against such attacks.
This assumes that the compromised server is not a Yubikey validation server. If this is the case, you should assume the attacker can generate OTPs of any sequence number. In other words, the attacker virtually possesses a copy of the Yubikey. This is because validation servers know the secret that is stored on the Yubikey if the key is registered with them.