There are two options.
By default (i. e. if the -O resident parameter isn't set), nothing is stored on the YubiKey. Instead, the YubiKey generates a private key, encrypts it (and some metadata) with the device key and then lets you store the encrypted data as the SSH “private key” file. Older YubiKeys used a different but equivalent approach. To produce the actual private key for SSH authentication, the encrypted file content is loaded into the YubiKey and decrypted with the device key. If you use this option, then an attacker who steals your YubiKey but has no access to the SSH “private key” file cannot authenticate at your server, because the YubiKey itself doesn't contain any data whatsoever about the private key. Of course the SSH “private key” file without the YubiKey is also useless, since it's encrypted with the device key.
The second option (using -O resident) is to actually store the private key on the YubiKey. Then you don't need any SSH “private key” file at all for SSH authentication. Of course that means an attacker who has full access to the YubiKey can use it to authenticate at your server.
In the second case, the only way to prevent usage of a stolen YubiKey is to set a PIN beforehand. In the first case, the fact that the SSH “private key” file is also needed provides some protection. However, in terms of U2F tokens like YubiKey, this is more of a happy accident than a design feature, because in the U2F protocol, the encrypted private key isn't considered private at all. In fact, it is usually stored on the server which authenticates the user. Additionally, if the YubiKey is used for something other than SSH, then an attacker could still abuse those services. So in either case, I would set a PIN.
