You don't need access to your YubiKey to encrypt to the secret key stored on your YubiKey. You only need the public key, which need not be stored on (or accessed from) the YubiKey to be used securely.
I have a Yubikey 5 Series and would like to use it to encrypt a file, so that a physical presence of my Yubikey would be required to decrypt it.
You don't need your YubiKey to encrypt a file to the secret key stored on your YubiKey. To encrypt, you only need the public part of the key pair, which is not generally considered sensitive and can safely be stored on disk or retrieved on demand from a reliable key server. The only real security considerations are that:
- You must ensure you are encrypting to the right recipient using the corrrect public key to avoid man-in-the-middle attacks or possible decryption by an incorrect recipient. Typically, verifying the public key's fingerprint (which is not generally considered sensitive data either) is sufficient to ensure you're encrypting to the right recipient such as your stored-on-YubiKey private key.
- In the general case, encrypting to a given recipient can expose metadata such as the intended recipient, especially if the public key or its fingerprint is widely known, published publicly, or stored on an older key server that is vulnerable to enumeration attacks. NB: You did not define this use case as a threat, but I mention it as some people forget about metadata analysis in their threat modeling.
If you use any version of OpenPGP (e.g. GnuPG) you simply need the public key to encrypt to the recipient. Given any reasonable implementation of OpenPGP, you don't need the YubiKey to perform the encryption. By definition, while the public key can can derived from the secret key material, you don't need access to the secret key stored on the YubiKey in order to encrypt data that will require the YubiKey to decrypt.
Assuming you don't care about the metadata, and assuming your secret key material resides only on the YubiKey itself, then there's no cryptographic reason to avoid encrypting to the YubiKey using its public key. That's likely a better option than most other approaches if you're using public key cryptography in the first place.
The use case you've described is fundamental to the security and value proposition of public key cryptography, so I think you should carefully reconsider why you think you need the physical YubiKey to encrypt. You only need the stored private key material to sign or decrypt, not to encrypt!
After all, anyone else sending data to you wouldn't have access to your YubiKey either. They would only have your public key, hopefully with your public key fingerprint verified securely out-of-band, so it's difficult to understand the specific threat you perceive or are trying to mitigate here.