Using a Yubikey (or any other FIDO2/WebAuthN token) as a single factor is an option, but you certainly don't have to use it that way. There's literally nothing you can log into using only my Yubikey; it's the second factor I use on a ton of stuff (password manager, VPN, GitHub and Google and a bunch of other web sites / SSO providers, etc.). To get into them, you need my Yubikey (or a recovery key) AND my password.
In other words, you're really doing an apples-to-oranges comparison here. Nobody (that I've seen) uses an authenticator app as a single factor. It's a popular choice for a second factor, but by virtue of being a second factor, it's completely useless without the first factor (typically a password) too. I would argue that, in most cases, 2FA using password+TOTP authenticator app (on a separate device) is safer than 1FA WebAuthN using a Yubikey. Besides, if you really want to, you can require a passcode to unlock the private key in the Yubikey; most software doesn't do this, and I suspect some doesn't support it, but the token itself does.
On the other hand, let's do an apples-to-apples comparison, and consider a Yubikey (or other FIDO2 token) vs. Google Authenticator (or other TOTP app) as second factors.
- Phishing protection: HUGE win for the token. If you fall for a phishing page, you'll enter your TOTP code into it same as the legit login page, and the attacker will have access (at least for as long as they can keep the session alive... which is potentially a lot, if they can disable 2FA or enroll a new device using only the password). WebAuthN uses a handshake where the site (as reported by the browser) is one of the inputs; it simply won't work for a phishing site.
- Long-term security / key exposure: Win for the token. You can't (at least not without a lot of rare skill, caustic substances, and an electron microscope) get the key out of a Yubikey. It's not exposed anywhere, even briefly. By comparison, backing up the TOTP key is usually easy if you have access to the app's data - some apps will optionally do it for you, usually to a cloud server - and of course the other party has to have presented you with the key (usually as a QR code) at some point, and also stored it in the server. That's a lot of opportunities for it to leak, much more easily than the token's key.
- Ability to connect it to the user: Win for the token. It's pretty easy, generally speaking, to identify the owner of a stolen phone. Lots of people intentionally display that information, even. For a token, you wouldn't want to put your name on it (much less your email address or similar) any more than you'd put hour street address on your house key or your license plate number on your car key.
- Ability to authenticate the user: Slight win for the app. All mobile OSes support lock functions and encryption that requires a passcode to unlock, and usually several other methods for convenience. Apps can have their own security on top of that. By comparison, Yubikeys (like most hardware tokens) will work for anybody, unless special effort is taken to add a passcode that must be entered before the key unlocks. They do support such passcodes, but not all software which uses them does.
- Revocability: Small win for the token. Lots of sites will let you revoke a single hardware token without revoking others (such as the backup you hopefully have), but you can't revoke a single phone's TOTP app without revoking it on your old phone too (they don't usually allow multiple TOTP enrollments for one user at a time). On the other hand, you'll have to log into all those sites anyhow, so it won't cost you that much more time to rotate TOTP for each of them.
Overall, I think I much prefer the token's security, when comparing second factors.