Unfortunately, this can't be done directly, certainly not if you want the server to never see or be able to derive the user's key even using ephemeral data. As you noted, identity provider / OAuth services can tell you who a user is (and what they're allowed to do), and they can even give you access to secrets (stored with the identity provider or a trusted third party) if the user consents to this, but there's no password (or other user-controlled secret) that is only available to your client with such delegated authentication scenarios.
One option is, as you noted, to request a passphrase after the user authenticates. This isn't necessarily a huge burden - SSO is generally seamless enough that it's not significantly worse that just using the passphrase for both authentication and key derivation - but I understand it being undesirable UX (as for "hacky"... you're not entirely wrong, but sometimes one must do the hacky thing, and it's not that bad IMO). As Steffen noted in a comment, you can store the passphrase (or some value derived from it) locally in the client to at least limit the number of times the passphrase is needed. The client-side data could even be made useless without also having some data that is stored only on the server and released only after authentication, such as generating a random XOR mask for the derived key and storing the mask in one location (e.g. server) and the masked key in the other (e.g. client). Of course, the user would still need to provide the passphrase on any new client they wanted to use (possibly in addition to authenticating through SSO).
Alternatively, if you don't mind the server having ephemeral access to the user key, you could potentially have it stored either with the identity provider directly, or in a third-party service that the user also grants you access to (e.g. cloud storage). This is, to be clear, both unconventional and less secure. In addition to meaning your server would not be unable to get the user key while the user is logged in, it also means that the user would have to trust the identity provider with their key (or, again, at least part of it). You'd also need an identity provider (or usable third-party service) that lets you store arbitrary data for a user, and retrieve it if (and only if) that user is logged into your service.
As a side note, be aware that if you control the client - for example, because it's a web app and you control the JS for it - it's always possible for you to send JS to the client that exposes any secret known on the client to the server, or to anybody else. This "Host-based security" problem effectively renders true zero-trust systems impossible on the web, and difficult everywhere.