I'm not a super expert on CVSS calculation, but I'll post an answer and see if I get corrections in comments.
I think you evaluate the impact on the affected component component itself, regardless of what else is connected to / relying on it (that's what the "Scope" CVSS metric is for).
From the CVSSv3.1 spec:
The Impact metrics capture the effects of a successfully exploited vulnerability on the component that suffers the worst outcome that is most directly and predictably associated with the attack. Analysts should constrain impacts to a reasonable, final outcome which they are confident an attacker is able to achieve.
Let's take a look at the Impact Metric definitions according to the NVD CVSSv3 Calculator.
Confidentiality Impact (C)
This metric measures the impact to the confidentiality of the information resources managed by a software component due to a successfully exploited vulnerability. Confidentiality refers to limiting information access and disclosure to only authorized users, as well as preventing access by, or disclosure to, unauthorized ones.
If this vulnerability allows you to gain unauthorized information from the identity provider component? Then this is certainly yes.
If not, then it really comes down to whether it's a "predictable reasonable final outcome" that Service Provider (SP) / Relying Party (RP) is going to have a confidentiality issue due to a failure of the IdP's authentication.
Kinda unclear to me also.
Integrity Impact (I)
This metric measures the impact to integrity of a successfully exploited vulnerability. Integrity refers to the trustworthiness and veracity of information.
If this vulnerability lets you bypass the authentication mechanism, then information from the identity provider is no longer trustworthy. Almost certainly High (I:H).
Availability Impact (A)
This metric measures the impact to the availability of the impacted component resulting from a successfully exploited vulnerability. While the Confidentiality and Integrity impact metrics apply to the loss of confidentiality or integrity of data (e.g., information, files) used by the impacted component, this metric refers to the loss of availability of the impacted component itself, such as a networked service (e.g., web, database, email). Since availability refers to the accessibility of information resources, attacks that consume network bandwidth, processor cycles, or disk space all impact the availability of an impacted component.
Can this vulnerability be used to crash the identity provider, or otherwise cause it to be unavailable for legitimate users? Will it predictably cause availability issues in the SPs / RPs that rely on it?
In the end, I'm not sure that I have answered your question at all haha.