IdP's primary role is to "authenticate" the identity claimed by the user - and assure the SP that the user's identity has been verified.
For this, (most commonly), the password must be verified by the IdP - and hence it is stored with the IdP.
In a properly implemented system the SP should never be in a position to "access" the password and/or any other mechanism used to verify the identity of the user. So it is never stored with the SP.
How it works:
Typically, the IdP provides the user with a verifiable token that indicates to an SP that an authentication has been performed successfully. The user passes this on to the SP as "proof of successful authentication". Needless to say, the SP doesn't take the user's word for it. So the SP verifies this token with the IdP and only after confirming that the token is valid, proceeds to the next step (usually the authorization step).
In case of compromise of the IdP:
Yes, a compromise on the IdP's side does have higher impact. However, the IdP doesn't need to know all the credentials needed to access the SP's application. So a carefully architected SP application would use at least one attribute of the user; and not use just the token to open up access. The IdP would not aware of that attribute and hence would not be able to access the SP app, masquerading as a user. Hopefully, this is not over-engineered to the level of authentication.
However, I must confess I haven't seen this in practice. Most SPs are thankful just to get the IdP integration working well enough and leave it at that. :)