TL;DR: Self-signed certificates are fine, and even recommended at least in some contexts. Use long validity times to avoid key rollover problems, and if RSA, use at least 2048-bit keys.
SAML 2.0 includes the certificate that should be used for signature validation and as an encryption recipient as a part of the SAML metadata (technically, within the
The entity that uses the metadata is supposed to validate the metadata in a "known good" way. This could, for example, be by only downloading it over a trusted channel (for example through HTTPS with normal certificate validation), or by storing a copy of the metadata locally having obtained and verified it in some other trusted manner.
In practice, metadata tends to also be XML signed, which provides an additional layer of integrity validation above and beyond the protection afforded by for example HTTPS. (This is a good thing particularly given how many implementations appear to require actually manually downloading and storing metadata in places where it could easily be modified, accidentally or maliciously. At least then the XML signature fails to validate, causing the metadata to be rejected rather than potentially incorrect metadata to be used.)
If you are unable to exchange SAML metadata while ensuring the integrity of said metadata, then you have a significant bootstrapping problem, to the point that I'm not even sure it's possible to use SAML in a secure manner. If you can't trust the certificate that is part of the metadata, that implies that you cannot trust anything else in the metadata either, including for example the endpoint URIs.
If you are able to exchange SAML metadata while ensuring its integrity, then by definition it contains a trusted certificate blob, which is intended (subject to restrictions stated in the metadata) to be used to:
- Validate the signature on any incoming SAML data, including authentication requests and authentication responses, from the entity for which the metadata is generated
- Serve as a recipient key for encrypted SAML data to the entity for which the metadata is generated (note that encrypted SAML XML is not supported by all implementations)
As that certificate (from the trusted metadata) is by definition trusted, using a CA-signed certificate provides no significant benefit over using a self-signed certificate. There is thus no reason to use a CA-signed certificate, and some reason not to (for example, on Internet2 it is recommended to use certificates with at least 10 years validity; I'm not aware of any commercial CA providing certificates with that long validity).
To mention but one admittedly somewhat anecdotal example, Microsoft's Active Directory Federation Services (ADFS) appears to be perfectly happy about self-signed certificates in SAML metadata; I have done on the SP side pretty much exactly the same that you have, in testing, with ADFS acting as the IdP.