I understand that FIPS addresses cryptographic operations.
My question is, is it addressing exactly this?
For example, can an application loading a keystore use a non-FIPS provider?
Any input or relevant references is appreciated
The Federal Information Processing Standards (FIPS) is a series of publications specifying a wide range of standards, ranging from specs for smart cards (PIV) to algorithm specifications, testing requirements and even geographical place names. While in many cases, US federal agencies are required to adhere to the standards, they are not generally mandatory for commercial use.
Depending on which standards you're talking about, and what level of compliance you may be required to meet, the answers will vary.
I think the definition of FIPS might help: Federal Information Processing Standard. The key here is Standard, much like a W3C technical report, ISO standard etc. You are not under any obligation to conform to these standards.
But, there is always a reason for these standards. For one, conforming to them means you ought to be able to interoperate with other conforming software. For example, take C99, the ISO C standard. Any C99 code I write ought to work on any C99-compliant compiler.
In the case of cryptography and information security, I would really advise following these standards. For starters, they define how an algorithm or key store should behave, meaning not only interoperability but usually a reasonable level of security testing too. In other words, you know your environment (that you've just implemented) ought to not contain any weaknesses, library, OS and environment issues non-withstanding.
As an end user, you know your FIPS-compliant key store should give you some assurance of security, as opposed to whatever storage format Joe Bloggs dreamed up. You should find your software has some level of interoperability (for example, different standard, but many programs can read PKCS#11 keys and OpenPGP keys).
So it isn't forbidden per se, just really highly recommended. If you're choosing software, I would favour something that follows FIPS, PKCS or OpenPGP.
Another point to mention here is that FIPS, RFCs etc often define a usable part of what's out there (by way of a standard). Rijndael != AES, for example. The Rijndael cipher supports many more key lengths and block sizes than AES does, but simplifying that to a certain subset means it is simpler to implement, and that is exactly what AES (FIPS 197) does. What would happen if you said "implement Rijndael" is that many developers would not implement certain sizes due to time constraints, pure laziness or whatever. Software using the exact same core algorithm would then otherwise be incompatible.
So as a final answer, no it isn't forbidden, but it is highly recommended. Mostly, it is about interoperability, but there will be some level of assurance ("we're not recommending a broken cipher" (in current FIPS as opposed to retired ones)) too.