If the operating system is your enemy (i.e. has potential backdoors) then you are in a bad situation indeed... because a backdoored OS can inspect everything in your application code and memory.
I suppose that your question comes from the unspoken assumption that if the .NET (Windows) implementations of cryptographic algorithms have been maliciously altered (e.g. to leak critical data elements), then maybe those backdoors would not apply to some external code ? Unfortunately, this is not the case. Indeed, the most plausible backdoor, for these matters, is to spike the system PRNG. It is possible to modify a given PRNG so that its output looks random but can be predicting by someone who is privy to the generation details. If done properly, such a backdoor cannot be detected from the outside, and even if he problem is revealed (through reverse-engineering of the PRNG implementation), one can always claim incompetence rather than malice (the Debian maintainers managed to do exactly that).
To give an example, imagine a PRNG which works by encrypting a 16-byte counter with AES, the key being a 16-byte seed obtained through hashing "random events" such as timings of hardware interrupts. Then, suppose that some unfortunate programming bug in copying the hash output into the output results in about 80 bits of the key to be predictable. Attackers who know this flaw can predict the PRNG output by doing an exhaustive search on the 48 remaining bits, which is not free, but highly doable with a few PC. You would say, how can you botch a memory copy that bad ? Surely it would appear quite artificial, and could not be plausibly denied. Yet the Openwall people, of John-the-Ripper fame, achieved such a failure in their bcrypt implementation.
Now, imagine a fully open-source (say, "public domain", although this notion tends to have a precise legal meaning which is much stronger than "we can see the source code") implementation of cryptographic algorithms. That code will use some keys, and such keys will have to be imported or, crucially, generated. Key generation requires usage of some good randomness. In C#/.NET code, where will you look for when you need randomness ? Your code executes in the CLR Virtual Machine which is a very abstract execution model -- meaning that everything happens in a very deterministic way, and your code cannot escape well-established rails. No way to read arbitrary bytes in RAM, for instance; hardware is abstracted away and you cannot access it directly. The bottom-line is that if you want good randomness in .NET, you have to get it from the operating system, i.e. System.Security.Cryptography.RNGCryptoServiceProvider
. And then you are back to using the PRNG that could, potentially, be predicted by your enemy.
(The same point could be made about Java.)
Therefore, while using an external implementation of cryptographic algorithms is possible, it does not guarantee protection against backdoors in the cryptographic layer of the operating system.
On the other hand, there are good usability reasons to use the cryptographic algorithms provided by the OS:
- The OS is in good position to use private keys securely, e.g. keeping them out of virtual memory or even put them in kernel space to avoid disclosure in case of some partial, non-administrator exploits.
- By letting the OS handle the keys, you gain automatic support of smart cards and HSM.
- The native implementation will use native code, which will offer much better performance (e.g. about ten times faster for RSA -- notably because native code can use the 64x64->128 multiplication opcodes of 64-bit x86 CPU, which are not accessible from .NET code).
To sum up, IF you believe your OS to be hostile to you, THEN you will have trouble escaping its evilness -- using "public-domain" implementations of cryptographic algorithms might not suffice for that; but you would lose some very interesting features by doing that. So I would suggest that you trust your OS and let it do the job for you. If that worries you to much, then don't switch to an open-source cryptographic library; switch to an open-source operating system altogether. There are many possibilities, with all the Linux brands, the *BSD (NetBSD, FreeBSD, OpenBSD...), even OpenIndiana (a derivative from Solaris)...