A self signed certificate is completely fine if you are willing to explicitly trust it. The reason a certificate is signed is that an entity more trusted than the certificate holder is adding authority to the data with a digital signature. As you say in your question, a CA is only as good as the ways in which it is secured - an untrustworthy CA creates untrustworthy certificates.
For the most part, a certificate is required when two parts of a system must communicate. When a computer is simply processing information with not transaction between components, there's no need for any certificate at all. Certificates come into play when you need to:
- store data and retrieve it later
- send data or instructions from one point to another
In all cases, there's a risk that an unknown, untrusted party could exploit either the data or the channel. Thus the certificate provides identification, encryption, or integrity verification (or all of the above). It's the key pair itself that provides integrity and encryption - the signing of a certificate is largely concerned with the identification/non-repudiation security concern.
Self signing creates two basic concerns:
- Who are you? How does the system know that this self signed certificate is OK, but another one isn't.
- Are you still OK? When a self signed certificate is explicitly trusted, how do I know if the private key has been compromised.
Both concerns are usually handled by explicitly trusting the certificate - that tells the system that the certificate is the certificate of the permitted party, and if the private key should ever be compromised, the configuration setting can be changed, removing the trust.
That's manual in most systems, and it will totally work for any cases where you can trust that the administrator will keep on top of any compromises and be able to reconfigure the system in a timely manner. In practice, for a handful of simple systems, that's workable.
When the systems become more certificate dependent or when there are multiple points that rely on the connection to a self-signed certificate, the dependency on human error becomes too great, and most security designers will look for a way to do an automated certificate status check and provide a common entity (a CA) for the creation of all certificates. Then all dependant systems will explicitly trust the CA.
Either way will work. In either case, the security will depend on the security of the private key - loose the private and you've lost control of the identity of the system you're securing. After that, it's mostly a matter of what type of risk you are willing to accept.
