Whilst the current CA system works very well for a lot of people, it does put a lot of power into individual CAs' hands, and makes a CA hack potentially devastating for customers and business. What alternatives are there to certificate authorities, and what are the challenges in switching to such systems? I've seen proposals for a variety of distributed P2P-like systems, but nothing has ever come of them.
"Alternatives" depend on whether you want something which may work in the future, or something which works right now, with existing browsers.
Right now, Web browsers expect that servers send X.509 certificates, and then they validate them against the set of root CA integrated in the browser (or operating system). This means that if you want something that works without any extra software installation, then all you can do is alter the set of root CA (your trust anchors). You can:
- Remove some of the CA (it has been said that you can remove most of them without really impacting your browsing experience, because not all trusted CA are equally active).
- Add new trust anchors. Note that a trust anchor is "a name-and-public-key that you trust a priori". Trust anchors are usually encoded as certificates (because the format is convenient for that), and traditionally are self-signed (mostly because in a certificate, there is a field for a "signature" which cannot be left empty). But you can trust certificates which are not self-signed; you can even trust end-entity certificates (i.e. you trust the specific certificate of a given SSL server: this is called direct trust). With direct trust, you manually manage what you trust with very fine granularity. Note that, with direct trust, you lose the benefits of revocation (a directly trusted certificate cannot be untrusted except by a manual intervention on your part, whereas revocation is about propagating mistrust information automatically).
(One can note that the direct trust model is exactly what SSH follows, and it works well in the usage context of SSH.)
In the future, there could be other models. There are several proposals, some being backed by existing software add-ons. To my knowledge, none has reached the "generally usable" level yet. There are, for instance:
DNSSEC and DANE: DNSSEC is about mapping a PKI on the DNS system, primarily so as to authenticate the data returned by the DNS itself, but this can be leverage to distribute SSL server keys (what DANE is about). This is still a hierarchical PKI, with a limited set of trust anchors; the difference with the existing set of root CA is mostly a change of players: the new set of gatekeepers would be smaller, and only partially overlaps with existing root CA. Also, DNSSEC/DANE binds the process of authenticating Web servers to the DNS infrastructure and actors, and it is not clear whether this is good or bad.
A Web of Trust like OpenPGP does. The WoT model relies on graph superconnectivity. To keep things simple, with a WoT everybody is a CA, and each actor is its own and unique root CA; to cope with gullible or downright malicious "CA", WoT users (i.e. Web browsers) accept a target certificate as valid only if they can verify it through many paths which go through distinct CA and all concur to posit the target server key. The decentralized nature of WoT is very popular among people who distrust governments and anything which looks like an administration. However, WoT security is very dependent on critical mass: it will not give you much until sufficiently many people collaborate. See the Monkeysphere project for an implementation (with browser add-ons)(I don't know much about the project quality, though).
Convergence is a descendant of the Perspectives project. If you get down to it, Convergence relies on Hope and Laziness. What Convergence assumes is that wannabe attackers will find it too hard / expensive / tiresome to actively impersonate a given server with a fake certificate for both a long time and in the view of many people. Namely, practical Man-in-the-Middle attacks are either close to the victim (e.g. users of a given WiFi hotspot), in which case they impact only very few people, or close to the server, which requires something wide-scale (like heavy DNS poisoning) which will be quite conspicuous, hence will not last long. Under this model, Convergence is a nice, relatively inexpensive solution, with only minimal centralization (it still requires notaries, and de facto centralization can be expected, but not as thorough as the current set of root CA).
An important point is that the current system "works". Actual attacks which target the PKI are very rare (much publicized, but still rare). The actual system with roots CA is sufficiently hard to break that attackers find it easier to bypass it through some other ways (foremost of which being abusing the credulity of human users). As such, there is little incentive for replacing it with another system which, at best, will do the same. So, while the alternate proposals have some merit (mostly in financial and political terms), I deem it relatively improbable that any of them dislodges the existing root CA in the near future, for "the Web" as we know it.
For closed architectures which are not the Web, and where you control both client and server, you can use whatever system you want, but closed architecture are precisely the situation where hierarchical PKI work very well.
One alternative is DANE/TLSA, which is a RFC standard that allows self signed keys in DNS.
One of the challenges is that this is very new and most client software doesn't yet support it. (Chrome being a new addition) In addition DNS is subject to various attacks, especially on WiFi networks and the like.
One way to remedy the DNS attack is to use DNSSec, and to ensure the client software will only use DNSSec for domains that offer it. If this latter feature doesn't exist then that would open users up to an SSL-strip type of vulnerability.
SSL also supports Secure Remote Password (SRP). This is where you authenticate using a password instead of a certificate. Too bad Browsers don't support it.
There is also Convergence which is a drop in replacement for the CA/PKI system. In theory you could use a self-signed certificate and still defend your self against MITM attacks. It also provides protection against rogue CAs.
It's worth pointing out that the chain-of-trust concept isn't itself any more or less vulnerable than web-of-trust, shared-secret or any other mechanism if implemented at the same scale as our current PKI.
The primary problem is the proliferation of trusted CAs as well as the lack of differentiation between CAs for trust purposes. For application-specific use-cases (i.e. not web browser), you can work around this and create a very solid and very trustworthy private PKI. Since SSL/TLS implementations allow you to specify your own trust roots, there's no reason why you have to included the popular CA lists provided by Mozilla and Microsoft. In fact, if you're not using a browser, then there's little reason why you should include the global CA list and plenty of reasons why you shouldn't. If you're the only CA, then your attack surface is significantly limited (as Microsoft recently learned).
As for the browser PKI, a more ideal solution would be to create segmented TLDs by trust category (e.g.
*.bank for financial institutions, etc.) and most importantly limit CA authority to only a given TLD. That way the Hong Kong Post Office can no longer sign for Paypal or Google.
But since the current technology is effectively set in stone by popular use, the chance of this or any other alternate security infrastructure becoming mainstream is effectively zero.