Whilst the current CA system works very well for a lot of people, it does put a lot of power into individual CAs' hands, and makes a CA hack potentially devastating for customers and business. What alternatives are there to certificate authorities, and what are the challenges in switching to such systems? I've seen proposals for a variety of distributed P2P-like systems, but nothing has ever come of them.
"Alternatives" depend on whether you want something which may work in the future, or something which works right now, with existing browsers.
Right now, Web browsers expect that servers send X.509 certificates, and then they validate them against the set of root CA integrated in the browser (or operating system). This means that if you want something that works without any extra software installation, then all you can do is alter the set of root CA (your trust anchors). You can:
(One can note that the direct trust model is exactly what SSH follows, and it works well in the usage context of SSH.)
In the future, there could be other models. There are several proposals, some being backed by existing software add-ons. To my knowledge, none has reached the "generally usable" level yet. There are, for instance:
An important point is that the current system "works". Actual attacks which target the PKI are very rare (much publicized, but still rare). The actual system with roots CA is sufficiently hard to break that attackers find it easier to bypass it through some other ways (foremost of which being abusing the credulity of human users). As such, there is little incentive for replacing it with another system which, at best, will do the same. So, while the alternate proposals have some merit (mostly in financial and political terms), I deem it relatively improbable that any of them dislodges the existing root CA in the near future, for "the Web" as we know it.
For closed architectures which are not the Web, and where you control both client and server, you can use whatever system you want, but closed architecture are precisely the situation where hierarchical PKI work very well.
One alternative is DANE/TLSA, which is a RFC standard that allows self signed keys in DNS.
One of the challenges is that this is very new and most client software doesn't yet support it. (Chrome being a new addition) In addition DNS is subject to various attacks, especially on WiFi networks and the like.
One way to remedy the DNS attack is to use DNSSec, and to ensure the client software will only use DNSSec for domains that offer it. If this latter feature doesn't exist then that would open users up to an SSL-strip type of vulnerability.
SSL also supports Secure Remote Password (SRP). This is where you authenticate using a password instead of a certificate. Too bad Browsers don't support it.
There is also Convergence which is a drop in replacement for the CA/PKI system. In theory you could use a self-signed certificate and still defend your self against MITM attacks. It also provides protection against rogue CAs.
It's worth pointing out that the chain-of-trust concept isn't itself any more or less vulnerable than web-of-trust, shared-secret or any other mechanism if implemented at the same scale as our current PKI.
The primary problem is the proliferation of trusted CAs as well as the lack of differentiation between CAs for trust purposes. For application-specific use-cases (i.e. not web browser), you can work around this and create a very solid and very trustworthy private PKI. Since SSL/TLS implementations allow you to specify your own trust roots, there's no reason why you have to included the popular CA lists provided by Mozilla and Microsoft. In fact, if you're not using a browser, then there's little reason why you should include the global CA list and plenty of reasons why you shouldn't. If you're the only CA, then your attack surface is significantly limited (as Microsoft recently learned).
As for the browser PKI, a more ideal solution would be to create segmented TLDs by trust category (e.g.
But since the current technology is effectively set in stone by popular use, the chance of this or any other alternate security infrastructure becoming mainstream is effectively zero.