HTTPS is widely adopted, why isn't encrypted e-mail as popular?

Question

I don't have an education in computer science, I've just become interested in information security and encryption lately.

I'm struggling to understand why encrypted web browsing using HTTPS has been so widely adopted but at the same time most emails are unencrypted. From what I understand when using PGP the exchange of the public keys are a bit of a hassle, the recommended method seems to be meet in person or get the key from the person's homepage (which uses HTTPS I guess).

Here's my naive suggestion of another way, I would appreciate you to say where I'm wrong:

• Email companies start to provide the ability for me to upload my public PGP key to their server

• My friends want to send me an email without having my public key beforehand. My friends' email client can get my public key automatically from my email provider, for example fastmail. The downloading of the public key takes place after the "send email"-button is pressed.

• Because the connection to fastmail would be encrypted using TLS, one can be certain that the connection actually goes to fastmail. And one can be certain that fastmail gives my friend the right key that I've uploaded there.

• If I don't care so much, fastmail could generate the whole keypair for me and store both my private key and public key. That way I can still read my email using webmail.

This seems simple, and also much easier when I want to change the key. Just like if I want to change ssh keys I just generate a new pair and put the public part on the server.

So, where have I gone wrong in this idea? Or are there already a solution like this, but people don't care to use it?

Answer 1

The biggest obstacle to your proposal is user adoption and behavior change. Imagine having to explain to everyone what a public key is and how great it is to have. This is just not going to happen.

Instead, email security has moved to the mail server side of things, with multiple goals:

• transport encryption. This is already fairly widely deployed
• sender authentication (for authentication of the sending domain, not the individual user) which is a bit more tedious and relies on considerable knowledge by individual email server admins (as someone who's had to setup SPF/DKIM/DMARC, I can tell you it's not much fun).

Your proposal minus uploading your personal key (instead having it generated automatically) is more or less transport security, but without authentication. The authentication part is the tricky one and is what the mentioned acronyms try to do, albeit tediously.

As a side note: proper end-to-end email encryption would require you to either 1) trust the web-based mail provider with your keys, or 2) use a local client that knows about your private key. The former is undesirable for many, the latter is inconvenient for most people.

Another side note: HTTPS was widely adopted because it is (mostly) invisible to most users, bar the browser warnings. Modern email encryption/authentication is the equivalent of that. But the equivalent of everyone having a key pair for email would be asking people to use client certificates to log into websites. ugh!

Answer 2

It may seem simple, but it's not. It's actually very complicated.

There are a couple moving parts that are difficult to fix:

• user education: don't count on people knowing what a keypair is, how to create one, how to protect their keys.

• forgotten/lost keys: if a TLS Certificate is lost, the owner just requests another one. No traffic is lost. But if a user loses his key, all his previous emails are unreadable. Forever.

• MiTM: if your provider is both storing your emails and your key pair, it can read and alter any email. If they only have your public key, they can MitM your emails by providing your friends with a key they own, and re-encrypting with your key before forwarding to you. Unless you send them the key offband (SMS, email from another server, in person), they will not know your key is not really your key.

Given that even TLS is seamless and people still click on errors and load unsafe sites with forged certificates, and use password as a password, I doubt this would get widespread usage and users would be safe.

Answer 3

It's been touched on in other answers and comments, but I think the fundamental difference between web and e-mail traffic is who the parties involved are.

HTTPS actually does two things:

• It encrypts the communication so that it can't be read by an attacker. This is achieved using a stateful session negotiated directly between the user's browser and the web server. This happens on the same TCP (or QUIC) connection the actual messages will be sent over.
• It authenticates the communication so that it can't be tampered with by an attacker. This is achieved using a hierarchy of trusted authorities, with at one end a static list which every client has to maintain, and the other end a unique certificate that each server has to obtain.

Both of these take advantage of the particular topology of the web: many clients directly connecting to a much smaller number of servers. Intermediaries which need to read the plain-text traffic in order to pass it on are relatively rare.

For e-mail, both of these are problematic:

• For encryption, the actual sender of a message is not generally connecting directly to its recipient, so a stateful session between them cannot exist in the same way. Individual connections where the message is transmitted can be encrypted (and now frequently are) - e.g. from your desktop mail client to GMail, and from GMail to FastMail - but there is no equivalent to the end-to-end TCP connection where HTTPS is negotiated.
• For authentication, the entities that need to be authenticated are the millions of individual users, not a small number of servers. This means we need some hierarchy of trust that can go from every mail client (which is going to choose an authenticated key-pair) to every individual address. Trusting Fastmail to provide keys for every @fastmail.com address doesn't really solve anything - you're back to encrypting the transport of the message, rather than proving anything about who received it. This is further complicated by the authentication you want for e-mail actually being the other way around: to avoid spam and impersonation, you want to authenticate the sender of every message not the recipient.

This all leads as others have said to the current state of affairs:

• Transport-level encryption in POP3, IMAP, and SMTP is common, and generally completely transparent.
• Senders negotiating authenticated encryption to particular recipients is rare outside closed networks.
• Various protocols exist for recipients to authenticate senders (e.g. DKIM etc), complicated by the lack of a direct connection to negotiate over, and the complex ways in which users interact with the network. If you look at addresses ending @gmail.com, it seems simple; but imagine how many clients and services are authorised to send and receive e-mail for addresses ending @apple.com.

Answer 4

The topic is very complex and is difficult to explain in a single answer. I understand that you disclosed your lack of CS education, so here we are to explain.

Transport vs end to end

There is a huge difference between transport encryption and end-to-end encryption. You should not confuse them.

Https is born as transport encryption (transport security layer), so that communication between browser and server stays protected. If you are logging in to your home banking, transport equals end to end because your bank is the other end of communication. If you are logging in to webmail, it is only transport because your provider can read your emails in order to show them.

Email is already (mostly) transport-encrypted

What you might not know is that emails are sent via TLS (the protocol underneath https) already. With the exception of some small-office networks, smallest ISPs, homebrew servers, etc. all emails are transferred encrypted between ISPs. Only them know the contents of the emails.

So the scope of your question might be confusing a bit. To simplify, emails are already transferred with the equivalent of https. You said "https is popular", I say that TLS is popular for emails as well.

The burden of end to end encryption

Https is easy to deploy. Only the server must deploy a certificate, every connection is stateless and forgets everything about the history.

Encrypted end-to-end emails are a huge pain for consumers.

• Need to set up certificates. Not all people have enough expertise
• What if user loses key/device? They lose all email history
• Today you just type username/password, what additional configuration steps will e2e-protected email require? Does my grandma accept to do all sorts of security configuration?
• What about multiple devices? How to handle multiple devices? Outlook + mobile for example. Oh, and webmail when you go roaming

Take an example: Whatsapp. It never had a feature to share the conversation history on multiple devices (Whatsapp desktop version downloads messages from your phone which must be connected). If you lose or format your phone and don't have an unencrypted backup, your history is lost.

Answer 5

There is an important difference between the two.

In Decision Theory, there is the idea of Utility, i.e. the value that someone assigns to the various options in a decision. For an infrastructure network such as a road network, railway, the Internet, or Email, the value for an individual is in the number of potential connections / other individuals that are part of the network, the value for the operator of the network as a whole is the number of connections, which is on the order of the square of the members.

The problem with this is that both for the individual member and the operator, initially the cost is very high, while in turn the value is low. It requires crossing a certain threshold (called the critical mass) until the value actually outweighs the cost. For the operator, this generally means that only big operators can actually afford building such a network. Historically, this has meant government organizations. It also means that it makes no sense to have multiple networks: the bigger the network the higher the value, and once past the critical mass the value grows faster than the cost. These two in combination lead to what is called a natural monopoly where a single operator "wins" and displaces all others without even working towards that goal. The operator becomes a monopolist not through action but simply due to the way the market for this particular good works.

Long story short: for encrypted email, there is no entity willing to invest the cost for the network, and the individuals are not going to invest the cost, because … well … why would they? Why would I go through all the hassle of setting up encrypted email if there is nobody I can send them to?

The situation is very different for HTTPS: here, there is a benefit for every individual server operator. Protecting their users protects their business. The value is on the order of the number of users, whereas the cost is almost a constant (and rather low, almost non-existent with services like Let's Encrypt) with only a slight linear cost in power consumption per user. You don't have to add TLS to a large number of servers in the network to see a benefit, and there needs to be no massive upfront investment. It can be done server-by-server by every individual server operator with low up-front and running cost, and immediate value.

(I am glossing over the necessary certificate infrastructure here, which is again an example of an infrastructure network with a natural monopoly, but it is a much smaller problem, because the participants are essentially only the Certificate Authorities, not "all web users", which would be a completely intractable problem.)

Answer 6

It's key distribution.

I won't go into all the gory deets, but when you connect to an HTTPS site, a few things happen. Your computer exchanges keys with the site and, importantly, validates that the site (e.g. your bank) is indeed your bank. If it didn't do this, something could pretend to be your bank, decode your traffic, read your passwords and send the traffic onto the bank (this is called a Man In the Middle (MITM) attack).

To stop this, when you set up an SSL site you have to obtain a certificate that is vouched for by a trusted party who has verified your ownership of the domain name. This used to be quite difficult and expensive (hundreds of dollars) but because it's only needed for the sites, rather than the end user, it was tolerated. (Recently it's got cheaper and easier, but it's still non trivial).

In order for a similar system to be used for email, end users need to go through a similar verification process. Since users have an expectation that email is free, they have been reluctant to do so.

(The other way is that you have a distributed system rather than a trusted organisation model - this is cheaper, popular as a matter of politics but is awkward in practice).

Answer 7

To answer the main question:

For many HTTPS sites you are not even offered a choice. They redirect you to HTTPS and often use HSTS what prevents you from changing back to HTTP.

When a site does not redirect you, there are always uses who just use HTTP without TLS, because they do not care about security indicators in the address bar. When HTTP (without TLS) was more popular, sites often enforced TLS on login pages, so your passwords are protected, offering you no choice either.

So when the user does not care, the decision is on the side of the website/e-mail admin. Adding TLS is easy and it works on all modern browsers and if you really want to, you can support (very) old browsers by offering HTTP without TLS. As this is mostly transparent to the user as mentioned above, you can decide freely and adding encryption can only improve your rating by (power) users.

For e-mail this looks completely different. When you want to communicate with the user in an encrypted way, the user needs to install mail program or browser addons, create a key, manage the key, make sure the key is available on all devices he is using and possibly enter a passphrase from time to time.

This is a burden for the user and when he does not understand what this is about, it will make him think about it as annoyance. So rolling out encryption is not transparent at all, but requires the user to learn a new workflow and to use a more complicated workflow. For a user who does not know the difference or does not care about this, it is a reason against your service.

Just as HTTPS, there is transport layer security for e-mail and because it is just as transparent to the user as HTTPS is, it is widely deployed. There are some down sides (like fallbacks to unencrypted connections), but in general many mails are transmitted encrypted between the e-mail servers and between the first/last e-mail server and the senders/recipients client.

Answer 8

See https://moxie.org/2022/01/07/web3-first-impressions.html for some interesting thoughts on this problem by security researcher Moxie Marlinspike.

Notably:

A protocol moves much more slowly than a platform. After 30+ years, email is still unencrypted; meanwhile WhatsApp went from unencrypted to full e2ee in a year.

...

This isn’t a funding issue. If something is truly decentralized, it becomes very difficult to change, and often remains stuck in time.

...

A sure recipe for success has been to take a 90’s protocol that was stuck in time, centralize it, and iterate quickly.