46

I don't have an education in computer science, I've just become interested in information security and encryption lately.

I'm struggling to understand why encrypted web browsing using HTTPS has been so widely adopted but at the same time most emails are unencrypted. From what I understand when using PGP the exchange of the public keys are a bit of a hassle, the recommended method seems to be meet in person or get the key from the person's homepage (which uses HTTPS I guess).

Here's my naive suggestion of another way, I would appreciate you to say where I'm wrong:

  • Email companies start to provide the ability for me to upload my public PGP key to their server

  • My friends want to send me an email without having my public key beforehand. My friends' email client can get my public key automatically from my email provider, for example fastmail. The downloading of the public key takes place after the "send email"-button is pressed.

  • Because the connection to fastmail would be encrypted using TLS, one can be certain that the connection actually goes to fastmail. And one can be certain that fastmail gives my friend the right key that I've uploaded there.

  • If I don't care so much, fastmail could generate the whole keypair for me and store both my private key and public key. That way I can still read my email using webmail.

This seems simple, and also much easier when I want to change the key. Just like if I want to change ssh keys I just generate a new pair and put the public part on the server.

So, where have I gone wrong in this idea? Or are there already a solution like this, but people don't care to use it?

  • 4
    If fastmail has the private key, how is this different than merely encrypting the messages in transit between mail servers with TLS? – Mooing Duck Jun 2 at 21:47
  • 30
    PGP isn't analogous to HTTPS. STARTTLS over SMTP and IMAPS are, and are used., – JCRM Jun 2 at 23:34
  • 4
    @JCRM That depends what you're comparing. The full infrastructure around HTTPS effectively offers end-to-end encryption and one-way authentication - e.g. between a user and their bank. TLS used with SMTP and IMAP can't offer the equivalent for that user sending an email to their bank manager. – IMSoP Jun 3 at 10:13
  • but not when the web form just sends an email @IMSoP – JCRM Jun 3 at 11:09
  • @JCRM Sure, there's plenty of cases where the web can be seen as "just transport", but in general a browser is under much closer user control than a shared SMTP server, so can provide more direct evidence that information has not been intercepted. Even if you treat GMail's back-end as the "client", it's a single client shared among thousands of users, as compared to a local application on local hardware. – IMSoP Jun 3 at 13:15
43

The biggest obstacle to your proposal is user adoption and behavior change. Imagine having to explain to everyone what a public key is and how great it is to have. This is just not going to happen.

Instead, email security has moved to the mail server side of things, with multiple goals:

  • transport encryption. This is already fairly widely deployed
  • sender authentication (for authentication of the sending domain, not the individual user) which is a bit more tedious and relies on considerable knowledge by individual email server admins (as someone who's had to setup SPF/DKIM/DMARC, I can tell you it's not much fun).

Your proposal minus uploading your personal key (instead having it generated automatically) is more or less transport security, but without authentication. The authentication part is the tricky one and is what the mentioned acronyms try to do, albeit tediously.

As a side note: proper end-to-end email encryption would require you to either 1) trust the web-based mail provider with your keys, or 2) use a local client that knows about your private key. The former is undesirable for many, the latter is inconvenient for most people.

Another side note: HTTPS was widely adopted because it is (mostly) invisible to most users, bar the browser warnings. Modern email encryption/authentication is the equivalent of that. But the equivalent of everyone having a key pair for email would be asking people to use client certificates to log into websites. ugh!

| improve this answer | |
  • 6
    WhatsApp forces people to have their own keys per device. I'm not sure your assertion that user adoption is a barrier is true... – schroeder Jun 2 at 14:26
  • 11
    The barrier is the distribution of the key per user across multiple devices, even devices the user might not own. And the ability to email someone that you do not know. These are technical barriers, not behavioural ones. – schroeder Jun 2 at 14:28
  • 36
    @schroeder WhatsApp controls the client, the server, the protocol and implementation, and the infrastructure. Email is descentralized: countless servers, countless clients, SMTP/POP/IMAP, and lots of different implementations... – ThoriumBR Jun 2 at 14:31
  • 5
    @ThoriumBR yep. But the focus of my statement was the user behaviour part. – schroeder Jun 2 at 15:29
  • 3
    @schroeder WhatsApp was always that way, though, right? People signing up for it know what they're getting into. Email started out without the encryption stuff. You'd probably have a harder time convincing someone to change how they use something they know vs convincing them to switch to a different service altogether. – TheWanderer Jun 3 at 3:15
21

It may seem simple, but it's not. It's actually very complicated.

There are a couple moving parts that are difficult to fix:

  • user education: don't count on people knowing what a keypair is, how to create one, how to protect their keys.

  • forgotten/lost keys: if a TLS Certificate is lost, the owner just requests another one. No traffic is lost. But if a user loses his key, all his previous emails are unreadable. Forever.

  • MiTM: if your provider is both storing your emails and your key pair, it can read and alter any email. If they only have your public key, they can MitM your emails by providing your friends with a key they own, and re-encrypting with your key before forwarding to you. Unless you send them the key offband (SMS, email from another server, in person), they will not know your key is not really your key.

Given that even TLS is seamless and people still click on errors and load unsafe sites with forged certificates, and use password as a password, I doubt this would get widespread usage and users would be safe.

| improve this answer | |
  • 4
    honestly, I've been using OpenPGP for about 18 years now or so. I still hate its usability, even integrated via Enigmail into Thunderbird. You simply forget the usability aspect of real-world existing solutions there. In companies with a SMIME monoculture where people are given a configured Outlook, you get 100% encrypted internal email; I seent it! – Marcus Müller Jun 2 at 16:55
  • 9
    I tried making family use it. Others did. I had to give in. Even the most user-friendly GPG frontends simply fail at being consistent enough for actual end-user deployment; if my dad can't judge whether a warning is because a signature is invalid, or because GPG felt like checking for USB tokens today and failed at that, then I might as well not sign my emails. – Marcus Müller Jun 2 at 16:57
  • @MarcusMüller 100% encrypted internal e-mail in a company just means that the company cannot read the work e-mail exchanged by its employees anymore, right? It seems strange that a company could accept this as a good thing; usually the principle is that your work e-mail "belongs to the company" and they must be able to retrieve it if you get hit by a bus. – Federico Poloni Jun 4 at 10:30
  • @FedericoPoloni no, it doesn't mean that – there's even companies that manage the keys centrally – which is questionable from a crypto point of view. But yeah, I know large institutions where when you get hit by a bus, nobody can decrypt your mail. in fact, I work at one. Is that a good solution for people in "interchangeable" roles, eg. a sales team? no. Is that a appropriate for engineers in an institution with dozen to hundred small teams and a high degree of interest in keeping intellectual property? Maybe more so. Also, it solves the "do not transport confidential personal info – Marcus Müller Jun 4 at 12:08
  • over public channels unencrypted" problem: If it's inside, it's always actually safe, and if it's external communication, then they have to exchange keys with you, which is trivial, because all public email addresses have their public keys publicly listed. The "Mister Smith got hit by a bus, and that means we're bancrupt" is a pretty rare scenario. – Marcus Müller Jun 4 at 12:10
13

It's been touched on in other answers and comments, but I think the fundamental difference between web and e-mail traffic is who the parties involved are.

HTTPS actually does two things:

  • It encrypts the communication so that it can't be read by an attacker. This is achieved using a stateful session negotiated directly between the user's browser and the web server. This happens on the same TCP (or QUIC) connection the actual messages will be sent over.
  • It authenticates the communication so that it can't be tampered with by an attacker. This is achieved using a hierarchy of trusted authorities, with at one end a static list which every client has to maintain, and the other end a unique certificate that each server has to obtain.

Both of these take advantage of the particular topology of the web: many clients directly connecting to a much smaller number of servers. Intermediaries which need to read the plain-text traffic in order to pass it on are relatively rare.

For e-mail, both of these are problematic:

  • For encryption, the actual sender of a message is not generally connecting directly to its recipient, so a stateful session between them cannot exist in the same way. Individual connections where the message is transmitted can be encrypted (and now frequently are) - e.g. from your desktop mail client to GMail, and from GMail to FastMail - but there is no equivalent to the end-to-end TCP connection where HTTPS is negotiated.
  • For authentication, the entities that need to be authenticated are the millions of individual users, not a small number of servers. This means we need some hierarchy of trust that can go from every mail client (which is going to choose an authenticated key-pair) to every individual address. Trusting Fastmail to provide keys for every @fastmail.com address doesn't really solve anything - you're back to encrypting the transport of the message, rather than proving anything about who received it. This is further complicated by the authentication you want for e-mail actually being the other way around: to avoid spam and impersonation, you want to authenticate the sender of every message not the recipient.

This all leads as others have said to the current state of affairs:

  • Transport-level encryption in POP3, IMAP, and SMTP is common, and generally completely transparent.
  • Senders negotiating authenticated encryption to particular recipients is rare outside closed networks.
  • Various protocols exist for recipients to authenticate senders (e.g. DKIM etc), complicated by the lack of a direct connection to negotiate over, and the complex ways in which users interact with the network. If you look at addresses ending @gmail.com, it seems simple; but imagine how many clients and services are authorised to send and receive e-mail for addresses ending @apple.com.
| improve this answer | |
  • For email encryption, the "session" is asynchronous. If you are encrypting (with OpenPGP), then you have the recipient's public key, which tells you what algorithms the recipient will accept, and your agent chooses one set. – Mark Wood Jun 3 at 15:21
  • @MarkWood You're skipping a step - how do you have the recipient's public key? In HTTPS, acquiring the certificate is all part of setting up the single session. The closest comparison would be an entire email conversation, starting with exchanging and verifying keys somehow, and proceeding with transmitting messages over a secure channel. It's not that that's impossible with email, it's just fundamentally more complex than securing an already-synchronous point-to-point connection. – IMSoP Jun 3 at 16:58
9

The topic is very complex and is difficult to explain in a single answer. I understand that you disclosed your lack of CS education, so here we are to explain.

Transport vs end to end

There is a huge difference between transport encryption and end-to-end encryption. You should not confuse them.

Https is born as transport encryption (transport security layer), so that communication between browser and server stays protected. If you are logging in to your home banking, transport equals end to end because your bank is the other end of communication. If you are logging in to webmail, it is only transport because your provider can read your emails in order to show them.

Email is already (mostly) transport-encrypted

What you might not know is that emails are sent via TLS (the protocol underneath https) already. With the exception of some small-office networks, smallest ISPs, homebrew servers, etc. all emails are transferred encrypted between ISPs. Only them know the contents of the emails.

So the scope of your question might be confusing a bit. To simplify, emails are already transferred with the equivalent of https. You said "https is popular", I say that TLS is popular for emails as well.

The burden of end to end encryption

Https is easy to deploy. Only the server must deploy a certificate, every connection is stateless and forgets everything about the history.

Encrypted end-to-end emails are a huge pain for consumers.

  • Need to set up certificates. Not all people have enough expertise
  • What if user loses key/device? They lose all email history
  • Today you just type username/password, what additional configuration steps will e2e-protected email require? Does my grandma accept to do all sorts of security configuration?
  • What about multiple devices? How to handle multiple devices? Outlook + mobile for example. Oh, and webmail when you go roaming

Take an example: Whatsapp. It never had a feature to share the conversation history on multiple devices (Whatsapp desktop version downloads messages from your phone which must be connected). If you lose or format your phone and don't have an unencrypted backup, your history is lost.

| improve this answer | |
  • Interesting, I didn't know that email traffic are already mostly transport-encrypted. In the case of lost keys I totally agree that that is scary. To get around that you sort of have to trust your email service provider with your private key i guess, so you don't accidentally loose the key yourself. – anders Jun 4 at 9:09
  • An interested reading here. Gmail will tell you if the email you are about to send will be transported securely. Probably they have record of past deliveries – usr-local-ΕΨΗΕΛΩΝ Jun 4 at 11:13
  • Although mail does mostly use TLS for transport when it can, is it true that many mail servers do not actually verify the peer's TLS certificate because they cannot afford dropping mail if the recipient's server did not bother to set up a valid one? – user1686 Jun 5 at 9:00
  • 1
    "home backing" is "home banking", isn't it? – sharptooth Jun 5 at 14:43
  • @user1686: For backwards compatibility email allows using weak transport encryption and falls back to unencrypted connections. Email also lacks the strong, enforced PKI HTTPS has, for the same reason. Luckily, there's DANE (RFC 6698) allowing the sender to explicitly state that TLS should be enforced and only matching certificates used. With opportunistic DANE configuration the other party can maintain the backwards compatibility but also communicate securely with the domains utilizing DANE. – Esa Jokinen Jun 6 at 5:46
4

There is an important difference between the two.

In Decision Theory, there is the idea of Utility, i.e. the value that someone assigns to the various options in a decision. For an infrastructure network such as a road network, railway, the Internet, or Email, the value for an individual is in the number of potential connections / other individuals that are part of the network, the value for the operator of the network as a whole is the number of connections, which is on the order of the square of the members.

The problem with this is that both for the individual member and the operator, initially the cost is very high, while in turn the value is low. It requires crossing a certain threshold (called the critical mass) until the value actually outweighs the cost. For the operator, this generally means that only big operators can actually afford building such a network. Historically, this has meant government organizations. It also means that it makes no sense to have multiple networks: the bigger the network the higher the value, and once past the critical mass the value grows faster than the cost. These two in combination lead to what is called a natural monopoly where a single operator "wins" and displaces all others without even working towards that goal. The operator becomes a monopolist not through action but simply due to the way the market for this particular good works.

Long story short: for encrypted email, there is no entity willing to invest the cost for the network, and the individuals are not going to invest the cost, because … well … why would they? Why would I go through all the hassle of setting up encrypted email if there is nobody I can send them to?

The situation is very different for HTTPS: here, there is a benefit for every individual server operator. Protecting their users protects their business. The value is on the order of the number of users, whereas the cost is almost a constant (and rather low, almost non-existent with services like Let's Encrypt) with only a slight linear cost in power consumption per user. You don't have to add TLS to a large number of servers in the network to see a benefit, and there needs to be no massive upfront investment. It can be done server-by-server by every individual server operator with low up-front and running cost, and immediate value.

(I am glossing over the necessary certificate infrastructure here, which is again an example of an infrastructure network with a natural monopoly, but it is a much smaller problem, because the participants are essentially only the Certificate Authorities, not "all web users", which would be a completely intractable problem.)

| improve this answer | |
  • 1
    Except the UK Government have and maintain a secure email system (end-to-end encrypted) called GSI (government secure intranet) which has interoperability with third-party services like Egress (www.egress.com) and CJSM (www.cjsm.net) – HomoTechsual Jun 3 at 12:37
  • 1
    What is the percentage of UK email users that actually use this system? What is the percentage of emails (looking only at emails sent from a sender located in the UK to a recipient located in the UK) that use this system? If I were to randomly select two friends both living in the UK, what would be the probability that they use this system to communicate with each other? – Jörg W Mittag Jun 3 at 12:53
  • 1
    Incredibly low, but there are end-to-end encrypted email services in use - I’m not suggesting that it’s in wide-spread use, merely that there are entities that have invested the time/money into the infrastructure albeit for specific niches, generally aimed at non-profits, governments and business - but which work without as much complexity as PGP. I’m sure other counties have similarly rolled their own (and afaik Egress have a large US and European customer base as well) – HomoTechsual Jun 3 at 12:57
  • 2
    I don't buy the argument that the value is lower. Remember when Goldman Sachs accidentally sent a confidential email to the wrong person? The issue seems to be that there are far more players who'd have to co-operate. For a single website to use SSL, every browser and that one website, plus one CA have to co-operate. For email, the sky is the limit. – jpaugh Jun 3 at 19:37
2

It's key distribution.

I won't go into all the gory deets, but when you connect to an HTTPS site, a few things happen. Your computer exchanges keys with the site and, importantly, validates that the site (e.g. your bank) is indeed your bank. If it didn't do this, something could pretend to be your bank, decode your traffic, read your passwords and send the traffic onto the bank (this is called a Man In the Middle (MITM) attack).

To stop this, when you set up an SSL site you have to obtain a certificate that is vouched for by a trusted party who has verified your ownership of the domain name. This used to be quite difficult and expensive (hundreds of dollars) but because it's only needed for the sites, rather than the end user, it was tolerated. (Recently it's got cheaper and easier, but it's still non trivial).

In order for a similar system to be used for email, end users need to go through a similar verification process. Since users have an expectation that email is free, they have been reluctant to do so.

(The other way is that you have a distributed system rather than a trusted organisation model - this is cheaper, popular as a matter of politics but is awkward in practice).

| improve this answer | |
  • Actually, for many common scenarios, obtaining a certificate is now entirely trivial - tick a box in a control panel, and a free certificate is issued, verified automatically, and with the same level of browser trust as the ones selling for tens of dollars. The problem with doing this for email isn't that the user would have to pay, it's that we'd need to invent a new system of "email address ownership" that the issuer was proving. – IMSoP Jun 3 at 6:43
  • 1
    Getting an SSL certificate is now absolutely trivial (see LetsEncrypt). To add a cert for a new domain literally all I have to do is add the new domain to the bottom of a text file then run a script. – Asteroids With Wings Jun 3 at 14:58
  • IMSoP touches lightly something very important: what does a certificate certify? A free cert. certifies that someone was able to access an email sent to a particular address, which isn't saying much. I do not understand why banks don't see this as a nice value-add. My bank's business is securing my money, they know who I am, and they have an office near me. Their CPS can say exactly what they checked before certifying my request. They should be begging me to submit a signing request. – Mark Wood Jun 3 at 15:30
  • @MarkWood A common misconception is that a paid certificate ever verified more than that; "EV" certificates added a check that you had some paperwork with a company name on, but that name wasn't necessarily what customers knew you as anyway. I'm not really sure what you're suggesting banks ask for, but I suspect the answer to why not is that there would be far more con artists who understood it than customers. – IMSoP Jun 3 at 17:09
  • @IMSoP I think he's suggesting that banks offer digital certifications of identity to individuals, which could then be used to generate and/or sign public/private keypairs in that individual's name. – jpaugh Jun 3 at 19:30
0

To answer the main question:

For many HTTPS sites you are not even offered a choice. They redirect you to HTTPS and often use HSTS what prevents you from changing back to HTTP.

When a site does not redirect you, there are always uses who just use HTTP without TLS, because they do not care about security indicators in the address bar. When HTTP (without TLS) was more popular, sites often enforced TLS on login pages, so your passwords are protected, offering you no choice either.

So when the user does not care, the decision is on the side of the website/e-mail admin. Adding TLS is easy and it works on all modern browsers and if you really want to, you can support (very) old browsers by offering HTTP without TLS. As this is mostly transparent to the user as mentioned above, you can decide freely and adding encryption can only improve your rating by (power) users.

For e-mail this looks completely different. When you want to communicate with the user in an encrypted way, the user needs to install mail program or browser addons, create a key, manage the key, make sure the key is available on all devices he is using and possibly enter a passphrase from time to time.

This is a burden for the user and when he does not understand what this is about, it will make him think about it as annoyance. So rolling out encryption is not transparent at all, but requires the user to learn a new workflow and to use a more complicated workflow. For a user who does not know the difference or does not care about this, it is a reason against your service.

Just as HTTPS, there is transport layer security for e-mail and because it is just as transparent to the user as HTTPS is, it is widely deployed. There are some down sides (like fallbacks to unencrypted connections), but in general many mails are transmitted encrypted between the e-mail servers and between the first/last e-mail server and the senders/recipients client.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.