Is there such a thing as a "Black Box" that decrypts Internet traffic?

Question

I have been reading about the Snoopers charter bill that was passed in the UK this week.

It mentions a "Black Box" which is cited here: ‘Black boxes’ to monitor all internet and phone data.

It states it works like so:

When an individual uses a webmail service such as Gmail, for example, the entire webpage is encrypted before it is sent. This makes it impossible for ISPs to distinguish the content of the message. Under the Home Office proposals, once the Gmail is sent, the ISPs would have to route the data via a government-approved “black box” which will decrypt the message, separate the content from the “header data”, and pass the latter back to the ISP for storage.

It is very vague on how it works to "decrypt" the data.

Is there such a thing as a "Black Box" and should I be concerned?

Answer 1

Yes. It's called a Man-in-the-Middle attack. You terminate the SSL session at a mid-point, thereby having the encryption key, then create a new session to the target server, so you have that encryption key too. The data path now goes User->MitM->Server, where each of the arrows is an encrypted connection. Data returned from the server goes Server->MitM->User, again, where each arrow is encrypted, but the other points are not.

There are ways to prevent this from working, but in the case of a government mandated system, it seems likely that these will be specifically avoided - there may be regulations for companies to provide valid certificates for the "black boxes", so that HPKP keeps working, for example. It is unclear whether such rules would apply to companies which don't operate directly in the UK, or whether there would be penalties for attempting to bypass these rules (for example, by the use of VPNs based in other countries).

Edit based on comments: Note that it is technically possible to create such a device, but the problems mostly come from requiring cooperation from a large number of parties. As a government, there are options available which aren't possible for smaller actors. For example, it would be possible (if unlikely) to require that all internet connected devices sold in the UK come pre-configured with a government issued root CA certificate, and to prosecute anyone using a device which does not have this installed. This would be terrible for internet security, but so is the overall concept, so it depends on security experts convincing the government just how bad this idea is.

Answer 2

No, there's no way such a thing could possibly exist, for any significant amount of time.

There's two big hints at this in the article itself:

many technical experts are raising equally serious doubts about its feasibility

A Home Office spokesman said – “We have not issued any hardware or software specifications.

The only way this could work on a properly secured website is if either:

• The user cooperated with the surveillance
• The website cooperated with the government
• All the governments in the world decided to cooperate to surveil everyone

Obviously, there is still a high (but lowering) number of insecure websites. And telephony systems are completely insecure, from a cryptography perspective. However, the quoted example - Gmail - is about as secure as you can get.

In general, considering how much noise Snooper's Charter caused in security circles, even if there is such a "Black Box" being designed, it'll be useless long before it is actually used.

The reasons for this are a bit complex to understand, but read on if you're interested on the gory details

---

As other answers mention, the mechanism that you described resembles the well studied man-in-the-middle attack.

^{diagram by Miraceti}

Let's see how it works in practice. Here, Alice (e.g.: the user) and Bob (e.g.: the website) want to communicate confidentially. They communicate through a communication medium that Mallory controls. Clearly, if the messages are not encrypted, Mallory can read and change any messages.

What happens if Alice and Bob use a properly secured channel, though?

• Mallory can't read the messages at all. This property is called confidentiality, and is usually provided by symmetric encryption.
• Mallory can't change the messages. This property is called integrity, and is usually provided by a message authentication code
• Mallory can, at most, prevent messages from being delivered.

Now comes the tricky part. For all these mechanisms to work, Alice and Bob have to agree on a secret key - something resembling a long randomly generated password. Because Alice and Bob might not have communicated before, this is usually done through asymmetric cryptography.

---

Assume Alice and Bob never communicated before. How can they agree on a secret "password" to use, in a way that Mallory can't possibly learn it? Let's use an analogy with the old postal system:

• Alice sends a letter to Bob saying they want to communicate
• Bob receives the message, and sends Alice a package with a open padlock
• Alice receives the padlock, generates a "password", puts it inside a box, locks the box with the padlock, and sends the box to Bob
• Bob gets the locked box, unlocks it, and reads the "password".

Obviously, Mallory can't open the padlock without Bob's padlock key. What Mallory can do, though, is intercept the padlock when Bob sends it to Alice, and replace it with a padlock of their own.

To prevent this easy attack, there is usually a trusted third party - let's call her Faythe. Faythe is responsible for "photographing" everyone's padlocks, and distributing these (very detailed) photographs. Because Alice trusts Faythe, she can check the padlock she receives against the photograph, and make sure it belongs to Bob.

In the web world, Faythe is called a Certificate Authority (CA), and the photographs are called certificates (technically, signed certificates).

---

Now it becomes clear how the government's plan might work: because they can't force Alice (the user) or Bob (the website) to cooperate (in general), they can try to persuade Faythe (the CA) to "send" fake certificates to Alice.

Researchers in the security community are aware of this scenario, and there's several mechanisms to prevent it, both technical and social:

• Shame CAs that do this. Although this sounds quite basic, it's extremely powerful. CAs are usually commercial entities. Their reputation, is, quite literally, their only asset. Once their reputation is ruined, they're essentially worthless. A CA that forged certificates would become distrusted very quickly. If Alice doesn't trust the CA anymore, the government attack would stop working.

• Sanity check the certificate. In practice, certificates for a given website don't change often, and usually only at well defined intervals. If they change outside of the expected interval, you can assume the certificate is compromised. This technique is called certificate pinning.

• Cross-check the certificate. Since a particular website's certificate remains constant globally, you can detect compromised CAs by cross-checking the certificates users receive across the world. There's several projects doing this, including Google's Certificate Transparency, EFF's SSL Observatory, MonkeySphere, Convergence, Perspectives, and probably others I'm not aware of.

Note that all of these mechanisms have been developed before any government even thought of publicly doing this.

---

Given all this knowledge, and assuming the worse possible case (a "blackbox" that works for a short period of time) - what can you do to protect yourself?

• Raise awareness of the issue. The more people are informed, the better
• Install the HTTPS Everywhere browser extension. It will warn you if any such "blackbox" is ever deployed. You should only do this if you trust the EFF (who is usually well regarded in these matters)

Answer 3

The black box is theoretically possible, but is practically impossible. In order for it to work the CAs would have to cooperate and be willing to provide the Black Box with legitimate certificates for every website that provides email services. Otherwise end users would receive certificate warnings in their browsers which would warn them that a MITM attack is occurring. Furthermore, even if the CAs agreed to do this (and I doubt they would), it would only work for traffic routed for a particular country, so any user could use a VPN which is outside of that country and the Black Box would be circumvented. Some countries have tried to prevent access to foreign VPNs by blacklisting known IPs, but since VPNs tend to move around quickly, the only effective way to prevent their use is by blocking everything except government approved IP blocks, and IMHO no country with true democratic elections would ever be able to get away with that level of censorship.

Answer 4

Conceptually, this is a UK version of the US patriot act. It will require the cooperation of the ISPs - easy that are under UK law - and major mail providers. For that latter part, many users simply use the mail box of their provider, mainly if they use SMTP/IMAP. Here no problem every non encrypted email is unencrypted at the server side and can be easily passed to the BlackBox.

Some users directly use webmails from international (US) companies such as Google. In that case, all unencrypted data can be given to US agencies in charge of legal regulations because of the Patriot Act. In that case, servers are generaly implanted in different countries to balance the load. Two possibilities here: ask the mail server to give all mail received on a server in UK to the BlackBox, or ask the mail company to give a valid key to the backbox to allow it to perform a Man In The Middle Attack.

It this really a new threat for confidentiality? No because the SSL only protect the mail between the sender and the server, and if the server is a US company, US government agencies can already have a copy of it. And if you use an end to end encryption (the data itself is encrypted), the BlackBox will only get that encrypted data.

Answer 5

It is possible to normally decrypt Internet traffic only if all websites like Google, Facebook would regularly submit their private keys, which is non-doable because Google and Facebook are US companies protecting privacy of their users, and decryption of such huge amount of traffic would lead to massive compromise of privacy.

However, it is quite possible, that they would perform MITM attacks with use of Certification Authorities (CAs), so this way they would be able to implement per-case decryption. For example, they would be able to MITM and decrypt emails going to specific email addresses.

This leads us to a very important question regarding how much CAs are protecting their keys and if they don't share it with other parties and if they do cooperate with authorities to perform MITM attacks. It's an important question since Rooot CAs are mostly in English-speaking counties except China, etc., so it could be, that government is able to MITM any Public CA-backed encrypted traffic and the only solution is to use your own CA to encrypt data.

Private CAs won't work helping with setting up SSL on public websites, but it would work perfectly fine for internal corporate systems. Another thing is how the root keys are protected and how the certificates are issued. It would be unwise to do it on a Windows machine with windows updates configured, because this way government would still have access to this machine. It would be recommended to run a minimum secure system disconnected from the Internet.

Answer 6

One type of such devices is sold and deployed for company LANs on a regular basis, but it works by an extra set of certificates that the IT department has to deploy to every client PC. These devices reencrypt the traffic with the owner-controlled certificate.

Setting such a system up countrywide could happen via browbeating CAs into compliance, OR browbeating users into installing such certificates, AND/OR by deploying your MITM certificates via PC and OS vendors (it would not cover the whole installed base, but a significant amount).

Another type of device, and that is what the UK is planning to introduce judging from your description, is mandated to be deployed in ISP data centers and is given access straight to the mail (or other application) servers themselves, so there is nothing to decrypt. Such has been in practical use in some European countries (check the German TKüV law, which mandates such equipment for German email providers that are more than 10,000 inboxes strong!) for quite a long time.

Answer 7

The linked Channel 4 article in the question misrepresents what is actually proposed in the Investigatory Powers Act 2016. There is a clue in the beginning of the article, where it says:

The government has insisted that the actual content of messages won’t be stored, but until now it has not been clear how communications companies will be able to separate content from “header data”, such as the sender and recipient of a message, and the date it was sent.

The Investigatory Powers Act 2016 requires communication companies to extract retain for a year source and destination IP-adresses and the destination domain (but not the full URL) from packages sent over the Internet, and not the actual package content. There is nothing unclear about how you do this. This is the routing information for the packet and it is not encrypted. Even if the actual content is encrypted (using SSL/TLS as will be the case with Gmail used as an example further down in the article).

Hence, the whole Channel 4 story is based upon false assumption that Investigatory Powers Act 2016 will require communications companies to decrypt content, rather than retaining unencrypted metadata. (For the record: I don't think the government should require metadata to be collected indiscriminately - so this act is evil - but that is what has been proposed, not that content must be decrypted).

So the “black box” required by the Investigatory Powers Act 2016 is not required to decrypt anything. It is required to extract the required metadata from the envelope of the package, and retain it for a year.

As for Gmail (also mentioned in the article) we know from the Snowden leaks that the NSA to access to the content after it has been decrypted at the receiving end, and that the NSA shares this data with the GCHQ, so Gmail content is without doubt compromised - but not by means of a “black box” that decrypts messages en route.

As for the question: It is possible to construct and deploy as part of UK ISP infrastructure a MitM “black box” that decrypts SSL/TLS if the government made such a thing mandatory?

Because key exchange using SSL/TLS happens in-band, this is feasible: You just need to insert a MitM that masquerade as the web service requested, and anyone handling DNS (which an ISP do by default for its own customers) is in the right position to do that. However, to make it work you also need to add ingridients such as DNS cache poisoning and outlawing DNSSEC to the mix, which would also make the UK a paradise for non-government cybercrime - so I hope this scenario is unlikely.

If the UK govermenment were to make it mandatory for UK ISPs to carry out MitM attacks against their own customers, those thinking serious about their privacy have a number of remedies. The simplest would be to stop trusting their ISPs DNS service (since it must be compromised for this to work). A slightly more advanced user would use end-to-end encryption (rather than SSL/TLS) where the key exchange are conducted through secure channels (i.e. outside the control of the government or ISP). There already exists a number easy to-use implementations of PKI (Enigmail for Thunderbird is the one I use) that allows anyone who is interested in managing his/her own keys capable of doing so.

Answer 8

It depends on how the data is sent, and new ways to unlock different kinds of data are found all the time, though usually fixed. I know from my former Gov Job (in the US) that their proxy servers automatically performed a MITM attack, the S in HTTPS was actually established with the proxy not the remote server, but you can easily look at the certificate chain and see who all signed it. (They owned the PC's and had set them all up to trust root Gov issued certs).

Like others have said they would need the proxy to have to have a signing cert your computer considers a trusted root authority otherwise there would be an error message, EXCEPT if the government knows of a security flaw with the encryption type used and is keeping that information secret; lots of conspiracy theorists think that since the NSA helped create AES which is generally considered the best form, that they were able to design it with a backdoor no one has discovered yet. Sometimes security holes are found that allow near instant access to every version of a software program for the last 15-20 years+.

And as I sorta almost started saying: the last likely small but possible chance; if the Gov had some sort of extremely fast computer, like what could potentially be in everyone's home as average in another 10 years from now. Nearly all encryption used online can be decoded with time. What takes 10 or 100 years right now will definitely be decodable on the fly by average computers within a couple decades-easily.

Answer 9

Nope, for a simple reason, it would require cooperation with google and a government agency + collaboration with the all the different ISPs, that would requires to introduce new code, a lot of new code which may have bugs and be faulty.

If that scenario would be a real scenario, than Google would have to use a different certificate for each user and you will notice (using some simple program or browser extension) that each connection/account has a different SSL certificate.

The reason to use vast amounts of certificates in this case is that investigations by agencies of 1 government would open "the doors" to all other governments allowing in case of leaking of the certificate to read all mails of google for all users in the whole world from all the countries.

• If the scenario is real, you will see unusual certificates quantity on your connection or either the government/google are taking a big risk at exposing national security! (Which I highly doubt).

• Also note that since you are assuming that the certificate can be manipulated (unless some scientist found a way to factor quickly big primes) and hence big web agencies and governments are involved anyway already, it is much more viable that google expose a Read-only API to government agencies (that would be much more tecnhically secure and feasible and would also allow to re-use existing software, not to mention it is even easier to "limit" the amount of data that can be accessed).

• Also the paper you cites does not know at all how SSL works: there's a "handshake" that uses strong asymmetric key, but then the transaction just use a simmetric key, so actually the "BlackBox" just need access to the simmetric key. O_O

Answer 10

Yes, entirely possible, currently already done, has been done forever.

All those saying it's impossible are somewhat lacking in imagination & research into the various cool tech and cunning tricks the spooks have employed over the years.

Lots of it is nothing new & is in freely available books by ex-spies and heads of organisations. It's been done in some way or another since they were steaming open people's envelopes and bugging typewriters back in the year dot, but extrapolate that activity forward and you've got to conclude it's all entirely within the realm of the possible & likely - and frankly it's what most people would expect their national spy agencies to be doing in the interest of national security.

Other details, more modern versions etc. of intercepting & monitoring calls and internet traffic have been leaked by Mr Snowden and the like. I haven't bothered delving into it but the broad strokes are that they can see it all if they need to. I just assume they can see anything they want. Even if your government aren't looking, the Russians or Chinese certainly might be so it's safest to assume everyone can see your stuff.

Incidentally, I recommend the book Spycraft by H. Keith Melton and Henry Robert Schlesinger which is a history of the CIA's Office of Technical Services and covers some of this sort of shenanigans.