62

I have been reading about the Snoopers charter bill that was passed in the UK this week.

It mentions a "Black Box" which is cited here: ‘Black boxes’ to monitor all internet and phone data.

It states it works like so:

When an individual uses a webmail service such as Gmail, for example, the entire webpage is encrypted before it is sent. This makes it impossible for ISPs to distinguish the content of the message. Under the Home Office proposals, once the Gmail is sent, the ISPs would have to route the data via a government-approved “black box” which will decrypt the message, separate the content from the “header data”, and pass the latter back to the ISP for storage.

It is very vague on how it works to "decrypt" the data.

Is there such a thing as a "Black Box" and should I be concerned?

  • 12
    It seems like they would perform a man in the middle attack with the boxes. All traffic would be routed to their "black box" prior to its destination, where they would have the proper certificates to decrypt the traffic. That being said, there isn't a "black box" you can go buy that will MitM any TLS traffic. I believe at this moment these boxes have only been conceptualized, not produced. And if they ever are, it won't be something anyone can get their hands on. – INV3NT3D Nov 23 '16 at 13:57
  • 2
    I was under the impression a significant amount of traffic was already subject to this. GCHQ has a large monitoring station at the large undersea fibre cables from the US to the UK. – Tim Nov 23 '16 at 18:11
  • 1
    Wow, screw it. I'm just going back to face-to-face communication and sending letters via snail mail. At least that communication is not being recorded/monitored...heh, right guys? nervous chuckle – MonkeyZeus Nov 23 '16 at 21:25
  • 2
    The links included in the original question are a few years out of date...I have not looked at the detail of the new draft legislation to see how it compares to the defunct previous proposal. It does not materially impact the actual question, but the 'emotive' context presented is not necessarily accurate...just saying. – R15 Nov 24 '16 at 12:32
  • 2
    @kayleeFrye_onDeck Yes, unless they block it. Then you have to find ways to use Tor without looking like you're using Tor. Or they make it illegal to use Tor, and then you're too scared to do that because if you get it wrong you go to jail (without passing go). – immibis Nov 26 '16 at 1:22

10 Answers 10

65

Yes. It's called a Man-in-the-Middle attack. You terminate the SSL session at a mid-point, thereby having the encryption key, then create a new session to the target server, so you have that encryption key too. The data path now goes User->MitM->Server, where each of the arrows is an encrypted connection. Data returned from the server goes Server->MitM->User, again, where each arrow is encrypted, but the other points are not.

There are ways to prevent this from working, but in the case of a government mandated system, it seems likely that these will be specifically avoided - there may be regulations for companies to provide valid certificates for the "black boxes", so that HPKP keeps working, for example. It is unclear whether such rules would apply to companies which don't operate directly in the UK, or whether there would be penalties for attempting to bypass these rules (for example, by the use of VPNs based in other countries).

Edit based on comments: Note that it is technically possible to create such a device, but the problems mostly come from requiring cooperation from a large number of parties. As a government, there are options available which aren't possible for smaller actors. For example, it would be possible (if unlikely) to require that all internet connected devices sold in the UK come pre-configured with a government issued root CA certificate, and to prosecute anyone using a device which does not have this installed. This would be terrible for internet security, but so is the overall concept, so it depends on security experts convincing the government just how bad this idea is.

  • 4
    If ISP's were to implement this "Black Box" and perform a Man in the Middle attack on all requests, doesn't this render Encryption useless? – user1 Nov 23 '16 at 14:10
  • 24
    @user1 It renders SSL useless. However, if the payload is independently encrypted and signed by someone who has not surrendered their keys to the MitM, the payload's encryption is not compromised and can be verified to come from the originator. – Michael Nov 23 '16 at 17:19
  • 5
    Do you have any supporting evidence when you mention regulations forcing CAs to provide forged certificates to government authorities (in the UK)? Or even a single report of this actually happening? I'm finding it hard to believe, but would love to be proved wrong – goncalopp Nov 23 '16 at 21:47
  • 6
    Can you expand on how HPKP (certificate pinning) could possibly keep working, considering it was designed to prevent this very scenario? – goncalopp Nov 24 '16 at 0:49
  • 7
    @Aron "specifically designed AGAINST MITM attack": True; but the precondition is that the MITM doesn't have a certificate fraudulently authenticating him as the end point, I think. That is, the user connects to secure.site.com, and the black box pretends to be that site (and can "prove" it). Apparently Symantec (a CA) issued an intermediate certificate to BlueCoat; iiuc this enables bluecoat to issue certificates for arbitrary domains (gmail.com) which are trusted by all browsers (all browsers trusting Symantec, that is). – Peter A. Schneider Nov 24 '16 at 10:35
34

No, there's no way such a thing could possibly exist, for any significant amount of time.

There's two big hints at this in the article itself:

many technical experts are raising equally serious doubts about its feasibility

A Home Office spokesman said – “We have not issued any hardware or software specifications.

The only way this could work on a properly secured website is if either:

  • The user cooperated with the surveillance
  • The website cooperated with the government
  • All the governments in the world decided to cooperate to surveil everyone

Obviously, there is still a high (but lowering) number of insecure websites. And telephony systems are completely insecure, from a cryptography perspective. However, the quoted example - Gmail - is about as secure as you can get.

In general, considering how much noise Snooper's Charter caused in security circles, even if there is such a "Black Box" being designed, it'll be useless long before it is actually used.

The reasons for this are a bit complex to understand, but read on if you're interested on the gory details


As other answers mention, the mechanism that you described resembles the well studied man-in-the-middle attack.

MITM diagram diagram by Miraceti

Let's see how it works in practice. Here, Alice (e.g.: the user) and Bob (e.g.: the website) want to communicate confidentially. They communicate through a communication medium that Mallory controls. Clearly, if the messages are not encrypted, Mallory can read and change any messages.

What happens if Alice and Bob use a properly secured channel, though?

  • Mallory can't read the messages at all. This property is called confidentiality, and is usually provided by symmetric encryption.
  • Mallory can't change the messages. This property is called integrity, and is usually provided by a message authentication code
  • Mallory can, at most, prevent messages from being delivered.

Now comes the tricky part. For all these mechanisms to work, Alice and Bob have to agree on a secret key - something resembling a long randomly generated password. Because Alice and Bob might not have communicated before, this is usually done through asymmetric cryptography.


Assume Alice and Bob never communicated before. How can they agree on a secret "password" to use, in a way that Mallory can't possibly learn it? Let's use an analogy with the old postal system:

  • Alice sends a letter to Bob saying they want to communicate
  • Bob receives the message, and sends Alice a package with a open padlock
  • Alice receives the padlock, generates a "password", puts it inside a box, locks the box with the padlock, and sends the box to Bob
  • Bob gets the locked box, unlocks it, and reads the "password".

Obviously, Mallory can't open the padlock without Bob's padlock key. What Mallory can do, though, is intercept the padlock when Bob sends it to Alice, and replace it with a padlock of their own.

To prevent this easy attack, there is usually a trusted third party - let's call her Faythe. Faythe is responsible for "photographing" everyone's padlocks, and distributing these (very detailed) photographs. Because Alice trusts Faythe, she can check the padlock she receives against the photograph, and make sure it belongs to Bob.

In the web world, Faythe is called a Certificate Authority (CA), and the photographs are called certificates (technically, signed certificates).


Now it becomes clear how the government's plan might work: because they can't force Alice (the user) or Bob (the website) to cooperate (in general), they can try to persuade Faythe (the CA) to "send" fake certificates to Alice.

Researchers in the security community are aware of this scenario, and there's several mechanisms to prevent it, both technical and social:

  • Shame CAs that do this. Although this sounds quite basic, it's tremendously powerful. CAs are usually commercial entities. Their reputation, is, quite literally, their only asset. Once their reputation is ruined, they're essentially worthless. A CA that forged certificates would become distrusted very quickly. If Alice doesn't trust the CA anymore, the government attack would stop working.

  • Sanity check the certificate. In practice, certificates for a given website don't change often, and usually only at well defined intervals. If they change outside of the expected interval, you can assume the certificate is compromised. This technique is called certificate pinning.

  • Cross-check the certificate. Since a particular website's certificate remains constant globally, you can detect compromised CAs by cross-checking the certificates users receive across the world. There's several projects doing this, including Google's Certificate Transparency, EFF's SSL Observatory, MonkeySphere, Convergence, Perspectives, and probably others I'm not aware of.

Note that all of these mechanisms have been developed before any government even thought of publicly doing this.


Given all this knowledge, and assuming the worse possible case (a "blackbox" that works for a short period of time) - what can you do to protect yourself?

  • 16
    There is one thing nobody has mentioned yet. The Government (they!) can become a CA or subCA and use this to create certificates on the fly, whenever a SSL session is created. The blackbox only needs to intercept and modify the SSL handshake accordingly. Peoble may choose to not trust this CA, but it turns out every certificate you ever see is signed by this ominous CA, because the black box always replaces certificates with its own corresponding ones. Yeah, as a user you would detect the black box, but thats not a big deal seeing that the government flat out said they would use these. – user1129682 Nov 24 '16 at 8:59
  • After reading the other two answers, I was going to post an answer stating that such a device isn't feasible (without massive co-operation between many different parties). Your answer explains the issues really well – and includes an excellent explanation of how CA-based PKI operates. This really should be the accepted answer. – Anthony Geoghegan Nov 24 '16 at 9:54
  • 2
    @user1129682 quite true - this is the "user cooperates with the surveillance" scenario. The fact is, most users are not aware of this issue at all. I'd think that if the government suddenly made everyone's browser scream out "your connection is not secure!", the amount of negative publicity would ensure they would retract it pretty soon. Even if they didn't, SSL/TLS with a single CA would soon be replaced by something more secure. – goncalopp Nov 24 '16 at 11:57
  • 1
    @user1129682 Not just "can" - several governments already run CAs which are trusted by your browser. If you want to be quite dismayed, you should explore the list of CAs in your browser in depth... – Michael Hampton Nov 24 '16 at 16:32
  • 2
    +1 for Mallory!! I didn't know Eve had an accomplice! =O – Mehrdad Nov 25 '16 at 20:52
12

The black box is theoretically possible, but is practically impossible. In order for it to work the CAs would have to cooperate and be willing to provide the Black Box with legitimate certificates for every website that provides email services. Otherwise end users would receive certificate warnings in their browsers which would warn them that a MITM attack is occurring. Furthermore, even if the CAs agreed to do this (and I doubt they would), it would only work for traffic routed for a particular country, so any user could use a VPN which is outside of that country and the Black Box would be circumvented. Some countries have tried to prevent access to foreign VPNs by blacklisting known IPs, but since VPNs tend to move around quickly, the only effective way to prevent their use is by blocking everything except government approved IP blocks, and IMHO no country with true democratic elections would ever be able to get away with that level of censorship.

  • 5
    @user1 - It's difficult to detect a MITM attack over http, but over https it's much easier because the browser can detect if the certificate presented is from the correct site AND authenticated by an approved CA. If it isn't, your browser will show you a red bar error or other warning. Note that this is the main reason why CAs and paid ssl certificates exist in the first place - to detect MITM attacks when ssl is used. – TTT Nov 23 '16 at 16:38
  • 3
    @TTT - in the case being discussed (eg. where the Certificate Authorities are issuing legitimate certificates), even your web browser wouldn't alert you as the certificate it is being served is actually a legitimately signed cert. – Joel C Nov 23 '16 at 16:44
  • 18
    The government could only force CAs in the same country to cooperate. But if they do so, then browsers will quickly remove those CAs from the list of trusted CAs. So all that would achieve was to put all CAs in the country out of business while allowing SSL to keep being just as secure using CAs from other countries. So the government would have to force their population to install the root certificate of a government sanctioned CA without the help of any major browser vendor. – kasperd Nov 23 '16 at 20:04
  • 11
    @immibis Browsers tend to remove those CAs when they violate rules set by the browser producing companies. And a nation wide mitm attack would be noticed. For example recently Mozilla and Google both decided to make their browsers stop trusting StartCom because of infractions which are laughable compared to this nation-state-mitm scenario. (they didn't remove trust for all certificates, just for new ones, effectively putting the company out of business without impacting end users. I'm sure they would have removed trust completely had there been a real breach of security) – Nobody Nov 23 '16 at 21:36
  • 7
    @ThatBrazilianGuy Netflix prevents you from using their service through a VPN. A government preventing you from using the internet is a slightly different issue. – goncalopp Nov 24 '16 at 0:44
5

Conceptually, this is a UK version of the US patriot act. It will require the cooperation of the ISPs - easy that are under UK law - and major mail providers. For that latter part, many users simply use the mail box of their provider, mainly if they use SMTP/IMAP. Here no problem every non encrypted email is unencrypted at the server side and can be easily passed to the BlackBox.

Some users directly use webmails from international (US) companies such as Google. In that case, all unencrypted data can be given to US agencies in charge of legal regulations because of the Patriot Act. In that case, servers are generaly implanted in different countries to balance the load. Two possibilities here: ask the mail server to give all mail received on a server in UK to the BlackBox, or ask the mail company to give a valid key to the backbox to allow it to perform a Man In The Middle Attack.

It this really a new threat for confidentiality? No because the SSL only protect the mail between the sender and the server, and if the server is a US company, US government agencies can already have a copy of it. And if you use an end to end encryption (the data itself is encrypted), the BlackBox will only get that encrypted data.

  • 1
    So, because the US government agencies may already have a copy, it's not a problem is the ones in the UK do? Can you send me a copy of your email as well? – goncalopp Nov 24 '16 at 20:47
  • @goncalopp If you do nothing to protect your privacy against the patriot act, I can see no reason why you should act differently for UK. As far as I am concerned, I will never use gmail nor any US organization for private data. And if I know that my ISP uses such a BlackBox, I will encrypt my mails. – Serge Ballesta Nov 24 '16 at 23:22
  • @goncalopp But I will considere at same level US and UK agencies, maybe because I'm not a US citizen. – Serge Ballesta Nov 24 '16 at 23:25
  • That should be capitalized as PATRIOT Act. – immibis Nov 26 '16 at 1:27
2

It is possible to normally decrypt Internet traffic only if all websites like Google, Facebook would regularly submit their private keys, which is non-doable because Google and Facebook are US companies protecting privacy of their users, and decryption of such huge amount of traffic would lead to massive compromise of privacy.

However, it is quite possible, that they would perform MITM attacks with use of Certification Authorities (CAs), so this way they would be able to implement per-case decryption. For example, they would be able to MITM and decrypt emails going to specific email addresses.

This leads us to a very important question regarding how much CAs are protecting their keys and if they don't share it with other parties and if they do cooperate with authorities to perform MITM attacks. It's an important question since Rooot CAs are mostly in English-speaking counties except China, etc., so it could be, that government is able to MITM any Public CA-backed encrypted traffic and the only solution is to use your own CA to encrypt data.

Private CAs won't work helping with setting up SSL on public websites, but it would work perfectly fine for internal corporate systems. Another thing is how the root keys are protected and how the certificates are issued. It would be unwise to do it on a Windows machine with windows updates configured, because this way government would still have access to this machine. It would be recommended to run a minimum secure system disconnected from the Internet.

2

One type of such devices is sold and deployed for company LANs on a regular basis, but it works by an extra set of certificates that the IT department has to deploy to every client PC. These devices reencrypt the traffic with the owner-controlled certificate.

Setting such a system up countrywide could happen via browbeating CAs into compliance, OR browbeating users into installing such certificates, AND/OR by deploying your MITM certificates via PC and OS vendors (it would not cover the whole installed base, but a significant amount).

Another type of device, and that is what the UK is planning to introduce judging from your description, is mandated to be deployed in ISP data centers and is given access straight to the mail (or other application) servers themselves, so there is nothing to decrypt. Such has been in practical use in some European countries (check the German TKüV law, which mandates such equipment for German email providers that are more than 10,000 inboxes strong!) for quite a long time.

1

The linked Channel 4 article in the question misrepresents what is actually proposed in the Investigatory Powers Act 2016. There is a clue in the beginning of the article, where it says:

The government has insisted that the actual content of messages won’t be stored, but until now it has not been clear how communications companies will be able to separate content from “header data”, such as the sender and recipient of a message, and the date it was sent.

The Investigatory Powers Act 2016 requires communication companies to extract retain for a year source and destination IP-adresses and the destination domain (but not the full URL) from packages sent over the Internet, and not the actual package content. There is nothing unclear about how you do this. This is the routing information for the packet and it is not encrypted. Even if the actual content is encrypted (using SSL/TLS as will be the case with Gmail used as an example further down in the article).

Hence, the whole Channel 4 story is based upon false assumption that Investigatory Powers Act 2016 will require communications companies to decrypt content, rather than retaining unencrypted metadata. (For the record: I don't think the government should require metadata to be collected indiscriminately - so this act is evil - but that is what has been proposed, not that content must be decrypted).

So the “black box” required by the Investigatory Powers Act 2016 is not required to decrypt anything. It is required to extract the required metadata from the envelope of the package, and retain it for a year.

As for Gmail (also mentioned in the article) we know from the Snowden leaks that the NSA to access to the content after it has been decrypted at the receiving end, and that the NSA shares this data with the GCHQ, so Gmail content is without doubt compromised - but not by means of a “black box” that decrypts messages en route.

As for the question: It is possible to construct and deploy as part of UK ISP infrastructure a MitM “black box” that decrypts SSL/TLS if the government made such a thing mandatory?

Because key exchange using SSL/TLS happens in-band, this is feasible: You just need to insert a MitM that masquerade as the web service requested, and anyone handling DNS (which an ISP do by default for its own customers) is in the right position to do that. However, to make it work you also need to add ingridients such as DNS cache poisoning and outlawing DNSSEC to the mix, which would also make the UK a paradise for non-government cybercrime - so I hope this scenario is unlikely.

If the UK govermenment were to make it mandatory for UK ISPs to carry out MitM attacks against their own customers, those thinking serious about their privacy have a number of remedies. The simplest would be to stop trusting their ISPs DNS service (since it must be compromised for this to work). A slightly more advanced user would use end-to-end encryption (rather than SSL/TLS) where the key exchange are conducted through secure channels (i.e. outside the control of the government or ISP). There already exists a number easy to-use implementations of PKI (Enigmail for Thunderbird is the one I use) that allows anyone who is interested in managing his/her own keys capable of doing so.

  • "we know from the Snowden leaks that the NSA to access to the content after it has been decrypted at the receiving end" Not quite. We know that they did have access to the Google trans-datacenter links, where data flowed in plain text. We also know that Google claims to have started encrypting all of that data flowing between their datacenters. I don't think we can know for certain that it is actually encrypted, and/or that nation-state adversaries cannot break the encryption, but it would be a pretty big PR mess for Google to clean up if it became known that they aren't encrypting it. – a CVn Nov 26 '16 at 19:38
  • "source and destination IP-adresses and the destination domain (but not the full URL)" So IP headers plus SNI data (which is in plain text) from TLS. Add monitoring of DNS traffic and you have pretty well covered also the small fraction of TLS traffic that doesn't use SNI. No need to MITM anything to get access to that. – a CVn Nov 26 '16 at 19:39
  • Others answers are great but this one is the best, you have addressed the real issue instead of the fantasy and explained that it's impossible to perform MitM attacks if the user and the other party agree on sharing the key using a secure channel that the government can't monitor. – Ulkoma Nov 27 '16 at 11:48
1

It depends on how the data is sent, and new ways to unlock different kinds of data are found all the time, though usually fixed. I know from my former Gov Job (in the US) that their proxy servers automatically performed a MITM attack, the S in HTTPS was actually established with the proxy not the remote server, but you can easily look at the certificate chain and see who all signed it. (They owned the PC's and had set them all up to trust root Gov issued certs).

Like others have said they would need the proxy to have to have a signing cert your computer considers a trusted root authority otherwise there would be an error message, EXCEPT if the government knows of a security flaw with the encryption type used and is keeping that information secret; lots of conspiracy theorists think that since the NSA helped create AES which is generally considered the best form, that they were able to design it with a backdoor no one has discovered yet. Sometimes security holes are found that allow near instant access to every version of a software program for the last 15-20 years+.

And as I sorta almost started saying: the last likely small but possible chance; if the Gov had some sort of extremely fast computer, like what could potentially be in everyone's home as average in another 10 years from now. Nearly all encryption used online can be decoded with time. What takes 10 or 100 years right now will definitely be decodable on the fly by average computers within a couple decades-easily.

-1

Nope, for a simple reason, it would require cooperation with google and a government agency + collaboration with the all the different ISPs, that would requires to introduce new code, a lot of new code which may have bugs and be faulty.

If that scenario would be a real scenario, than Google would have to use a different certificate for each user and you will notice (using some simple program or browser extension) that each connection/account has a different SSL certificate.

The reason to use vast amounts of certificates in this case is that investigations by agencies of 1 government would open "the doors" to all other governments allowing in case of leaking of the certificate to read all mails of google for all users in the whole world from all the countries.

  • If the scenario is real, you will see unusual certificates quantity on your connection or either the government/google are taking a big risk at exposing national security! (Which I highly doubt).

  • Also note that since you are assuming that the certificate can be manipulated (unless some scientist found a way to factor quickly big primes) and hence big web agencies and governments are involved anyway already, it is much more viable that google expose a Read-only API to government agencies (that would be much more tecnhically secure and feasible and would also allow to re-use existing software, not to mention it is even easier to "limit" the amount of data that can be accessed).

  • Also the paper you cites does not know at all how SSL works: there's a "handshake" that uses strong asymmetric key, but then the transaction just use a simmetric key, so actually the "BlackBox" just need access to the simmetric key. O_O

-2

Yes, entirely possible, currently already done, has been done forever.

All those saying it's impossible are somewhat lacking in imagination & research into the various cool tech and cunning tricks the spooks have employed over the years.

Lots of it is nothing new & is in freely available books by ex-spies and heads of organisations. It's been done in some way or another since they were steaming open people's envelopes and bugging typewriters back in the year dot, but extrapolate that activity forward and you've got to conclude it's all entirely within the realm of the possible & likely - and frankly it's what most people would expect their national spy agencies to be doing in the interest of national security.

Other details, more modern versions etc. of intercepting & monitoring calls and internet traffic have been leaked by Mr Snowden and the like. I haven't bothered delving into it but the broad strokes are that they can see it all if they need to. I just assume they can see anything they want. Even if your government aren't looking, the Russians or Chinese certainly might be so it's safest to assume everyone can see your stuff.

Incidentally, I recommend the book Spycraft by H. Keith Melton and Henry Robert Schlesinger which is a history of the CIA's Office of Technical Services and covers some of this sort of shenanigans.

  • 2
    There are concrete mathematical reasons to believe that the current generation of symmetric ciphers is unbreakable when used correctly, and will remain so for the foreseeable future. Your position is far too pessimistic. – zwol Nov 25 '16 at 20:48
  • 1
    @zwol While I agree with what you say, there's a huge caveat in what you say: when used correctly. Far too many people don't even install security updates, to the point that Microsoft now forces non-enterprise versions of Windows to install updates; running old code with security flaws significantly weakens endpoint security. And I don't think we've seen the last security-related bug in any of the major browsers. That said, active attacks involve high stakes, because unless and until the machine is fully p0wned, they can be detected if someone happens to look in the right place. – a CVn Nov 26 '16 at 19:44
  • Why the downvotes? I thought it was pretty widely known (especially in security circles) the capabilities they have (especially post-snowden), and that if a government wants to do something like this, it can & will. – John U Nov 28 '16 at 11:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.