How does IPS terminate/decrypt encrypted traffic?

Question

With regards to IPS, I am not completely across how they are supposed to terminate or decrypt encrypted traffic? It makes absolutely sense to deploy them inline, but how is traffic supposed to be broken up so it can be inspected? Is this working similar as any CASB infrastructure that would be put in place?

Answer 1

There are four ways an IPS can deal with encrypted traffic:

1. An IPS can act as a TLS interception proxy, where it becomes a man-in-the-middle for the TLS connection. This requires you add a root certificate created by the IPS software to your root trust store, or every encrypted website you try to visit will give a connection error. This allows an IPS to monitor the contents of encrypted traffic transparently, with some exceptions. This can be risky if the IPS software is insecure or untrusted, as you are delegating all trust to it.

2. An IPS can take the loss and analyze whatever it can without needing to decrypt the traffic (for example source and destination IP and port, traffic analysis-based heuristics, TCP header information, SNI, etc). This is of course less thorough inspection, but it does not rely on each client behind the IPS to install its custom TLS certificate, which can be a security risk.

3. If the software is integrated with the application using encrypted connections (which is possible if the IPS is running on the TLS endpoint itself, rather than on a separate hardware firewall), it can inspect the encrypted traffic after it has already been decrypted. This has the benefit of not requiring TLS interception while still being able to read encrypted traffic, but has the downside of only working with supported applications, and needing to run on the endpoint.

4. Any encrypted traffic can simply be blocked. This allows the IPS to read most traffic and deny that which it can't read, but at the expense of usability. However, an unauthorized entity can still communicate by using a non-standard or obfuscated encrypted protocol that the software is unable to understand. Using protocol whitelisting is not a solution either, since it would be possible to send encrypted data in the contents of an HTML page over plain HTTP.

As for terminating an encrypted session, that is easy. TLS is on a different layer than TCP (in the OSI model, TLS is layer 6, whereas TCP is layer 4). That is, TLS is encapsulated within the TCP connection and does not encrypt TCP-related information at all. In order to tear down a connection, all the IPS has to do is send out an RST. Once the TCP connection is dead, so is the TLS session.