I would like to be able to not just centrally monitor but also filter any organizational data moving out our edge routers, regardless of the sender application and regardless of the protocol/port used by the sender application.

For example, the sender application could be an ssh/sftp client, a browser (http/s), email client(??), or even a from-scratch handwritten TCP socket-based client/server program.

If the outgoing data happens to be encrypted (say, as in case of https/ssl, ssh/sftp), I would still like to be able to decode it via an MITM-like pattern (as employed by a program like Squid) and reject or allow the data to pass through based on the decoded content.

For example, if a large file was sent out by a user, I would like to be able to extract and assemble the file into a single logical unit (from the IP packets?) and then be able to run further checks on this assembled file to decide whether or not to allow it to pass through.

Given a Linux-based environment, which (FOSS) tools and techniques should I employ to achieve this? I'm new to this area, and don't know how to proceed further. Hence the question.

  • 2
    You might want to try keywords like "data loss prevention" (DLP) which is essentially what you are looking for. – logicalscope Apr 26 '12 at 18:00

First you need to decide how you will identify this data - this is probably going to be the biggest issue and it is not necessarily technical.

  • Do you aim to tag all documents and files with a rating? This works for organisations that do tag ever single piece of data, as the gateway then just looks for the tag and acts accordingly.

  • Do you set a blacklist of data to block? For example sequences of 16 digits can be blocked as they may be credit card numbers. You'll get false positives, but it can work in some environments.

If you have that sorted, the gateway aspect is relatively simple - force all devices to connect through a proxy that acts as an SSL endpoint. As you stated, this is effectively a MITM. Many proxies, including squid, will let you prevent or allow traffic based upon various criteria, but in the FOSS space you may be limited in the complexity level possible. Those two options above should work though.

You may find that trying to run deep analysis of packet streams in order to do this real time may be beyond the reach of FOSS (Have a look at next gen firewalls such as Palo Alto for some options here)

  • Thanks for your quick response, Rory. Identifying the data should not be a problem once some tool can assemble it for me. Secondly, is it true that Squid+ICAP server can only peek at http/s protocol traffic and not other arbitrary traffic? E.g., Would Squid+ICAP be able to even realize/get alerted to the fact that an arbitrary sockets program written by the user is sending something over either a well-known port (such as 80, 123, ...) or over an arbitrary port (> 1023) using an arbitrary protocol? – Harry Apr 26 '12 at 16:16
  • I think Snort would be a much better tool for identifying patterns. Use squid as the proxy, but use snort for it's IDS analysis capability. – Rory Alsop Apr 26 '12 at 22:23

Sounds like you're looking for Data Loss Prevention (DLP) solutions. Although there are plenty of commercial tools, I recall Google, Snort, and a few other organizations offer FOSS DLP capabilities.

Commercial DLP vendors like Symantec (Vontu) and Websense offer complete solutions. The last I checked, FOSS DLP solutions were still quite lacking in features/functionality when compared to commercial DLP solutions.


I'm not aware of any FOSS solution that provides all the features you're looking for. Even commercial DLP solutions aren't able to peek into ssh sessions (https is much easier since you can just employ a l7 proxy architecture). The challenge is building meaningful rules. Signature-based network devices operate at l3/l4 and they don't rebuild packets into conversations. Even then, network-only data-in-motion detectors are limited to seeing plaintext traffic (with the exception of an inline l7 proxy for http/https/ftp) without implementing some sort of endpoint solution as well.

Can you explain with a bit more detail what type of data you're trying to protect? Perhaps a feature rich data-in-motion FOSS network-based DLP might not be available but you may be able to apply protection to the data itself. Otherwise, your challenge becomes protecting information assets without an appropriate budget, which is almost saying the data isn't worth protecting in the first place.

  • Yes, I'm after DLP. I don't have budget for commercial solutions, and opensource ones (just like you say) aren't good enough. – Harry Apr 27 '12 at 3:17
  • "Can you explain with a bit more detail what type of data you're trying to protect?" Code, fingerprinted documents (such as design specs, product internals), and anything and everything that should not get out of the organization. I was hoping that I'd be able to piece together IP packets and run some DLP analysis in realtime, or if not, at least postprocess the logged IP packets somehow. Would you be able to suggest at least an appropriate forum to discuss this further with experts? – Harry Apr 28 '12 at 12:02
  • Also, could you elaborate just a little bit why https can be peeked into via l7 proxy but not ssh? Can't MITM approach be used with ssh as well? – Harry Apr 28 '12 at 16:45
  • You could ask on the serverfault forum. I haven't seen a FOSS solution that adequately addresses all the use cases. As far as ssh, you can launch a MITM against ssh, but ssh is more similar to PGP/GPG than PKI-based infrastructure for SSL/TLS. HTTPS MITM is more commonly implemented in DLP tools - I haven't seen ssh mitm implementations yet. Good luck! – bangdang Apr 29 '12 at 15:25

SSL Bump, to be integrated into a Squid proxy, can decrypt outgoing SSL sessions, subject to some conditions:

  • A specific CA certificate must be added to the "trust store" of clients. SSL Bump works by creating a fake certificate for the target server, and doing a man-in-the-middle attack.
  • The client browsers must be convinced to use your proxy. This can be enforced at the network level, depending on the level of control you have on the firewalls and routers.
  • This breaks certificate-based client authentication. Client certificates are rare on the Web, but some banks issue certificates to their clients.
  • The users will be able to see it. Most human users tend to react poorly to the discovery of such filtering of their supposedly "secure" connections. You'd better warn them proactively. Transparency is a great asset for employer/employee relations.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.