I am designing an anomaly-based intrusion detection system in java. It basically consists of a sniffer that identifies HTTP header fields and then analyze them according to a previous configured XML file.
My question is: what if my system uses HTTPS? how can I read HTTPS header fields if they are all encrypted?
Edit: just so I preempt comments along the lines you know nothing about how IPS works, I will stipulate that, and instead direct my answer towards the general method of inspecting HTTPS traffic.
The short is answer is you can't.
The long answer is you would have to perform MITM attacks against the connections. This can be done in cases when you only have serverside authenticated HTTPS connections if you can force all your clients to trust a certificate that is under your control. It's often used but in my opinion it is a bad security practice since usually whatever certificate you use, will have lower guarantees of safety then the ones usually trusted by browsers and users. Even if it doesn't you are adding a new point of attack.
If you do have to use the MITM attack make sure you at least check that the certificates you are getting (from the servers) are correct, applicable for the serversm trusted and not on CRL's.
One other by-product of this approach is you are most probably teaching your users questionable security practices since they will be willing to trust obviously faked certificates for servers they communicate with. Then when they go to a hotel and see the same issue they might just assume that they are doing the same thing and happily divulge all company secrets over insecure connections.
Classical IDS work at the network layer and thus cannot handle encrypted connections like HTTPS which happen at the application or presentation layer (see OSI model). But, since you are interested only in detecting attacks by checking HTTP header fields (i.e. attacks at the application layer) you could built a (transparent) HTTP proxy instead and then do analysis directly at the application layer without caring about the network layer. At this level you could also do SSL interception and there are various examples how this can be done (like mitmproxy).
Of course, this way you don't do any longer passive network analysis but you are instead an active application in the network path which forwards the data after it verified it. This is usually not called an IDS but instead an application level gateway (ALG).
According to NIST - Special Publication 800-94 Guide to Intrusion Detection and Prevention Systems IDPS:
Network-based IDPSs cannot detect attacks within en crypted network traffic, including virtual private network (VPN) connections, HTTP over SSL (HTTPS), and SSH sessions. As previously mentioned, some network-based IDPSs can do some analysis of the setup of encrypted connections, which can identify that the client or server software has known vulnerabilities or is misconfigured. To ensure that sufficient analysis is performed on payloads within encrypted network traffic, organizations should use IDPSs that can analyze the payloads before they are encrypted or after they are decrypted. Examples include placing network-based IDPS sensors to monitor unencrypted traffic (e.g., traffic that entered an organization through a VPN gateway and has since been decrypted) and using host-based IDPS software to monitor activity within the source or destination host.
I will keep developing my IDS to listen to port 80 only. My IDS is anomaly-based for it uses Bayesian Networks to reduce false positive rates. Since Snort is signature-based it would not attend to my objectives.
Maybe I did not make myself clear but the main reference is of my work can be found here.
