As @aus indicates, you can try to observe the exact characteristics of the ClientHello and other SSL features (e.g. how messages are split over records) to get some indication about whether the client is a genuine Web browser, or not. However, this is not robust:
- Different browsers act differently in that matter. You may be engaging into a never-ending uphill quest, with continuous evaluation of all major browser versions.
- An OS/browser update may change its usage patterns (e.g. some cipher suite becomes enabled by default), so your detection mechanism may be plagued with surges of false positives at some unpredictable moments (when Microsoft / Mozilla / Google decides to push some technical update).
- If users are aware of such fingerprinting then they may begin to use clients who mimic the features that you are trying to detect. "Perfect" emulation of the SSL characteristics of a "normal Internet Explorer" is possible. Provide an incentive (a detection mechanism which triggers retaliation), and it will happen.
You might want to make some traffic analysis based on the timing and size of requests; when a browser connects to a server, it will first send a request with a header whose size depends on a lot of things, but will not be too small. In the specific case of SSH-within-SSL, there is a nifty underhanded way by which distinction is easy: add a delay.
Specifically, when a Web server receives a connection, it waits: the client is supposed to send a request. From the outside, you observe a SSL handshake, culminating in the two Finished messages (they would show up as "encrypted handshake" since you are monitoring from the outside), then the client will send an "application data" record (in SSL, record contents are encrypted, but the type of the record is not: it is either handshake, alert, change_cipher_spec, or application data). As long as the "application data" from the client is not sent, the server will wait.
With SSH, however, things are different. As soon as the connection is established, the client and the server send each other their "banner" (which indicates protocol version and so on), in no particular order. Therefore, when your monitoring tool sees a new SSL handshake, it may temporarily delay the first application data record from the client (by, say, 0.5 seconds), to see whether the server would spontaneously send an "application data" record of its own without waiting for the record from the client. If the server does it, then it is not speaking HTTP within the SSL tunnel.
Of course, if you begin implementing such a detection mechanism, users will adapt, by modifying their SSH clients and servers to make the servers wait for the client banner before talking. This is likely to degenerate into the same kind of sterile warfare as between virus and antivirus. This is, in the long run, exhausting.
Apart from SSH, your biggest problem will be Web proxies. Namely, the user:
- Controls an external Web proxy.
- Runs on that external machine a SSL server which simply forwards data bytes to the Web proxy.
- Runs on his local machine a simple process which listens on local port 3128 and forwards all data to the SSL server.
- Configures his browser to use "localhost:3128" as proxy for Web requests.
With this setup, all the user's Web browsing of the server will go through the SSL connection, out of reach of your monitoring systems; and, crucially, all the requests will be HTTP requests from a normal Web browser, thus indistinguishable from HTTP requests from a normal Web browser. This ultimately defeats traffic analysis.
The conclusion is that what you are trying to do is to fight a war that you cannot win. I suggest you begin looking for alternate models. One way to look at it is the following: if you allow BYOD on your workplace, then you must trust employees for not abusing it, and for enforcing good security practices on themselves.
