I'm not sure you could reliably detect private browsing, but I think you may be able to apply some heuristics to make a good guess that a user is using various privacy-enhancing features. As indicated in my comment on the question, whether this is good enough or fits your application depends on what you want to be able to do in reaction to detecting private browsing. As Sonny Ordell mentioned, I'm also not sure that you can distinguish private browsing from the ad hoc use of various privace-enhancing features (e.g. manually clearing history or cookies).
Let's assume you operate a web application, and you want to detect when one of your users (with an account) switches to private browsing. I'm specifying that the user has an account, because this strategy relies on tracking various bits of behavior data. The aspects of private browsing are (at least in Firefox): history, form/search entries, passwords, downloads, cookies, cache, DOM storage. I'm not sure how to probe for downloads, but I think the others can be probed. If you get a positive detection on all of them, it seems more likely that your user is private browsing.
In the trivial case, you keep track of
(IP, user-agent) for each user. When you get a cookie-less request for a matching
(IP, UA) record, you might infer that the corresponding user is private browsing. This method fails (no detection) if:
- He uses something like ProxySwitchy or TorButton to activate Tor during private browsing, thus changing IP.
- He switches to a different browser (e.g. usually uses FF and switches to Chrome for Incognito mode).
- The switch to private browsing is not immediate and his ISP has issued a new IP (e.g. on Friday he was 10.1.2.3, he didn't use your app over the weekend, and on Monday he is 10.1.4.5).
As mentioned in Sonny Ordell's answer, if another person uses the same browser in private browsing mode to access a separate account on your site, you will get a detection -- but this is a slightly different case than if the "normal" user simply switches to private browsing mode.
You'll get a false-positive if the user simply clears his cookies for your site, or uses a secondary profile (e.g. I keep a few different Firefox profiles with different sets of plugins for certain testing and/or to avoid tracking, though I'd guess this is very uncommon).
You may be able to defeat issue 1 (Tor) by detecting access via a Tor exit node, and combining this with fingerprinting. This seems like it would only be helpful in a narrow range of cases.
Instead of just cookies for the checks above, test
localStorage. If it's typically enabled, and your key isn't in the storage for this visit, and the fingerprint matches, then this is probably private browsing. Obviously, if the user normally has storage disabled, then you can't use it. The failure modes are similar to those described above for cookies.
I haven't tested or developed the idea, but I suppose you could play games with
Cache-Control. (A quick search reveals that this isn't an original idea -- that project has what looks like proof-of-concept code.) This strategy fails if the user goes through a shared caching proxy -- the meantime page mentions anonymizer.com. Firefox, at least, doesn't use the cache in private browsing mode. (See this site for a demo of cache-based tracking.) So you could combine this with the UA/fingerprinting mentioned above: if your cache tracker indicates this is a first visit, then you can guess that the user is private browsing. This fails with a false positive if the user cleans his cache; combine with other techniques to get a better guess.
You could detect and track, for each user, whether the browser autofills a certain form element. If you detect that a given user doesn't get autofill on that form element, you might infer private browsing. This is brittle -- perhaps the user is not using his "primary" computer, but you could combine it with fingerprinting as mentioned above for a more reliable guess.
Side-channel timing attack: detect and track the typical time it takes for each user to log into your app. There will be variations, but I'm guessing that you could get an accurate guess about whether someone is using password autofill. If a user normally uses password autofill (i.e. fast transition through the login page), and then for a given visit (with a matching fingerprint) is not using autofill, you can infer private browsing. Again this is brittle; combine with other techniques for a better guess. You'll also want to detect and correct for network latency on a given page load (e.g. perhaps the user's network is just slow on a given day, and a slow login page transition is just latency and not a lack of autofill). You can be slightly evil and auto-logout the user (give them a bogus error message, "please try again") to get a second data point if you're willing to annoy your users a bit.
Combine this with what you mentioned in the question about detecting if the user is logged in to other services (e.g. Facebook), and you can have more confidence in your guess.
If you're really motivated, you could play games with DNS and tracking page load times. A quick test of FF 3.6 and Chrome 15 seems to indicate that neither browser clears the DNS cache in private browsing mode. And the browser has absolutely no control over the local system's DNS cache. If you use a side-channel DNS timing attack to perform user tracking as an alternative (or in addition to) fingerprinting, you may get a more reliable guess. I'm not sure how reliable tracking via DNS timing will be.
Detection of "anonymous" users in private browsing mode will be much harder, since you haven't had the opportunity to accumulate data on their "typical" behavior. And, since most of the features only kick in when they end the browser session, you don't really know if they're ever going to be back.
Everything above is full of holes -- plenty of room for false positives or negatives. You'll probably never know if I'm using private browsing, or if I'm running a browser in a VM with no persistent storage. (What's the difference?)
The worst part is probably that if you do get an answer with a reliable method for detecting private browsing is that it seems unlikely to remain viable for very long as browsers either "fix" it or users find workarounds to avoid detection.