I'm trying to put some anomoly detection in place and I'm looking at referrer data (I know that this is optional, and can be fudged in some circumstances). But I'm seeing too many cases of what I believe to be false positives.
From rfc2616:
The Referer[sic] request-header field allows the client to specify,
for the server's benefit, the address (URI) of the resource from
which the Request-URI was obtained
I take this to mean that the referer should be populated only when the request is generated from a link, form submission, ajax request, or related content explicitly containing the target URL. i.e. if the user navigates to my site using a bookmark, manually entered address or via another page site (not counting the possibility that this might issue a 30x redirect) then I shouldn't see the page they were on previously.
But I see a large number of cases where the URL supplied as the referrer has no link to my site. Further, many of these referrers are reputable operations (including large banks) who should be protected against XSS type attacks, therefore it's unlikely that the cases I see were due to the user accessing a modified page.
I am aware that there are long outstanding issues on iPads and iPhones (CVE-2009-2797, CVE-2008-3171) which give rise to this behaviour (before I started filtering these out, ipads and iphones were disproprortinately represented in the output) however I am seeing these oddities for MSIE, Firefox and 26, and very occassionally Google Chrome.
I've been unable to replicate the behaviour on Firefox (same version as reported from my production data) using
- a simple href (referer is prev page)
- a form (both GET and POST) (referer is previous page)
- a change to window.location via javascript (referer is prev page)
- an img ref (referer is prev page)
- bookmark (no referer)
- manually entered location (no referer)
I've not looked at Greasemonkey type scripts - can anyone advise if these will give rise to this behaviour? I don't see any evidence of pre-flighting which would imply a CORS.
Is there something else I am missing?