Anomaly detection in HTTP REFERER

Question

I'm trying to put some anomoly detection in place and I'm looking at referrer data (I know that this is optional, and can be fudged in some circumstances). But I'm seeing too many cases of what I believe to be false positives.

From rfc2616:

The Referer[sic] request-header field allows the client to specify,
for the server's benefit, the address (URI) of the resource from
which the Request-URI was obtained

I take this to mean that the referer should be populated only when the request is generated from a link, form submission, ajax request, or related content explicitly containing the target URL. i.e. if the user navigates to my site using a bookmark, manually entered address or via another page site (not counting the possibility that this might issue a 30x redirect) then I shouldn't see the page they were on previously.

But I see a large number of cases where the URL supplied as the referrer has no link to my site. Further, many of these referrers are reputable operations (including large banks) who should be protected against XSS type attacks, therefore it's unlikely that the cases I see were due to the user accessing a modified page.

I am aware that there are long outstanding issues on iPads and iPhones (CVE-2009-2797, CVE-2008-3171) which give rise to this behaviour (before I started filtering these out, ipads and iphones were disproprortinately represented in the output) however I am seeing these oddities for MSIE, Firefox and 26, and very occassionally Google Chrome.

I've been unable to replicate the behaviour on Firefox (same version as reported from my production data) using

• a simple href (referer is prev page)
• a form (both GET and POST) (referer is previous page)
• a change to window.location via javascript (referer is prev page)
• an img ref (referer is prev page)
• bookmark (no referer)
• manually entered location (no referer)

I've not looked at Greasemonkey type scripts - can anyone advise if these will give rise to this behaviour? I don't see any evidence of pre-flighting which would imply a CORS.

Is there something else I am missing?

Answer 1

It's a common method of (SEO) spamming to give fake referrers with the hope of being listed in some place or another, this might explain some of the entries.

Answer 2

If a web page does not directly link to your web page, but instead sends users through a redirection script which redirects users with a 301 Moved Permanently status code, the referrer will be the page the link was on instead of the script.

For example, consider the following scenario:

Page 1

This is a page which links to your site through a redirection script.

(…)<a href="/redirect.php?goto=your_site_url">Your Site</a>(…)

Page 2

This is the redirection script. It could be as simple as:

<?php if(!isset($_GET['goto'])) die("No redirect URL found");
  header('HTTP/1.1 301 Moved Permanently');
  header('Location: '.$_GET['goto']);//You should not do this in your application: http://websecurity.com.ua/3386/ https://www.owasp.org/index.php/HTTP_Response_Splitting https://www.owasp.org/index.php/Open_redirect
  die("Moved <a href='".$_GET['goto']."'>Here</a>.");//This is a bad idea as well: https://www.owasp.org/index.php/Cross-site_Scripting_(XSS)
?>

Now, if a user clicks the link on page 1, he will be sent to your site through the redirection script, but with the page with the link on it as the Referer [sic].