I'm trying to put some anomoly detection in place and I'm looking at referrer data (I know that this is optional, and can be fudged in some circumstances). But I'm seeing too many cases of what I believe to be false positives.

From rfc2616:

The Referer[sic] request-header field allows the client to specify,
for the server's benefit, the address (URI) of the resource from
which the Request-URI was obtained

I take this to mean that the referer should be populated only when the request is generated from a link, form submission, ajax request, or related content explicitly containing the target URL. i.e. if the user navigates to my site using a bookmark, manually entered address or via another page site (not counting the possibility that this might issue a 30x redirect) then I shouldn't see the page they were on previously.

But I see a large number of cases where the URL supplied as the referrer has no link to my site. Further, many of these referrers are reputable operations (including large banks) who should be protected against XSS type attacks, therefore it's unlikely that the cases I see were due to the user accessing a modified page.

I am aware that there are long outstanding issues on iPads and iPhones (CVE-2009-2797, CVE-2008-3171) which give rise to this behaviour (before I started filtering these out, ipads and iphones were disproprortinately represented in the output) however I am seeing these oddities for MSIE, Firefox and 26, and very occassionally Google Chrome.

I've been unable to replicate the behaviour on Firefox (same version as reported from my production data) using

  • a simple href (referer is prev page)
  • a form (both GET and POST) (referer is previous page)
  • a change to window.location via javascript (referer is prev page)
  • an img ref (referer is prev page)
  • bookmark (no referer)
  • manually entered location (no referer)

I've not looked at Greasemonkey type scripts - can anyone advise if these will give rise to this behaviour? I don't see any evidence of pre-flighting which would imply a CORS.

Is there something else I am missing?

  • 1
    Are the sources of these referers dynamic pages? It's possible that when you view the page there is no link to your site, but when someone else views it there is. Another possibility is that spammers are including legitimate referers as well as their own sites, to try to avoid being blacklisted. – paj28 Feb 9 '14 at 22:04

It's a common method of (SEO) spamming to give fake referrers with the hope of being listed in some place or another, this might explain some of the entries.

  • For several reasons, not all of which I can really discuss here) I'm very sure this is highly unlikely - or at least that the users concerned are knowingly attempting to do this. :( – symcbean Jan 11 '14 at 21:14

If a web page does not directly link to your web page, but instead sends users through a redirection script which redirects users with a 301 Moved Permanently status code, the referrer will be the page the link was on instead of the script.

For example, consider the following scenario:

Page 1

This is a page which links to your site through a redirection script.

(…)<a href="/redirect.php?goto=your_site_url">Your Site</a>(…)

Page 2

This is the redirection script. It could be as simple as:

<?php if(!isset($_GET['goto'])) die("No redirect URL found");
  header('HTTP/1.1 301 Moved Permanently');
  header('Location: '.$_GET['goto']);//You should not do this in your application: http://websecurity.com.ua/3386/ https://www.owasp.org/index.php/HTTP_Response_Splitting https://www.owasp.org/index.php/Open_redirect
  die("Moved <a href='".$_GET['goto']."'>Here</a>.");//This is a bad idea as well: https://www.owasp.org/index.php/Cross-site_Scripting_(XSS)
?>

Now, if a user clicks the link on page 1, he will be sent to your site through the redirection script, but with the page with the link on it as the Referer [sic].

  • Thanks, I suspect it still doesn't explain the traffic I'm seeing when the hosts apearing the referers should be be really on top of their security. – symcbean Jun 10 '14 at 12:36

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.