This is a situation that I have encountered on a number of occasions. I will conduct a web search in Google, a link to a file (.pdf for example) will take you to that file on the server. It opens up in your browser, you can save the file.

But when you backtrack through the URL to find out more about the website that holds it, and if there is other information that can aid your search, you are denied access. You are either given an error message or taken to the log in page for a web portal. Which can be accounted for by configuring the server to prevent directory traversal.

While I understand the lack of a robot.txt file could contribute to this, some of these sites would not rely on these for protection since a rogue web crawler would just ignore the file anyway. Not to mention the robot.txt would enumerate a lot of your files and directory structure to a hacker.

So it would be too much to presume that some of these sites were not protected by physical firewalls, and given the server authentication which denied moving to other parts of the site, that crawling the site could be done without having log in credentials.

So what loop holes is the Google crawler exploiting that allows it to bypass possible firewall, server authentication, and portal protection to find out the directory structure of that server and create direct links that allow someone without credentials to see that file?

  • 1
    Many sites have checks to see if the user-agent belongs to the Google crawler, and bypass certain access checks. It's always a good thing to check for if you're testing a site.
    – Polynomial
    May 24, 2014 at 8:22

1 Answer 1


I some times get the illusion that GoogleBot has special access to the directory structure but that doesn't mean that there is a loophole or a vulnerability.

Some of the reasons for that illusion:

  • The links might be from outside of the website. I can post on my blog a link to a Pastebin that is not listed anywhere on pastebin.com.
  • The pages that had the links to your documents might have been deleted or hidden but Google still has the link to the file.
  • The sitemap file can have a list of links to the files on that website.
  • The owner of the website can use Google Webmaster Tools and load a list of links on the website.
  • Something that is almost but not quite a vulnerability is that GoogleBot decompiles flash resources and indexes text and links found there.
  • It can also execute JavaScript and parse Ajax content.
  • There is a new Googlebot user agent that looks for mobile content. The mobile interface can be missconfigured to show more information and have more access than the classic one.
  • As others commented, websites can provide more or different details when the visiting IP or user agent indicates Googlebot. This is called clocking and is not tolerated by Google or any other search engine.

Googlebot has simple rules, it follows links and respects the robots.txt file and the nofollow tag. It indexes information that has to be accessible to Google's users and not break the security or privacy of the crawled websites. If there is anything that indicates otherwise, then I suspect there is a website missconfiguration rather than Googlebot malice or magic.

  • 1
    I would add that there exist the possibility that site owners give special privileges to the GoogleBot (god knows why) by checking the User-Agent of the visitor. It is just a very very rare possibility, but there it is.
    – kinunt
    May 24, 2014 at 9:28
  • @Cristian Dobre while this explains a lot, it would seem to apply mostly to commercial websites, web sites that incorporate any Google services such as Search or Adsense, or more open/insecure sites. But how would this apply to more secure corporate, academic or governmental sites? Even if the GoogleBot attempted to follow links, decompile information from flash resources or the sitemap protocol (assuming they were used despite their insecurities), there are still the authentication measures that should prevent it from freely crawling that site or allowing web surfers access to protected files
    – tk1974
    May 25, 2014 at 3:47
  • I updated my answer to hopefully have a definite answer to your questions. May 25, 2014 at 9:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .