How can the Google web crawler bypass portals?

Question

This is a situation that I have encountered on a number of occasions. I will conduct a web search in Google, a link to a file (.pdf for example) will take you to that file on the server. It opens up in your browser, you can save the file.

But when you backtrack through the URL to find out more about the website that holds it, and if there is other information that can aid your search, you are denied access. You are either given an error message or taken to the log in page for a web portal. Which can be accounted for by configuring the server to prevent directory traversal.

While I understand the lack of a robot.txt file could contribute to this, some of these sites would not rely on these for protection since a rogue web crawler would just ignore the file anyway. Not to mention the robot.txt would enumerate a lot of your files and directory structure to a hacker.

So it would be too much to presume that some of these sites were not protected by physical firewalls, and given the server authentication which denied moving to other parts of the site, that crawling the site could be done without having log in credentials.

So what loop holes is the Google crawler exploiting that allows it to bypass possible firewall, server authentication, and portal protection to find out the directory structure of that server and create direct links that allow someone without credentials to see that file?

Answer 1

I some times get the illusion that GoogleBot has special access to the directory structure but that doesn't mean that there is a loophole or a vulnerability.

Some of the reasons for that illusion:

• The links might be from outside of the website. I can post on my blog a link to a Pastebin that is not listed anywhere on pastebin.com.
• The pages that had the links to your documents might have been deleted or hidden but Google still has the link to the file.
• The sitemap file can have a list of links to the files on that website.
• The owner of the website can use Google Webmaster Tools and load a list of links on the website.
• Something that is almost but not quite a vulnerability is that GoogleBot decompiles flash resources and indexes text and links found there.
• It can also execute JavaScript and parse Ajax content.
• There is a new Googlebot user agent that looks for mobile content. The mobile interface can be missconfigured to show more information and have more access than the classic one.
• As others commented, websites can provide more or different details when the visiting IP or user agent indicates Googlebot. This is called clocking and is not tolerated by Google or any other search engine.

Googlebot has simple rules, it follows links and respects the robots.txt file and the nofollow tag. It indexes information that has to be accessible to Google's users and not break the security or privacy of the crawled websites. If there is anything that indicates otherwise, then I suspect there is a website missconfiguration rather than Googlebot malice or magic.