4

A customer found about a dozen valid URLs pointing to existing customer related documents at Yahoo. These URLs were not public and certainly not searchable at the customer's site. The documents have hard to guess names like https://site/dir/hardtoguessname.pdf; according to Burp's sequencer the entropy of hardtoguessname is estimated to be more than 100 bits - that should be good enough to prevent plain guessing.

The whole affair is odd for two reasons: First, there are regulary hundreds or thousands of these documents there - why where just those few indexed? Second, these URLs were indexed only by Yahoo but neither by Google nor by Bing.

I don't think that those URLs were indexed by ordinary crawling. Is it possible that a user could by chance got those URLs indexed, say, by using the Yahoo tool bar or by using Yahoo mail?

4

Using a hard to guess name is in no way a proper method to prevent search engine indexing. While it may sometimes work, it is the least reliable and effective way to do it.

You should instead use the officially supported method which is placing a robots.txt file in your web root detailing which files and directories should be indexed. This is supported by all major search engines including Google, Bing, Yahoo, AOL, etc.

The syntax is pretty straight-forward. A simple example to prevent indexing the private directory is:

User-agent: *
Disallow: /private/

See the relevant Wikipedia page on robots.txt for more details.

Note that private documents should not be accessible directly either way. Your website should use some form of authentication system to control what the user may access and private information or documents should not be accessible from inside the web root.

  • The caveat here is as a pentester, the robots.txt is the first file I'm going to look at to see what you don't want me to see. Password protecting the data or folder is the better approach imo to ensure it's not indexed. – DKNUCKLES Aug 16 '16 at 15:45
  • True, but that is irrelevant to search engines. Of course, private documents should be protected and limited to logged in authorized users. It's a good point though and I'll add a note about it. – Julie Pelletier Aug 16 '16 at 15:48
  • Note that robots.txt can prevent crawling, not indexing. A search engine might still index a URL which it’s not allowed to crawl. – unor Aug 18 '16 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.