8

Is there a way to basically have an allow all policy on your top level domain, and any subdomains disallow all? I would rather that my public facing app servers not get indexed, but from what I can tell, no robots are even polling for robots.txt when it hops from our top level domain to a subdomain through a link.

I've always felt like mapping out all of the off limits places for robots.txt is giving away a roadmap to anyone who wants to know where the good stuff is at.

What type of Robots.txt security strategies would be best for a webapp environment?

Improve this question
  • 3
    there are no robots.txt security strategies, robots.txt has nothing to do with security. – ordag Jan 31 '12 at 17:11
11

You are incorrect in your assumption that robots.txt on subdomains are ignored. Most search engines will grab and obey the robots.txt for individual sub-domains.

If you wish you can instruct spiders not to index elements of your website without listing them in robots.txt, through either:

  1. Adding <meta name="robots" content="noindex" /> to the HTML of the web pages
  2. Adding X-Robots-Tag: noindex to your HTTP headers

More details available here

However, regardless of this you should not be relying upon robots.txt for security. What you are leaning towards here is security through obscurity which is widely regarded as a bad idea.

Improve this answer
  • Bad idea indeed, but it can be used to add a layer of security: the algorithm/method used would be uncommon and possibly harder to guess than a password (like changing the protocols, especially ciphered ones, think of it like using a salt on hashing algorithms). – Aki Feb 5 '12 at 12:25
10

Security wise, robots.txt usage has two rules.

  1. Do not try to implement any security through robots.txt. The robots file is nothing more than a kind suggestion, and while most search engine crawlers respect it, malicious crawlers have a good laugh and continue at their business. If it's linked to, it can be found.
  2. Do not expose interesting information through robots.txt. Specifically, if you rely on the URL to control access to certain resources (which is a huge alarm bell by itself), adding it to robots.txt will only make the problem worse: an attacker who scans robots.txt will now see the secret URL you were trying to hide, and concentrate efforts on that part of your site (you don't want it indexed, and it's named 'sekrit-admin-part-do-not-tell-anyone', so it's probably interesting).

So, by all means, use robots.txt to tell search engines which parts of your site you want them to index, and when to revisit them, but never use it for security. If you have things to hide, use actual protection (and if you don't want something to show up in search engine results, chances are you should be password-protecting it anyway).

Improve this answer
6

You should never rely on robots.txt to offer you any kind of discretion or security.

Sure, the big engines will respect it, but anyone can write a crawler and find "the good stuff", as you called it.

If there is a resource on your webserver that you don't want everyone to be able to access, you should restrict the permissions using .htaccess or a similar mechanism, depending on the server.

Improve this answer
  • htaccess rules and mod_security is already rolled out. This isn't so much about beating those that have the means to crawl, but rather just tuning the google/yandex/bing/etc bots to avoid having something added to an easy google dork, and thus having it sweep my stuff as well. – neil Jan 31 '12 at 7:30
2

As Kenny said, don't rely on robots.txt for security. If you don't want a page to be indexed, you have three options (if the crawler follows the guidelines, which some won't):

  1. add a rel=nofollow,noindex to link tags that a crawler shouldn't follow or index.
  2. Add a robots.txt to each domain, and set the robots.txt on subdomains to deny /
  3. Add the header tag <meta name="robots" content="noindex,nofollow" /> to each page that the crawlers shouldn't index.
  4. (optional) There is also a HTTP header X-Robots-Tag: noindex,nofollow which does the same as the header tag.

There will be crawlers that ignore this, but the big ones should follow these rules and not index these pages.

Improve this answer
0

As everyone has mentioned, your robots.txt file does not provide a mechanism for security, and if you want to tell crawlers NOT to index your subdomains, you can use individual robots.txt files for each subdomain. Here's what one of those robots.txt files would look like:

User-agent: *
Disallow: /

For more information about robots.txt, here are a few resources:

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.