Robots.txt security strategy?

Question

Is there a way to basically have an allow all policy on your top level domain, and any subdomains disallow all? I would rather that my public facing app servers not get indexed, but from what I can tell, no robots are even polling for robots.txt when it hops from our top level domain to a subdomain through a link.

I've always felt like mapping out all of the off limits places for robots.txt is giving away a roadmap to anyone who wants to know where the good stuff is at.

What type of Robots.txt security strategies would be best for a webapp environment?

Answer 1

You are incorrect in your assumption that robots.txt on subdomains are ignored. Most search engines will grab and obey the robots.txt for individual sub-domains.

If you wish you can instruct spiders not to index elements of your website without listing them in robots.txt, through either:

1. Adding <meta name="robots" content="noindex" /> to the HTML of the web pages
2. Adding X-Robots-Tag: noindex to your HTTP headers

More details available here

However, regardless of this you should not be relying upon robots.txt for security. What you are leaning towards here is security through obscurity which is widely regarded as a bad idea.

Answer 2

Security wise, robots.txt usage has two rules.

1. Do not try to implement any security through robots.txt. The robots file is nothing more than a kind suggestion, and while most search engine crawlers respect it, malicious crawlers have a good laugh and continue at their business. If it's linked to, it can be found.
2. Do not expose interesting information through robots.txt. Specifically, if you rely on the URL to control access to certain resources (which is a huge alarm bell by itself), adding it to robots.txt will only make the problem worse: an attacker who scans robots.txt will now see the secret URL you were trying to hide, and concentrate efforts on that part of your site (you don't want it indexed, and it's named 'sekrit-admin-part-do-not-tell-anyone', so it's probably interesting).

So, by all means, use robots.txt to tell search engines which parts of your site you want them to index, and when to revisit them, but never use it for security. If you have things to hide, use actual protection (and if you don't want something to show up in search engine results, chances are you should be password-protecting it anyway).

Answer 3

You should never rely on robots.txt to offer you any kind of discretion or security.

Sure, the big engines will respect it, but anyone can write a crawler and find "the good stuff", as you called it.

If there is a resource on your webserver that you don't want everyone to be able to access, you should restrict the permissions using .htaccess or a similar mechanism, depending on the server.

Answer 4

As Kenny said, don't rely on robots.txt for security. If you don't want a page to be indexed, you have three options (if the crawler follows the guidelines, which some won't):

1. add a rel=nofollow,noindex to link tags that a crawler shouldn't follow or index.
2. Add a robots.txt to each domain, and set the robots.txt on subdomains to deny /
3. Add the header tag <meta name="robots" content="noindex,nofollow" /> to each page that the crawlers shouldn't index.
4. (optional) There is also a HTTP header X-Robots-Tag: noindex,nofollow which does the same as the header tag.

There will be crawlers that ignore this, but the big ones should follow these rules and not index these pages.

Answer 5

As everyone has mentioned, your robots.txt file does not provide a mechanism for security, and if you want to tell crawlers NOT to index your subdomains, you can use individual robots.txt files for each subdomain. Here's what one of those robots.txt files would look like:

User-agent: *
Disallow: /

For more information about robots.txt, here are a few resources:

• Block or remove pages using a robots.txt file
• A robots.txt File Guide That Won’t Put You to Sleep