Is there a way to basically have an allow all policy on your top level domain, and any subdomains disallow all? I would rather that my public facing app servers not get indexed, but from what I can tell, no robots are even polling for robots.txt when it hops from our top level domain to a subdomain through a link.
I've always felt like mapping out all of the off limits places for robots.txt is giving away a roadmap to anyone who wants to know where the good stuff is at.
What type of Robots.txt security strategies would be best for a webapp environment?