0

I know that robots.txt is not a security feature. But, indexed and cached information by search engines might be incredibly helpful in the reconnaissance phase of a penetration test and potentially for automated vulnerability scanners.

Let alone the fact that it's better not to expose query string, in source-code nor in a robots.txt file or anywhere else, in some cases to disallow search engines to cache content and that it's possible to guess (read: brute-force or check common) parameters.

Now, is it a good idea to preventive use something like the following robots.txt file:

User-agent: *
Disallow: /*?*
Disallow: /*&*

In case the source-code does show a reference to parameters in the URL you disallow all search engines specifically to index them. Then, finding the parameters in the reconnaissance phase is still possible (but slightly harder) but at least when that is fixed, the stuff won't be cached.

After content is cached, indexed and measures are taken to hide the query parameters (like rewriting URLs, the previously used parameters should be renamed in order to prevent them from being easily found or misused.

My point is, everything can be hacked, it's just a matter of resources (time, technically, effort and skills). Making things too easy will allow a attacker to use less resources. When the parameters in URLs are not indexed, it takes more time to find them since the source-code has to be inspected or site has to be crawled. So, is some kind of robots.txt like above, a do or a don't and why?

Last but not least, should we recommend developers (or is it a good practice) to do so?

  • Why would you not just use the standard: User-agent: * Disallow: / ? Am I missing the point? – Ian Jul 14 '16 at 9:41
  • @Ian you are right. Disallow: /*?* should be enough. The asterisks are working as a wildcard. And /*&* should work for malformed queries starting with &. I will edit my question. – Bob Ortiz Jul 14 '16 at 9:54
  • The thing is, seo or not. You don't want parameters indexed. And by default most sites are crawled and cached. In case the site removed the parameters already (so with recrawling they won't be found), they are still cached. While this robots file could prevent those from being cached in the first place. That's why I posted the question in InfoSec and not in Webmasters or Seo related site. I'm wondering about the security perspective of it, not the seo aspect. – Bob Ortiz Jul 14 '16 at 11:09
  • While it might not solve the problem you are trying to solve, it might solve a different one, namely search engine robots performing actions on your website you don't want them to perform. – Philipp Jul 14 '16 at 11:57
  • 1
    So what would you do when someone actively uses a link checker/translater/etc which will retrieve the content from your site and it's robots.txt allows said page to be indexed. Or if someone posts your html source on pastebin? Parameter names and values (especially in GET urls) aren't designed to be secrets and if your require that you may want to rethink your design. Maybe use encrypted query strings instead? – wireghoul Jul 14 '16 at 12:05
4

No, this adds no real security value.

Depending on the approach of your attacker, the first thing he might do is fingerprint the server and libs and do a crawl of the whole site himself (that's what I would do if the point is to find a vulnerability in the code).

Crawling a site like this is extremely trivial (and easily automated), so the gain is negligible. Also, in the afore-mentioned scenario, I would not use search engine data as I would want to be sure that the stuff I'm working on is "the real deal", not a cached version from some hours, days or whenever ago.

Finding parameters is 0.001% of the job. It is exploiting them that is harder. Slowing hackers on that task has no added value.

(Also, you would lose value SEO-wise but I understand that this is not your concern)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.