Is it a good idea to preventive use a robots.txt that disallows all search engines to index query strings in order to slow down reconnaissance?

Question

I know that robots.txt is not a security feature. But, indexed and cached information by search engines might be incredibly helpful in the reconnaissance phase of a penetration test and potentially for automated vulnerability scanners.

Let alone the fact that it's better not to expose query string, in source-code nor in a robots.txt file or anywhere else, in some cases to disallow search engines to cache content and that it's possible to guess (read: brute-force or check common) parameters.

Now, is it a good idea to preventive use something like the following robots.txt file:

User-agent: *
Disallow: /*?*
Disallow: /*&*

In case the source-code does show a reference to parameters in the URL you disallow all search engines specifically to index them. Then, finding the parameters in the reconnaissance phase is still possible (but slightly harder) but at least when that is fixed, the stuff won't be cached.

After content is cached, indexed and measures are taken to hide the query parameters (like rewriting URLs, the previously used parameters should be renamed in order to prevent them from being easily found or misused.

My point is, everything can be hacked, it's just a matter of resources (time, technically, effort and skills). Making things too easy will allow a attacker to use less resources. When the parameters in URLs are not indexed, it takes more time to find them since the source-code has to be inspected or site has to be crawled. So, is some kind of robots.txt like above, a do or a don't and why?

Last but not least, should we recommend developers (or is it a good practice) to do so?

Answer 1

No, this adds no real security value.

Depending on the approach of your attacker, the first thing he might do is fingerprint the server and libs and do a crawl of the whole site himself (that's what I would do if the point is to find a vulnerability in the code).

Crawling a site like this is extremely trivial (and easily automated), so the gain is negligible. Also, in the afore-mentioned scenario, I would not use search engine data as I would want to be sure that the stuff I'm working on is "the real deal", not a cached version from some hours, days or whenever ago.

Finding parameters is 0.001% of the job. It is exploiting them that is harder. Slowing hackers on that task has no added value.

(Also, you would lose value SEO-wise but I understand that this is not your concern)