I know that
robots.txt is not a security feature. But, indexed and cached information by search engines might be incredibly helpful in the reconnaissance phase of a penetration test and potentially for automated vulnerability scanners.
Let alone the fact that it's better not to expose query string, in source-code nor in a
robots.txt file or anywhere else, in some cases to disallow search engines to cache content and that it's possible to guess (read: brute-force or check common) parameters.
Now, is it a good idea to preventive use something like the following
User-agent: * Disallow: /*?* Disallow: /*&*
In case the source-code does show a reference to parameters in the URL you disallow all search engines specifically to index them. Then, finding the parameters in the reconnaissance phase is still possible (but slightly harder) but at least when that is fixed, the stuff won't be cached.
After content is cached, indexed and measures are taken to hide the query parameters (like rewriting URLs, the previously used parameters should be renamed in order to prevent them from being easily found or misused.
My point is, everything can be hacked, it's just a matter of resources (time, technically, effort and skills). Making things too easy will allow a attacker to use less resources. When the parameters in URLs are not indexed, it takes more time to find them since the source-code has to be inspected or site has to be crawled. So, is some kind of
robots.txt like above, a do or a don't and why?
Last but not least, should we recommend developers (or is it a good practice) to do so?