1

During a recent scan of my website, a robots.txt file was found and in it contains

Disallow: /subsite1/
Disallow: /subsite2/

In order not to show any paths in robots.txt, as I am afraid that hackers can try to hack the disallowed directories, what can I do to secure robots.txt and at the same time allowing search engines to use it for legitimate purposes?

11

Robots.txt is not, in any way, supposed to secure your site. The only purpose of the robots.txt file is to inform polite search engine bots what areas do not contain interesting information.

If you have sensitive areas in your robots.txt file, then you seriously need to review the architecture of your site. No sensitive directories should ever be accessible to the public in any way. In fact, anything exposed to the public should be secured, whether it is visible or not.

  • hi thanks. So what we can do is just to secure /subsite1/ and /subsite2/ properly , e.g checking directory access rights , config files access rights etc? The robots.txt can just leave it be? thanks – Pang Ser Lark Apr 14 '15 at 4:48
  • 2
    @PangSerLark: sensitive parts of the site should be protected, so they can only be accessed by authorized users. This means passwords or similar and not just hoping that nobody will find out the name of the URL. It's the same as doors: protect them with a lock instead of hoping that nobody will see the door. – Steffen Ullrich Apr 14 '15 at 5:02
3

Malicious users might use the robots.txt file during the information gathering phase of an attack. Having entries in it is not a vulnerability in and of itself, it's just a way for someone to find parts of the site that might not be findable other ways.

If simply knowing that a part of your site exists, results in a security vulnerability then you should have someone who has a firm grasp of web security take a look.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.