1

During a recent scan of my website, a robots.txt file was found and in it contains

Disallow: /subsite1/
Disallow: /subsite2/

In order not to show any paths in robots.txt, as I am afraid that hackers can try to hack the disallowed directories, what can I do to secure robots.txt and at the same time allowing search engines to use it for legitimate purposes?

Improve this question

2 Answers 2

Reset to default
11

Robots.txt is not, in any way, supposed to secure your site. The only purpose of the robots.txt file is to inform polite search engine bots what areas do not contain interesting information.

If you have sensitive areas in your robots.txt file, then you seriously need to review the architecture of your site. No sensitive directories should ever be accessible to the public in any way. In fact, anything exposed to the public should be secured, whether it is visible or not.

Improve this answer
2
  • hi thanks. So what we can do is just to secure /subsite1/ and /subsite2/ properly , e.g checking directory access rights , config files access rights etc? The robots.txt can just leave it be? thanks
    – JinPangPang
    Apr 14, 2015 at 4:48
  • 2
    @PangSerLark: sensitive parts of the site should be protected, so they can only be accessed by authorized users. This means passwords or similar and not just hoping that nobody will find out the name of the URL. It's the same as doors: protect them with a lock instead of hoping that nobody will see the door.
    – Steffen Ullrich
    Apr 14, 2015 at 5:02
3

Malicious users might use the robots.txt file during the information gathering phase of an attack. Having entries in it is not a vulnerability in and of itself, it's just a way for someone to find parts of the site that might not be findable other ways.

If simply knowing that a part of your site exists, results in a security vulnerability then you should have someone who has a firm grasp of web security take a look.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .