7

While scanning my website with uniscan it found my robots.txt file which disallows access to /cgi-bin/ and other directories, but they are not accessible in browser. Is there a way to access the directories or files which are Disallowed?

Improve this question
4
  • 2
    I hope you have other restrictions on cgi-bin - like .htaccess rules and suitable permissions set on the files in that folder.
    – HorusKol
    May 1, 2014 at 0:44
  • Also, your robots.txt file can contain entries that don't really exist so you might think security is stopping you from reading a url that really just isn't there.
    – David
    May 1, 2014 at 3:56
  • 2
    "but they are not accessible in browser" - If they are not accessible then either they don't exist, or there is something else that is blocking them. "robots.txt" is only an advisory document for good bots.
    – MrWhite
    May 1, 2014 at 11:20
  • 2
    robots.txt is not a security feature, it's not even a set of rules, it's just a "suggestion" really
    – MMM
    May 1, 2014 at 11:38

3 Answers 3

Reset to default
32

The robots.txt does not disallow you to access directories. It tells Google and Bing not to index certain folders. If you put secret folders in there, Google and Bing will ignore them, but other malicious scanners will probably do the opposite. In effect you're giving away what you want to keep secret. To disallow folders you should set this in Apache vhost or .htaccess. You can set a login on the folder if you want.

Improve this answer
4
  • 3
    As a side note; It is possible to build a trap page in robots.txt, to stop bad crawlers. Ref.: bsdguides.org/2012/a-simple-bot-trap-using-robots-txt - But this is not effective against humans, as they can comprehend that they triggered a trap, and continue the list with a new ip-address.
    – Dog eat cat world
    Apr 30, 2014 at 14:12
  • Nice tip, especially the fail2ban trick!
    – SPRBRN
    Apr 30, 2014 at 14:51
  • 1
    "not to index certain folders" - actually, it tells the good bots not to crawl certain files and folders. Disallowed pages can still be indexed (link only), particularly if other pages are linking to them.
    – MrWhite
    May 1, 2014 at 11:19
  • I wonder how this works. You could put a page inside such a folder with its own robot metatag to allow indexing, and I can imagine that has precedence. But I guess that if I link to a page in a folder with a disallow-index setting, that page should not be indexed. Google should follow the robots.txt guide, and the only exception should be a metatag in the page itself.
    – SPRBRN
    May 1, 2014 at 12:23
10

The robots.txt file isn't a security measure and has no incidence on access permission. This file only tells 'good' robots to skip a part of your website to avoid indexation. Bad robots don't even abide by those rules and scan all they can find. So security can never rely on the robots.txt file (that's not its purpose).

Is there a way to access the directories or files which are Disallowed?

Check your webserver's permissions.

Improve this answer
3

Yes, just go to the directory.

robots.txt is nothing more than something a crawler follows out of courtesy. Crawlers are free to entirely ignore the file. It isn't a real method to keep directories private.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .