Take the 2-minute tour ×
Information Security Stack Exchange is a question and answer site for Information security professionals. It's 100% free, no registration required.

While scanning my website with uniscan it found my robots.txt file which disallows access to /cgi-bin/ and other directories, but they are not accessible in browser. Is there a way to access the directories or files which are Disallowed?

share|improve this question
1  
I hope you have other restrictions on cgi-bin - like .htaccess rules and suitable permissions set on the files in that folder. –  HorusKol May 1 at 0:44
    
Also, your robots.txt file can contain entries that don't really exist so you might think security is stopping you from reading a url that really just isn't there. –  David May 1 at 3:56
    
"but they are not accessible in browser" - If they are not accessible then either they don't exist, or there is something else that is blocking them. "robots.txt" is only an advisory document for good bots. –  w3d May 1 at 11:20
    
robots.txt is not a security feature, it's not even a set of rules, it's just a "suggestion" really –  MMM May 1 at 11:38

3 Answers 3

up vote 26 down vote accepted

The robots.txt does not disallow you to access directories. It tells Google and Bing not to index certain folders. If you put secret folders in there, Google and Bing will ignore them, but other malicious scanners will probably do the opposite. In effect you're giving away what you want to keep secret. To disallow folders you should set this in Apache vhost or .htaccess. You can set a login on the folder if you want.

share|improve this answer
3  
As a side note; It is possible to build a trap page in robots.txt, to stop bad crawlers. Ref.: bsdguides.org/2012/a-simple-bot-trap-using-robots-txt - But this is not effective against humans, as they can comprehend that they triggered a trap, and continue the list with a new ip-address. –  Dog eat cat world Apr 30 at 14:12
    
Nice tip, especially the fail2ban trick! –  SPRBRN Apr 30 at 14:51
    
"not to index certain folders" - actually, it tells the good bots not to crawl certain files and folders. Disallowed pages can still be indexed (link only), particularly if other pages are linking to them. –  w3d May 1 at 11:19
    
I wonder how this works. You could put a page inside such a folder with its own robot metatag to allow indexing, and I can imagine that has precedence. But I guess that if I link to a page in a folder with a disallow-index setting, that page should not be indexed. Google should follow the robots.txt guide, and the only exception should be a metatag in the page itself. –  SPRBRN May 1 at 12:23

The robots.txt file isn't a security measure and has no incidence on access permission. This file only tells 'good' robots to skip a part of your website to avoid indexation. Bad robots don't even abide by those rules and scan all they can find. So security can never rely on the robots.txt file (that's not its purpose).

Is there a way to access the directories or files which are Disallowed?

Check your webserver's permissions.

share|improve this answer

Yes, just go to the directory.

robots.txt is nothing more than something a crawler follows out of courtesy. Crawlers are free to entirely ignore the file. It isn't a real method to keep directories private.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.