robots.txt is a file that contains path which cannot be crawled by bot most of time search-engine bots like Google bot, etc. It tells search-engine that this directory is private and cannot be crawled by them.

Here is the robots.txt file of a site . http://sppp.rajasthan.gov.in:

User-Agent: *
Allow: /

Which I simply get by http://sppp.rajasthan.gov.in/robots.txt.

Is there any risk in using the same?

Secondly, If I trying to search within the website using Google by putting in the keyword site:sppp.rajasthan.gov.in I get only few results. So, if robots.txt of this particular site doesn't have any restriction, why all the pages within the site are not indexed and shown in Google results?

closed as too broad by user45139, Question Overflow, StackzOfZtuff, paj28, schroeder Sep 20 '15 at 3:04

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 3
    The question about google not indexing all your sub-pages is best addressed to webmasters.stackexchange.com – Philipp Sep 18 '15 at 11:55
  • The second question is completely off-topic here (it isn't about security) and the first question, "is there any risk" is far too broad. What risks are you concerned about? – schroeder Sep 20 '15 at 3:03

Although it doesn't make sense what's written in this specific robots.txt, I do not see a security risk in having a robots.txt in general.

The web application / website should be secure enough, even if sensitive directories are revealed by robots.txt as security by obscurity should be avoided at all times.

In other words, even if I as an attacker know all your sensitive locations, I should not be able to download or view any sensitive information.

  • @Jeoron The above mentioned robots.txt should allow the search engine to crawl everything on this website as nothing has been disallowed.But then why site:sppp.rajasthan.gov.in results in very fewer results – Sudarshan Taparia Sep 18 '15 at 11:48
  • 2
    @SudarshanTaparia This has nothing to do with it security. – Philipp Sep 18 '15 at 11:56
  • @SudarshanTaparia robots.txt can't prevent any search engine from doing anything. It's just a guideline for good-mannered engines, and malware can just ignore it. – deviantfan Sep 18 '15 at 15:13
  • @deviantfan I know that.. what I want to know is if a robots.txt has no restrictions, then why a search engine like google(good search engine) hasnt crawled the entire website and indexed the same.Name of the site is sppp.rajasthan.gov.in when i do site:sppp.rajasthan.gov.in in google I get very less result – Sudarshan Taparia Sep 18 '15 at 16:10
  • @SudarshanTaparia Google has no obligation to index every page on the site though. A lot of things could also have happened between google crawling and your retrieval of robots.txt. For all we know, the server may serve a different version of the file depending on the user agent. – billc.cn Sep 18 '15 at 16:44

Not the answer you're looking for? Browse other questions tagged or ask your own question.