-1

robots.txt is a file that contains path which cannot be crawled by bot most of time search-engine bots like Google bot, etc. It tells search-engine that this directory is private and cannot be crawled by them.

Here is the robots.txt file of a site . http://sppp.rajasthan.gov.in:

User-Agent: *
Disallow: 
Allow: /

Which I simply get by http://sppp.rajasthan.gov.in/robots.txt.

Is there any risk in using the same?

Secondly, If I trying to search within the website using Google by putting in the keyword site:sppp.rajasthan.gov.in I get only few results. So, if robots.txt of this particular site doesn't have any restriction, why all the pages within the site are not indexed and shown in Google results?

2
  • 3
    The question about google not indexing all your sub-pages is best addressed to webmasters.stackexchange.com
    – Philipp
    Sep 18, 2015 at 11:55
  • The second question is completely off-topic here (it isn't about security) and the first question, "is there any risk" is far too broad. What risks are you concerned about?
    – schroeder
    Sep 20, 2015 at 3:03

1 Answer 1

1

Although it doesn't make sense what's written in this specific robots.txt, I do not see a security risk in having a robots.txt in general.

The web application / website should be secure enough, even if sensitive directories are revealed by robots.txt as security by obscurity should be avoided at all times.

In other words, even if I as an attacker know all your sensitive locations, I should not be able to download or view any sensitive information.

6
  • @Jeoron The above mentioned robots.txt should allow the search engine to crawl everything on this website as nothing has been disallowed.But then why site:sppp.rajasthan.gov.in results in very fewer results Sep 18, 2015 at 11:48
  • 3
    @SudarshanTaparia This has nothing to do with it security.
    – Philipp
    Sep 18, 2015 at 11:56
  • @SudarshanTaparia robots.txt can't prevent any search engine from doing anything. It's just a guideline for good-mannered engines, and malware can just ignore it.
    – deviantfan
    Sep 18, 2015 at 15:13
  • @deviantfan I know that.. what I want to know is if a robots.txt has no restrictions, then why a search engine like google(good search engine) hasnt crawled the entire website and indexed the same.Name of the site is sppp.rajasthan.gov.in when i do site:sppp.rajasthan.gov.in in google I get very less result Sep 18, 2015 at 16:10
  • @SudarshanTaparia Google has no obligation to index every page on the site though. A lot of things could also have happened between google crawling and your retrieval of robots.txt. For all we know, the server may serve a different version of the file depending on the user agent.
    – billc.cn
    Sep 18, 2015 at 16:44

Not the answer you're looking for? Browse other questions tagged .