robots.txt
is a file that contains path which cannot be crawled by bot most of time search-engine bots like Google bot, etc. It tells search-engine that this directory is private and cannot be crawled by them.
Here is the robots.txt
file of a site . http://sppp.rajasthan.gov.in:
User-Agent: *
Disallow:
Allow: /
Which I simply get by http://sppp.rajasthan.gov.in/robots.txt.
Is there any risk in using the same?
Secondly, If I trying to search within the website using Google
by putting in the keyword site:sppp.rajasthan.gov.in
I get only few results.
So, if robots.txt
of this particular site doesn't have any restriction, why all the pages within the site are not indexed and shown in Google results?