robots.txt is a file that contains path which cannot be crawled by bot most of time search-engine bots like Google bot, etc. It tells search-engine that this directory is private and cannot be crawled by them.
Here is the
robots.txt file of a site . http://sppp.rajasthan.gov.in:
User-Agent: * Disallow: Allow: /
Which I simply get by http://sppp.rajasthan.gov.in/robots.txt.
Is there any risk in using the same?
Secondly, If I trying to search within the website using Google
by putting in the keyword
site:sppp.rajasthan.gov.in I get only few results.
robots.txt of this particular site doesn't have any restriction, why all the pages within the site are not indexed and shown in Google results?