If I go to http://sppp.rajasthan.gov.in/robots.txt I get:

User-Agent: *
Allow: /

That means that crawlers are allowed to fully access the website and index everything. If so then why site:sppp.rajasthan.gov. in on Google Search shows me only few pages whereas it contains lots of documents including PDF files?

closed as off-topic by tlng05, Scott Pack, Xander, Stephane, M'vy Jul 20 '15 at 12:07

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question does not appear to be about Information security within the scope defined in the help center." – tlng05, Scott Pack, Xander, Stephane, M'vy
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    You may want to read what is on-topic here. This question has nothing to do with security. – Anonymous Jul 19 '15 at 18:37
  • Robots exclusion standard protocol is also a security issue of course :) – user45139 Jul 20 '15 at 5:22

By writing:

Allow: /

You are saying to the robots: everything is allowed and nothing is disallowed. This may be confusing for robots.

I suggest you to write those rules of Robots Exclusion Protocol clearer by changing the content of your robots.txt file to:

User-agent: *


User-agent: *
Allow: /

This way you are allowing robots to index everything.

Not the answer you're looking for? Browse other questions tagged or ask your own question.