1

If I had legitimate reasons to do so, would it be possible to hide a service from webcrawlers and with that, hide it from any search engines like Google?

Can I prevent an average joe from stumbling across my service by just searching for it, and only have people access that service, if they know that it exists and how to access it?

2

As forest@ suggested, you can use a robots exclusion file - which like he also said only compliant web crawlers will follow. But, this is like opening your house for everybody to visit and putting the sign "please don't look inside" on the open door to the bedroom . Guess what, there are visitors who are curious why they should not look and will exactly do this.

Thus, it is only a way to deter polite visitors like the major search engines. If this is enough for you then you might go for it.

... and only have people access that service, if they know that it exists and how to access it

This can be realized by either having a non-public link or by requiring more than just the link - i.e. make the door to the bedroom invisible or just lock the door with a key.

A non-public link can be simply realized by creating some file or directory which is not linked from somewhere else on your page (and also should not be visible in some automatically generated directory index - i.e. switch these off). Crawlers which don't know up-front about this link will not be able to crawl it since it is not linked from any place they've visited so far and thus they don't know it exist. Of course, the link must stay non-public for this to work, thus make sure that nobody publishes it.

Much better instead is to require a secret additionally to the link itself. This can be done by require authentication for a directory and setup some users, then give everybody not only the link but also the (maybe personalized) login credentials. Even if crawlers know the link they don't have the credentials and can thus not access it. There are numerous resources on how to setup authentication, like Password protect a directory using basic authentication.

  • It's not necessary to go to this extreme if you just want to prevent the "average joe" from finding a webpage via a Google search. If true authentication is necessary then that makes sense, but otherwise this seems overkill for something that robots.txt is designed for. +1 anyway though for providing an alternate solution that works for a wider variety of threat models. – forest Sep 1 '18 at 20:55
  • @forest: I agree with you but I thought I've made it clear enough in my answer when robots.txt is sufficient and when not: "Thus, it is only a way to deter polite visitors like the major search engines. If this is enough for you then you might go for it.". Personally I would not feel comfortable with just asking nicely to not look. Too much can go wrong with this, like the crawler not being polite or me accidentally messing up the syntax of robots.txt . Explicitly making access impossible is much clearer and also easier to test if it really works. And no way for the crawler around it. – Steffen Ullrich Sep 2 '18 at 0:14
3

This is exactly what the robots exclusion standard is for. You put a text file at the well known path /robots.txt, and compliant web crawlers will obey it. In particular, you can exclude specific crawlers, or exclude specific paths and resources from being crawled. This will effectively make your website "invisible" to standard search engines. Of course, spambots won't necessarily obey the exclusion policies you set up, so it will not protect you from misbehaving bots.

To prevent any bot from crawling any page on your site, you would use:

User-agent: *
Disallow: /

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.