How do people make their websites unsearchable through google ect.?

People commonly refer to the "deep web" as the class of websites that can't be found through the major search engines. They use really convoluted URLs that don't make any sense at face value, but my understanding is that google also combs through the content of web pages, so black market websites ect. should pop up this way since people very commonly search for information on drugs ect. and all the "sketchy" (and non-sketchy) stuff on the deep web.

What am I missing here?

• en.wikipedia.org/wiki/Deep_web – schroeder Jan 27 '17 at 14:54
• I find it very interesting that this thread producEd so many different answers and interesting discussion but was downvoted 3 times – thinksinbinary Jan 29 '17 at 17:05
• I would imagine it is downvoted as it is answerable by one simple Wikipedia search, as Schroeder linked - it's not a good fit here. The fact that it can be answered is separate to whether the question should have been asked. – Rory Alsop Feb 2 '17 at 8:34
• Should have been asked? "I would imagine it is downvoted as it is answerable by one simple Wikipedia search, as Schroeder linked" but there were several different answers, and the point of having a Q&A database is so that when people search for help on a topic, they pop up. So while I appreciate constructive criticism, I use the internet to easily find information, and as long as im not spamming message boards i think your "should" is completely irrelevant. Thank you. – thinksinbinary Feb 2 '17 at 23:02

People commonly refer to the "deep web" as the class of websites that can't be found through the major search engines. They use really convoluted URLs that don't make any sense at face value,

You're conflating a couple of ideas here. You might want to read Clearing Up Confusion - Deep Web vs. Dark Web.

The "deep web" consists of those web pages that aren't indexed in search engines. You need to know how to get there to get them, or follow links in a web page. This is most simply caused by the robots convention, as @OscarAkaElvis points out.

Note the similarity between "deep web" and "deep linking" - deep linking means providing direct links to pages buried within another server's hierarchy; the hosting server may have blocked robots in order to drive traffic through their front pages (and ads, usually). "Deep links" allow people to bypass that flow, and there's been legal action back and forth about doing it.

The "dark web" consists of web pages that are even further buried and, yes, do rely to some extent upon Tor's hidden service addresses ("convoluted URLs that don't make any sense"). These are also likely protected via robots, but since robots is a voluntary protocol, it can be ignored by anyone who cares. Google and Bing aren't going to have good search data on "dark web" sites, because they respect robots. The FBI and DEA, on the other hand, probably have a very comprehensive search engine which doesn't respect robots.

There's an excellent article called It's About To Get Even Easier to Hide on the Dark Web which talks about Tor hidden services, how they're not that hidden, and how the next generation of hidden services is going to get even trickier:

The next generation of hidden services will use a clever method to protect the secrecy of those addresses. Instead of declaring their .onion address to hidden service directories, they’ll instead derive a unique cryptographic key from that address, and give that key to Tor’s hidden service directories. Any Tor user looking for a certain hidden service can perform that same derivation to check the key and route themselves to the correct darknet site. But the hidden service directory can’t derive the .onion address from the key, preventing snoops from discovering any secret darknet address. “The Tor network isn’t going to give you any way to learn about an onion address you don’t already know,” says Mathewson.

add a robots.txt file on your root web dir with this content:

User-agent: *
Disallow: /


You have documentation here about robots.txt file.

They use really convoluted URL

I think you mean the ID-like url of TOR hidden services. This is actually a public key. The other answer mentions the robots.txt, but this is just asking the search robot nicely not to look any further. It doesn't offer any actual protection.

TOR hidden services work by letting everyone know they exist, but not where they are. That way someone could potentially host illegal content, and everyone could see it via TOR without knowing where the actual machine is. This security is based on cryptography, and does not depend on the search bot obeying the standards.

For more on how this is done, see the docs here: https://www.torproject.org/docs/hidden-services

You can put Header set X-Robots-Tag "noindex, nofollow" in your .htaccess or apache config file