Why can't search engines such as Google, Yahoo and Bing reach dark web and archive its content to display it in their results?
Assuming you are talking about Tor hidden services, then the answer is they can, but only indirectly. There are various "portal" sites which provide a gateway to hidden services. These gateways are normal websites with regular domains, but are running tor2web software, which uses the Tor client to relay traffic between non-Tor and Tor users (but note that they provide no anonymity). These can be indexed at will.
There are several reasons why Tor hidden services are not indexed frequently:
- As someone else pointed out earlier, it's very disjointed. Very few sites link to each other, limiting crawlers' ability to find new sites and new pages. It's like the open internet from the 90s.
- It uses its own protocol, so without portal/gateway sites, they would be unable to connect. Try connecting to a
.oniondomain in a normal browser. You'll see it won't even resolve.
- There are not that many hidden services out there. The myth that it is "vast" is ill-founded, based on a misunderstanding of the terminology. In reality, it is really quite small.
- Some sites are blocked by portal/gateway sites for legal reasons, so they can only be accessed using the Tor protocol. As search engine crawlers don't use this, they can't access the sites.
There is no single "database" of hidden services as there is for regular domains (root nameservers). A hidden service is based on an encoded, truncated hash of the server's public key. The client uses the service's domain name and looks up the hidden service's descriptor in a semi-public database, which contains its public key and a list of Introduction Points (relays chosen by the server). The client selects a random relay as a Rendezvous Point and sends that relay's ID to the hidden service over the Introduction Point. The server and client then meet through the Rendezvous Point over their own three-hop circuits.
Through a complex protocol, the client and server thus manage to form a connection without either of them needing to reveal their real IP. Because there is no IP address that the domain resolves to, a regular search engine can't reach it using standard HTTP with TCP/IP. In order for a search engine crawler to connect to these sites, it would have to use this protocol. That is not very practical for them.