Consider the following theoretical webserver:

On any request, produce an html response that consists of 2 links, which are each randomly chosen strings followed by ".html".

When the google bot attempts to crawl this site it will be presented with 2 more links. In the interest of mapping the site, it will attempt to navigate to them, each returning 2 more links. This will obviously happen infinitely(surely can come up with a solution that never gives the same page name twice).

My question is, do the bots have a mechanism to detect this? If so, what is it? If not, what kind of detection methods would be possible for this situation?

Not entirely sure if this was the right SE site to post this.

closed as too broad by Steve, Xander, tylerl, Adi, TildalWave Feb 19 '14 at 21:07

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • google bot, and others that are sensibly written, will have a link depth limit. But honeypots like this can be quite useful for other bots. – Rory Alsop Feb 19 '14 at 20:03
  • So this means that if my webpage happens to have heavily nested links naturally, that some may get missed? Do you know what that is approximately? Also, if it's only a link depth limit, could I produce a million links on the page? Then even a link depth of 3 would be huge – Cruncher Feb 19 '14 at 20:07
  • Frankly, this question is unanswerable without actually knowing how the bots function, and that is (I'm assuming) a trade secret very closely guarded by Google. Odds are pretty good they've got some wildly complex code to guard against this sort of thing. – Steve Feb 19 '14 at 20:18
  • Even if this problem wasn't already solved, the power of google's crawlers is probably such that your server would crash trying to produce domain names long before Google even started getting tired... – KnightOfNi Feb 19 '14 at 21:22

No. The number of pages Google (and any major search bot) is willing to crawl on your domain (or indeed whether they are willing to crawl it at all) is based on how relevant they think your domain is.

There are plenty of sites with an infinite number of pages. This problem was solved decades ago.

  • How does the bot determine relevance? Surely they don't spend manual man power on that? – Cruncher Feb 19 '14 at 20:44
  • 1
    They determine based on the relevance of inbound links, among other things. It's the core of what they do, and they spend vast amounts of manpower on it. – tylerl Feb 19 '14 at 21:30

Even without special engineering, certain site architectures can have long recursive paths - certain wikis, for instance. Any decently written bot should be able to cope with such site behaviour, at the very least having a recursion depth limit. I don't think anyone here is going to be able to answer concretely, since none of us have access to google's software directly, but I imagine google's bot is at least clever enough to avoid such traps; moreover, there'll definitely be a limit to how much time a bot will have allocated to spend on each site it crawls, even if it does get "lost" in its navigation.

Not the answer you're looking for? Browse other questions tagged or ask your own question.