0

Consider the following theoretical webserver:

On any request, produce an html response that consists of 2 links, which are each randomly chosen strings followed by ".html".

When the google bot attempts to crawl this site it will be presented with 2 more links. In the interest of mapping the site, it will attempt to navigate to them, each returning 2 more links. This will obviously happen infinitely(surely can come up with a solution that never gives the same page name twice).

My question is, do the bots have a mechanism to detect this? If so, what is it? If not, what kind of detection methods would be possible for this situation?

Not entirely sure if this was the right SE site to post this.

4
  • google bot, and others that are sensibly written, will have a link depth limit. But honeypots like this can be quite useful for other bots.
    – Rory Alsop
    Commented Feb 19, 2014 at 20:03
  • So this means that if my webpage happens to have heavily nested links naturally, that some may get missed? Do you know what that is approximately? Also, if it's only a link depth limit, could I produce a million links on the page? Then even a link depth of 3 would be huge
    – Cruncher
    Commented Feb 19, 2014 at 20:07
  • Frankly, this question is unanswerable without actually knowing how the bots function, and that is (I'm assuming) a trade secret very closely guarded by Google. Odds are pretty good they've got some wildly complex code to guard against this sort of thing.
    – Steve
    Commented Feb 19, 2014 at 20:18
  • Even if this problem wasn't already solved, the power of google's crawlers is probably such that your server would crash trying to produce domain names long before Google even started getting tired...
    – KnightOfNi
    Commented Feb 19, 2014 at 21:22

2 Answers 2

2

No. The number of pages Google (and any major search bot) is willing to crawl on your domain (or indeed whether they are willing to crawl it at all) is based on how relevant they think your domain is.

There are plenty of sites with an infinite number of pages. This problem was solved decades ago.

2
  • How does the bot determine relevance? Surely they don't spend manual man power on that?
    – Cruncher
    Commented Feb 19, 2014 at 20:44
  • 1
    They determine based on the relevance of inbound links, among other things. It's the core of what they do, and they spend vast amounts of manpower on it.
    – tylerl
    Commented Feb 19, 2014 at 21:30
1

Even without special engineering, certain site architectures can have long recursive paths - certain wikis, for instance. Any decently written bot should be able to cope with such site behaviour, at the very least having a recursion depth limit. I don't think anyone here is going to be able to answer concretely, since none of us have access to google's software directly, but I imagine google's bot is at least clever enough to avoid such traps; moreover, there'll definitely be a limit to how much time a bot will have allocated to spend on each site it crawls, even if it does get "lost" in its navigation.

Not the answer you're looking for? Browse other questions tagged .