Consider the following theoretical webserver:
On any request, produce an html response that consists of 2 links, which are each randomly chosen strings followed by ".html".
When the google bot attempts to crawl this site it will be presented with 2 more links. In the interest of mapping the site, it will attempt to navigate to them, each returning 2 more links. This will obviously happen infinitely(surely can come up with a solution that never gives the same page name twice).
My question is, do the bots have a mechanism to detect this? If so, what is it? If not, what kind of detection methods would be possible for this situation?
Not entirely sure if this was the right SE site to post this.