|
Hello my server is getting hit by spiders from bingbot, google, yahoo, yandex, and some place in the UK all at the same time crashing it. While I have no reason to think any one person did this, it strikes my odd that it happened all at the same time. Should I be worried? The only changes before this started was that I changed all the server passwords. |
|||||||||||
|
|
Bing, Google and Yahoo usually do not flood the the server with requests, but send far less than one request per second. Please double check that you really get many requests from them within a very short time frame (e. g. within a second). If your web application does have issues with handling occasional requests (for example on request per second), you can slow down Bing and Yahoo with the following entry in robots.txt: This will ask crawlers to wait at least 120 seconds between requests. For Google, you can define the delay in the webmaster tools. |
|||
|
|
|
This whole discussion possibly belongs over in Pro Webmasters, however; as pointed out in one of the comments is applicable here as well. Having several search engines hit your site at once is absolutely normal behavior. If you managed to get links to your site from other reputable sites, you will get indexed. The most brutal are Yahoo, Yandex and Baidu. The first time our website bogged, I went into DDOS mode and got mostly embarrassed by finding every IP was coming from recognized Microsoft, Yahoo and Google sources (verified by IP block ownership, not UA Strings). The answer was to bolster the website resources to handle the load. Despite what you surmize about the number of simultaneous users, your webserver must be able to handle Google, Bing and Yahoo simultaneously indexing your site plus your expected traffic. If your site cannot take the load from being indexed, you will need to either exclude or restrict the search engines. While there is this pathetic robots.txt entry that is almost supported, you will have better results by signing up with Google, Bing, Yandex webmaster tools and use their throttling pages to shape their indexing traffic to best fit your website's schedule. Verify where the traffic is coming from by analyzing your web server access logs. Look up the IPs (http://www.botsvsbrowsers.com/ is one such tool to help identify bot traffic sources). Yandex and Baidu are respectivly Eastern Europe and China. If they're not in your venue, ban them to save your bandwidth. EDIT: After looking through the access logs, do be on the lookout for sudden increases in traffic from a known search engine UA string with odd query strings. User Agent strings are not proof the traffic comes from who they say it comes from as the person sending the traffic can generate them at will and try to hide behind the UA identity. |
|||||||||||
|
|
It's crawling because of some link somewhere. You would need either to throttle bots or fix the websites so they are faster. Not sure about how to throttle the bots, you might want to check with google webmaster page or this: http://stackoverflow.com/questions/1328436/how-to-prevent-googlebot-from-overwhelming-site |
|||
|
|
|
I disagree that this is always normal behavior. I am under a similar DDOS attack form the search engines. We have ample resources and numerous redundant servers handling our site. All with DNS failover. And BOOM we got hit and knocked off the web. First I saw reports of failovers going up and down like crazy. Thought we had a bad AJAX or un-cached query page being hit. As I looked deeper into the logs and saw every bot, pretend bot and wanta be bot from both recognized and unknows, I knew it was an initiated attack. Why? Our robot.txt only allows googlebot, all else are excluded from our server fleet. So the robot.txt was being ignored, and we are being brought down. So if you ignore my robot.txt, my firewall will not ignore you. We firewalled (and still are) over 60 spiders so far. (all hitting at the same time, ignoring both the 360 delay and the disallow in our .txt). So I would not always think this is normal anymore. |
|||||
|
