Take the 2-minute tour ×
Information Security Stack Exchange is a question and answer site for Information security professionals. It's 100% free, no registration required.

Hello my server is getting hit by spiders from bingbot, google, yahoo, yandex, and some place in the UK all at the same time crashing it.

While I have no reason to think any one person did this, it strikes my odd that it happened all at the same time.

Should I be worried?

The only changes before this started was that I changed all the server passwords.

share|improve this question
1  
Are you even sure that Google etc. are crawling you? How have you checked? –  curiousguy Jun 28 '12 at 22:34
    
It sounds like your server needs to be upgraded. –  Ramhound Jun 29 '12 at 11:54
    
How many connexions per sec. are you receiving? –  curiousguy Jun 29 '12 at 16:29
    
On the low 10 on the high 30. I will check out the log when I get back to the office. –  Rick Jun 29 '12 at 19:53
add comment

4 Answers

up vote 6 down vote accepted

Bing, Google and Yahoo usually do not flood the the server with requests, but send far less than one request per second.

Please double check that you really get many requests from them within a very short time frame (e. g. within a second).

If your web application does have issues with handling occasional requests (for example on request per second), you can slow down Bing and Yahoo with the following entry in robots.txt:

Crawl-delay: 120

This will ask crawlers to wait at least 120 seconds between requests. For Google, you can define the delay in the webmaster tools.

share|improve this answer
add comment

This whole discussion possibly belongs over in Pro Webmasters, however; as pointed out in one of the comments is applicable here as well.

Having several search engines hit your site at once is absolutely normal behavior. If you managed to get links to your site from other reputable sites, you will get indexed. The most brutal are Yahoo, Yandex and Baidu. The first time our website bogged, I went into DDOS mode and got mostly embarrassed by finding every IP was coming from recognized Microsoft, Yahoo and Google sources (verified by IP block ownership, not UA Strings). The answer was to bolster the website resources to handle the load.

Despite what you surmize about the number of simultaneous users, your webserver must be able to handle Google, Bing and Yahoo simultaneously indexing your site plus your expected traffic. If your site cannot take the load from being indexed, you will need to either exclude or restrict the search engines. While there is this pathetic robots.txt entry that is almost supported, you will have better results by signing up with Google, Bing, Yandex webmaster tools and use their throttling pages to shape their indexing traffic to best fit your website's schedule.

Verify where the traffic is coming from by analyzing your web server access logs. Look up the IPs (http://www.botsvsbrowsers.com/ is one such tool to help identify bot traffic sources). Yandex and Baidu are respectivly Eastern Europe and China. If they're not in your venue, ban them to save your bandwidth.

EDIT: After looking through the access logs, do be on the lookout for sudden increases in traffic from a known search engine UA string with odd query strings. User Agent strings are not proof the traffic comes from who they say it comes from as the person sending the traffic can generate them at will and try to hide behind the UA identity.

share|improve this answer
    
Yea sorry I'm a stackoverflow kinda guy so I'm still learning the tags here. Thank you for you answer. –  Rick Jun 29 '12 at 13:29
    
"This whole discussion belongs over in Pro Webmasters." except that there is a trivial security issue here: HTTP headers are at the discretion of the HTTP client, so cannot be trusted –  curiousguy Jun 29 '12 at 16:28
    
Please explain further. Does this affect the IP address being recorded in the server access logs. We already know that user agent strings are useless unless verified. –  Fiasco Labs Jun 29 '12 at 18:03
    
@FiascoLabs Please use the @ syntax when you ask a question to someone in particular. "Does this affect the IP address being recorded in the server access logs." No. IP source address is sufficiently safe for these checks. "We already know that user agent strings are useless unless verified." Good. –  curiousguy Jun 30 '12 at 13:30
    
@curiousguy Will do on the @. Glad to hear on the IP being ok, I had to fend off some non-Baidu traffic about a year ago in an isolated script kiddie attack. The UA didn't match to the known Baidu blocks so they went into the router "black hole" ACL for the duration. –  Fiasco Labs Jun 30 '12 at 17:07
add comment

It's crawling because of some link somewhere. You would need either to throttle bots or fix the websites so they are faster. Not sure about how to throttle the bots, you might want to check with google webmaster page or this:

http://stackoverflow.com/questions/1328436/how-to-prevent-googlebot-from-overwhelming-site

share|improve this answer
add comment

I disagree that this is always normal behavior. I am under a similar DDOS attack form the search engines. We have ample resources and numerous redundant servers handling our site. All with DNS failover. And BOOM we got hit and knocked off the web. First I saw reports of failovers going up and down like crazy.

Thought we had a bad AJAX or un-cached query page being hit. As I looked deeper into the logs and saw every bot, pretend bot and wanta be bot from both recognized and unknows, I knew it was an initiated attack. Why? Our robot.txt only allows googlebot, all else are excluded from our server fleet.

So the robot.txt was being ignored, and we are being brought down. So if you ignore my robot.txt, my firewall will not ignore you. We firewalled (and still are) over 60 spiders so far. (all hitting at the same time, ignoring both the 360 delay and the disallow in our .txt). So I would not always think this is normal anymore.

share|improve this answer
1  
Did you verify that the spiders in question really were who they claimed to be? –  CodesInChaos Aug 22 '12 at 7:12
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.