Thousands of 404s from various search engines

Question

Since early April of 2015, my web site has been getting thousands of requests from many search engines for URLs that don't exist and never have existed.

I found a similar, but less detailed, stackexchange question. That question has Google, Bing, Yahoo, Yandex and "some place in the UK", just as I note in my description below.

So my question is, how does someone get at least 5 search engines to start asking a site for a pile of the same URLs all at once?

As near as I can tell, the maximum number of requests per second is 16, for 2015-06-11T18:51:11-0600, 8 requests each from 207.46.13.99 and 207.46.13.106. Both of these are in a Microsoft netblock, and identified themselves as "bingbot". I understand that the more reputable search engines throttle themselves, but 16 per second is hardly throttling.

Just to head off some likely responses, my site doesn't have, and has never had files that match these URLs. As near as I can tell, nobody has tried to move files by these names on to my site. Neither httpd.conf or .htaccess files have mod_rewrite rules that give out 301 or 302 HTTP responses for these URLs. My site does not have strange PHP files lying around that would re-direct or gateway these URLs from somewhere else.

All the URLs are by domain name (example.com), not IP address. As far as I know, I'm the only one to ever use that domain name. Similarly, the IP address for example.com has remained unchanged for some years. I have access_log files going back to 11/Feb/2009, and this type of URL doesn't show up in them until April 9, 2015.

I also, until 2015-08-22, had my 404.php set to redirect to "shock porn" for any of the URL components listed below. I did not consider that anyone would be thorough enough to try those URLs. I humbly apologize for not believing that people are sincere and thorough. Since mobody but search engies and spiders looks for those URLs (on my site), I've deleted that feature.

I should add that I'm running a WordPerfect honey pot, in order to collect passwords used by WP bruters, and to collect PHP malware. I'm not actually running WordPerfect at all. If you ask for a URL with wp-login.php in it, you get a simulated WP login.

You can see my write up here, but that was an earlier honey pot. I've totally re-written it.

One of my theories was that the porn URL requests are some kind of DDOS attempt by spammers/hackers that got caught in my honey pot, and actually realized it.

Here are the details:

Search engines that ask for non-existent URLs, based on user agent strings:

• Google
• Bing
• Majestic12 (?)
• Yandex
• Baidu
• Yahoo
• Yahoo Japan ("Y!J")
• Sogou web spider
• Yisou

I have Apache httpd set up so that the 404 page is a PHP script that records all values of $_SERVER, $_REQUEST, $_COOKIE and $_FILE, so I'm pretty certain of user agent strings, IP addresses and so forth, and that the server is finding the URLs in question non-existent. I have double-checked a few IP addresses. The addresses I checked matched the user agent strings.

My server fields as many as 6000 requests a day for what appear to be Indian or Indonesian porn. Some example URLs:

/egblzsd/foto-bugil-cewek-indonesia-dunia-panas.html
/egblzsd/WWW-Hiroin-Alia-Bhat-Hot-Sexy-Nude-phato-in.html
/xvlgorj/mallu-anty-faking-images.html

There are many others. If I google for "foto-bugil-cewek-indonesia-dunia-panas.html", I see some semi-scandalous pictures, but a lot of the "filenames", if googled for, lead to 404 pages elsewhere.

Every URL includes a component like this:

/zyjos/
/jhkbla/
/fakfxs/
/egblzsd/
/xvlgorj/
/vkbjnid/
/phxyy/
/lfgkvol/
/vnkdax/
/sxzocl/
/zfkdoh/
/brungh/
/bumlefz/
/tyndst/

All of those strings bring up lots of search results in Google, mostly sketchy seeming Indonesian porn sites, or other hinky sites. Following the links, I get a lot of 404s.

Answer 1

Your website has been compromised.

Any request that includes a URL component listed by you leads to a 301 permanent redirect to a random porn site serving advertisements.

GET /phxyy/whatever HTTP/1.1
Host: stratigery.com
Accept: */*


HTTP/1.1 301 Moved Permanently
Date: Sat, 22 Aug 2015 02:17:04 GMT
Server: Apache/2.4.16 (Unix) PHP/5.6.12
X-Powered-By: PHP/5.6.12
Location: http://www.somepornsite.massorgy/
Content-Length: 0
Content-Type: text/html; charset=UTF-8

It should be noted that the redirect is served out by your server, which can only mean that your server (not the authoritative name server) is compromised.

What you should do:

Take down your website and investigate the following locations for signs of obfuscated code:

/index.php
/wp-config.php (if using WordPRess)
/configuration.php (if using Joomla)
/wp-content/themes/yourtheme/functions.php (if using WordPress)

If you don't have the above, look for files containing long lines of code. Since you are using linux, you may try the following:

$ grep -rl --exclude-dir={image_folder} ".\{1000\}" /var/www

Also, to look for obfuscated code elsewhere, try this:

$ grep -rE --exclude-dir={image_folder} "eval|GLOBALS|error_reporting|chr\(|\\\\x?[0-9]{2}\\\\" /var/www

Although you have mentioned that you have checked your server configuration files, I would highly recommend you to do so again.

After confirming the cause, the next step is to re-install your server with the latest security updates and date from last known good backup to eliminate any possibility of undiscovered backdoors.