Since early April of 2015, my web site has been getting thousands of requests from many search engines for URLs that don't exist and never have existed.
I found a similar, but less detailed, stackexchange question. That question has Google, Bing, Yahoo, Yandex and "some place in the UK", just as I note in my description below.
So my question is, how does someone get at least 5 search engines to start asking a site for a pile of the same URLs all at once?
As near as I can tell, the maximum number of requests per second is 16, for 2015-06-11T18:51:11-0600, 8 requests each from 184.108.40.206 and 220.127.116.11. Both of these are in a Microsoft netblock, and identified themselves as "bingbot". I understand that the more reputable search engines throttle themselves, but 16 per second is hardly throttling.
Just to head off some likely responses, my site doesn't have, and has never had files that match these URLs. As near as I can tell, nobody has tried to move files by these names on to my site. Neither
.htaccess files have mod_rewrite rules that give out 301 or 302 HTTP responses for these URLs. My site does not have strange PHP files lying around that would re-direct or gateway these URLs from somewhere else.
All the URLs are by domain name (example.com), not IP address. As far as I know, I'm the only one to ever use that domain name. Similarly, the IP address for example.com has remained unchanged for some years. I have
access_log files going back to 11/Feb/2009, and this type of URL doesn't show up in them until April 9, 2015.
I also, until 2015-08-22, had my 404.php set to redirect to "shock porn" for any of the URL components listed below. I did not consider that anyone would be thorough enough to try those URLs. I humbly apologize for not believing that people are sincere and thorough. Since mobody but search engies and spiders looks for those URLs (on my site), I've deleted that feature.
I should add that I'm running a WordPerfect honey pot, in order to collect passwords used by WP bruters, and to collect PHP malware. I'm not actually running WordPerfect at all. If you ask for a URL with wp-login.php in it, you get a simulated WP login.
You can see my write up here, but that was an earlier honey pot. I've totally re-written it.
One of my theories was that the porn URL requests are some kind of DDOS attempt by spammers/hackers that got caught in my honey pot, and actually realized it.
Here are the details:
Search engines that ask for non-existent URLs, based on user agent strings:
- Majestic12 (?)
- Yahoo Japan ("Y!J")
- Sogou web spider
I have Apache httpd set up so that the 404 page is a PHP script that records all values of
$_FILE, so I'm pretty certain of user agent strings, IP addresses and so forth, and that the server is finding the URLs in question non-existent. I have double-checked a few IP addresses. The addresses I checked matched the user agent strings.
My server fields as many as 6000 requests a day for what appear to be Indian or Indonesian porn. Some example URLs:
/egblzsd/foto-bugil-cewek-indonesia-dunia-panas.html /egblzsd/WWW-Hiroin-Alia-Bhat-Hot-Sexy-Nude-phato-in.html /xvlgorj/mallu-anty-faking-images.html
There are many others. If I google for "foto-bugil-cewek-indonesia-dunia-panas.html", I see some semi-scandalous pictures, but a lot of the "filenames", if googled for, lead to 404 pages elsewhere.
Every URL includes a component like this:
/zyjos/ /jhkbla/ /fakfxs/ /egblzsd/ /xvlgorj/ /vkbjnid/ /phxyy/ /lfgkvol/ /vnkdax/ /sxzocl/ /zfkdoh/ /brungh/ /bumlefz/ /tyndst/
All of those strings bring up lots of search results in Google, mostly sketchy seeming Indonesian porn sites, or other hinky sites. Following the links, I get a lot of 404s.