I have an unusual problem I've been trying to diagnose for a while:
It's about a Debian server running a custom compile of apache 2.2 with PHP, Red5, MySQL 5.5 (standard binary), sendmail (distro version), and crashplan.
Every other day I see a high amount of HTTP requests to random files, mostly images - we're talking upwards a thousand concurrent connections.
These requests come from the servers own IP address (!).
It's usually a limited set of files that's requested over and over again. I see no real pattern, but it doesn't look like someone scraping information, it looks like a DoS attempt.
Cron runs a script that temporarily bans IPs with more than 200 connections, so this is usually curbed before it can get really problematic. After 1-3 10 minute bans, the attack usually stops.
This has been going on for months. Since the attacks are caught and curbed, I completely fail to see the point.
It's happening at random times and intervals, but usually around morning times UTC.
No referer or agent is being sent with these requests.
I've checked the webserver and red5 logs for related requests around the same time, in case a script on the server is abused to send queries to itself, but couldn't find anything. There's nothing in the apache errors logs or syslog around that time. Rkhunter didn't find anything out of the ordinary either. The server doesn't source route packets, so spoofing shouldn't be an option either.
I'm at a complete loss as to the method and the reason. Any ideas what to check would be greatly appreciated. :)
UPDATE: Following Isernis advice, I've prepared a mechanism to catch some information on the next occurrence. This is (a slightly generalized version of) the method: http://pastebin.com/6uSUKVbh
ANSWER: This is a social media site allowing mySpace type profiles utilizing FCK Editor. Since that is a bit of a security nightmare, the profiles posted by users undergo extensive checks, one of which probes links/images posted. For one I did not exclude the sites own domain in these checks and two due to a bug related to redirects, every link or image was hit 10 times instead of once. So when a user with a profile containing extensive linkage to the site itself hit the save button, the site would DoS itself. :P In particular this concerned one user who has a bazillion items in her profile and tends to save often.
Thanks to Iserni for the right idea how to diagnose this issue!
ANSWER EDIT: I was wrong about the bug. She actually does have some images 10 times or more inside the profile. More specifically near 1000 links and images to be checked each save. I didn't see that coming. :P