I got an email from Amazon that my Amazon EC2 server is being used for DDoS attacks and they have closed all ports except SSH. This is tragical for me but I have not been able to find out how and where this attack is running. I can not find anything in cron jobs. I am using ssh key-pairs to connect to the server so it is unlikely someone can have hijacked the root account. I did have FTP on the server but I changed to SFTP now. Could it be a PHP page being used for this and how do I find the source? is it possible someone has used a root kit without the root user access?
It could be that someone is indeed using your instance either by rootkit or through some malicious PHP scripts. Your system is compromised now. Nuke it from orbit and restore from a trusted state (backup).
Make sure to keep your server updated and use long, complex password hashes. Disallow root login (only allow login by key for instance). Install a HIDS (host based intrusion detection system) and a rootkithunter. OSSEC is a good choice on Linux.
Probably your server was compromised by a web vulnerability and a php script was uploaded to it which interacts with the orders of the attacker.
You should follow this steps in order to ensure you are not going to be hacked again:
Most important thing is to identify how they compromised your site, if you restore it with the same vulnerability the attacker's bot will scan your site again and you will have the same problem.
Also, keep your services updated, use long and complex passwords, etc...
I would try to establish the extent of the penetration. It may be limited to some type of script on your site they were able to abuse to create a DDOS attack. It not that, it may be limited to a script they were able to upload that performs some bad action.
If you have a known good configuration and can determine that they didn't escape the bounds of your scripting engines (and you had your scripting engines setup securely so that they couldn't alter the system itself), then you should be ok to restore to a known good configuration of anything that the scripting engines have access to.
If you can't say for sure that your scripting engines are configured securely or if it seems they may have actually gotten direct access to the box outside of the scripting engine, then there aren't measures to contain what they could have hidden and you should nuke it from orbit as suggested earlier.
The trick is you don't want to leave any chance of a re-infection which is highly likely if they've gotten low level access since it makes it easy to leave seemingly innocuous stuff behind that will allow reinfection more easily.
Exploiting a weakness in a local application, perhaps coupled with a root kernel exloit is the likely way. It would go something like this.
1) Hacker exploits a weakness in a PHP application that lets them create files owned by the web server, and then execute that code.
2) Their malicious code would download a script that allows easier access to run commands. Downloading a perl script to /dev/shm using wget is quite common.
3) The perl script would connect out to a remote server to form a tunnel, or may accept incoming connections, so that arbitrary commands can be run more easily.
4) C source code for a root exploit would be installed, compiled and run.
5) If successful the user would now have root privileges. At this point, the firewall may be disabled, the default iptables configuration changed, and backdoors such as a modified ssh could get installed to give access despite hosts.allow settings and the like.
This only scratches the surface, but there are tools that can help with intrusion detection, and steps that you could take to harden your setup such as preventing outbound connections to arbitrary IP's at the firewall (iptables rather than EC2's external firewall), not having a C compiler in the usual place or at all, and moving tools such as wget, though a script could achieve the same so that's merely a nuisance for the hacker. Keeping a record of running processes frequently (every 5 minutes or less) can aid in post mortem analysis as can web logs. A hacker will be very likely to create files or directories in /dev/shm or /tmp, and a utility that is monitoring for unexpected entries there could catch intrusions early. Locking down your server to allow access only to yourself should unexpected items be found, particularly in /dev/shm (where there'd usually be nothing) would not be excessive.
Deploying kernel updates is strongly recommended as there have been plenty of privilege escalation exploits in Linux.