One of my servers somehow managed to catch it at one datacentre, but 2 of my other servers running the same software, each hosted with a different provider, did not get the infection - a nasty one at that, causing the CPU (quad core) to max out.
After searching for an answer I came across this board and although I like the NUKE IT OUT OF ORBIT approach, I may have found a solution after trying many approaches for hours.
Here's the synopses:
I figured if I wanted to stop the existing processes from firing up, I need to obviously delete the executables - not exactly. Since we're dealing with an infection, the processes kept coming back after a few milliseconds. After I deleted the executables, they came back - I managed to catch an SSH connection going out to the C&C server, will post it here later once I log it on my experimental machine.
Next step I did to prevent it from launching, was to find all executables and instead of deleting them, I set their permission to nobody:nobody and 444 (I wanted to still review the contents you can set it to lower). What this did is occupy the name of the file used by the rogue program so that the SSH connection could not replace the contents of the file, and also prevented it from executing rendering it useless. This seems to work for stopping the processes from automatically starting and allowing Remote Code Execution and effectively quarantining the executables .
Please note this is only a patch solution and not a remedy to fix the problem. I will be reviewing the code commit by commit (in git) and eyeballing log files for a while until I figure out how the payload got in there in the first place.
#: sudo yum install htop
Sort by CPU% and note the name of the process consuming 99% or 100%
find the executable:
#: sudo find / -type f -iname "[FILENAME]"
replace [FILENAME] with the name you grabbed from htop
#: sudo chown nobody:nobody /path/to/[FILENAME]
#: sudo chmod 0444 /path/to/[FILENAME]
Shut down the currently running executables
#: sudo htop
find the process, select and hit the "K" key to kill the process. At this point, it should not come back. If a new process popped back up, repeat the steps above.
Maybe someone will have time to write a sh script to do it automatically.
ALso watch out for CRON jobs for user apache - runs every 15 mins to download the payload.