The best way to investigate this is to mirror the port the server is on to a different port and run a packet capture on a different system using a tool like wireshark on a windows box, or tcpdump/snoop if you are doing the capture on a unix/linux box. You can then examine the packet capture to try and figure out what's going on.
Alternatively you could shut the system down, pull the hard drive, and set it up as a secondary drive on a different system from which you can run a virus scan. Make sure it's stand-alone, not on the network, and is something that you can wipe completely without losing anything important is it could get infected from the drive you are scanning.
Neither of these two methods is likely to help you though, the problem is that once a system is rooted it's essentially impossible to restore your system, even if you somehow manage to get it working you'll never be certain the infection is gone. It can also take huge amounts of time and effort to do, with failure as the likely result. The best thing you can do is to copy off your essential data and configuration files, completely wipe the system, and rebuild it from scratch as it's the only way to be sure.