In the case of detecting that a system is infected with a root-kit, what if any thing can be done to trace route the origins of the remote out going packets to attempt to find out where the attacker was logging in from? I ask this question as more of a curiosity and as a wonder how in the professional world a hacker would be caught. I see a lot of articles stating that the attacker has been caught, but what is the procedure?
As a rule, this is a fruitless endeavor.
It is very uncommon for a hacker to log in from his home IP or from any server directly traceable back to him. It's far more common for hackers to use previously-hacked targets as jump-off points for future attacks. Often attackers will also use other relays (such as IRC bouncers or public IRC networks) to relay commands to infected servers.
You can often trace the attack back to the server from which the attack was launched, but that almost certainly will not belong to the perpetrator. Theoretically you can get the logs from that server and attempt to trace the attack hop-by-hop. But in practice, this never happens. Often the attacks cross political borders making this sort of cooperation effectively impossible. Furthermore, even if you can trace his original IP, it will many times belong to a location not directly traceable to him, such as a coffee shop or an internet cafe.
But if you have the logs, it can be helpful to get the IP address of the last hop primarily because you can then analyze all traffic logs relating to that IP to see how and when the attack happened, and to help you identify additional attack components that you may have missed.
As for how these hackers get caught
Very rarely is the attack actually traced directly back to the perpetrator, as covering your tracks is reasonably simple. Instead, attackers are found by other means. For example: tracking someone bragging about the attack on IRC, analyzing the use of stolen information, identifying the twitter account of someone who took credit for the attack, or plain old snitching by ex-friends or informants.
Generally this is not possible without the help of ISP. IP traceback is a name given to any method for reliably determining the origin of a packet on the Internet. Since the source IP address of a packet is not authenticated or most probably proxied. The problem of finding the source of a packet is called the IP traceback problem. IP Traceback is a critical ability for identifying sources of attacks and instituting protection measures for the Internet. There are number of techniques proposed most popular are
Probabilistic packet marking:
probabilistically marking packets as they traverse routers through the Internet. The router mark the packet with either the router’s IP address or the edges of the path that the packet traversed to reach the router.
Deterministic packet marking:
This technique attempt to put a single mark on inbound packets at the point of network ingress. Their idea is to put, with random probability of .5, the upper or lower half of the IP address of the ingress interface into the fragment id field of the packet, and then set a reserve bit indicating which portion of the address is contained in the fragment field. By using this approach they claim to be able to obtain 0 false positives with .99 probability after only 7 packets.
IP tracing, as others have stated, will not be helpful. What can be helpful is all the other data you can collect, like the code of uploaded files and other actions performed by the attacker.
For instance, I ran a honeypot that captured all downloaded files and keystrokes. From the code in the botnet he installed, the download location of the code, and the passwords he used, I was able to trace the person directly. I had enough information to correlate that I discovered his name, work email and phone number, and his favorite coffee shop in his hometown in Romania (thank you, Facebook).
As smart as he was, he used his full name as a password when he created a new user on my system, used a download location that he set up himself, and he used a pseudonym that he also used on a social networking site that he deleted, but Google had cached.
That's the kind of information law enforcement needs to catch an attacker.