My server was hacked with some files being overwritten to generate spam links on the index page. All my previous backups contain the same modified files, making the nuking from orbit option a puke. If the server root has not been compromised, I intend to just clear the infection and monitor further. What are the signs to look out for to determine if my root has been compromised?
Your computer has been rooted if an unauthorized user (how defined? becomes difficult on systems with many accounts) has gained access to it. If JoeUser is an account with little access to anything of importance (how defined? by whom? how is importance level tracked?) nothing much might be lost, but any unauthorized (ie, hostile) access has the potential for further breaches, given the way ordinary humans behave -- see the movie War Games for an entertaining and not too fantastic account with drama, or the Cuckoo's Egg by Cliff Stoll for a much more realistic, though still dramatic account from reality. Things have not improved much since Stoll's little adventure (and the browser / web / server / live database Internet has made things vastly more dangerous), and official (govt., corp., org., ...) resources are still pretty clueless.
On the other hand, if JoeUser is the local superuser account, any breach is more or less disastrous, as everything on the machine(s) is in principle completely lost. All data, all configuration information, and probably all connectivity information (topology, all passwords (if users have been foolish), ...). An attacker will likely want to continue to have access to the machine(s) and so will try to so configure the system as to perform normally, save that when the attacker wants access for further depredations, it is still available to him.
The techniques used for such things are as varied as people's imagination and low cunning. They are usually called rootkits. Since very few people are really good at this sort of thing, and since many attackers are scriptkiddie level folks, techniques are often reused. There are several rootkit scan programs (some good, some not so good, ...) which can find the traces left by some (perhaps many) of these schemes. But if the rootkit scanner you choose is not aware of whichever rootkitting technique was used on your machine, you won't hear about it. Or, even worse, what about the situation in which the rootkitting attack on your machine isn't traceable by a looking for traces on your disk or machine memory, or is really new and none of the rootkit scanning programs have any provision for finding evidence of it?
In actual reality, you won't generally have the time, nor sufficient expertise on your particular machine (OS), to find rootkit traces that no scanner can find. Thus as a practical matter, the reformat / reinstall from trusted media / reconfigure / acceptance test / pray option is likely the only one with any real credibility.
In your case, you may not have been rootkitted. Depending on the configuration of your server software, root access may not have been required to do the spam distribution you report. IF that is so, you may be able to merely reformat the server's partition (if hat's how you had it setup) / reinstall / .... Inappropriate configuration of the server and its resources (files, access to executables, and other accounts' files (eg, read only for SSL operations, special configuration files, exec PATH setup in the environment, ...) may have been the entry point.
Make sure your server is correctly locked down to prevent such access, not an easy task. Requires considerable time and expertise in following the boards, books, conferences, ... to be sure you're up with the current best practice -- which, of course, may have hidden vulnerabilities no one currently is aware of.
There are no easy answers. is not available and will not be until Internet standards are rather less porous than they are currently.