An IDS (intrusion detection system) or IPS (intrusion prevention system) is the standard solution. It generally works by watching traffic to your protected resources (such as a web server) and looking for characteristics associated with attacks. The big advantages is that it saves a vast amount of time and provides many more ways of scanning for attack vectors than you could on your own. With most systems of this type, you plan a way to get updates, so that attackers evolve, your system follows shortly after.
For any given type of resource, the attack vectors will vary and are always evolving. As you mention, for web servers, a pretty common vector is faulty inputs - code injection and buffer overflow being the two that come to mind most readily. But this is going to vary for any type of resource you want to protect and any protocol that can an attacker can use to access it.
A sophisticated system may combine:
- benchmarks for how the system currently behaves and observation of changes to those benchmarks
- analysis of protocols and misuse of aspects of those protocols
- repeated attempts to provide incorrect or invalid input
- any behavior pattern that shows a type of scanning
- detection of significantly unusual behavior
There's really no perfect answer to the eternal - "have we been successfully attacked?" question, the real key is knowing a given system, what "normal" is and thus what "abnormal" is, and knowing a given design or communication pattern well enough to know how and when it can be misused