This is exactly what a PCI Compliance test is for. Testing what for them is a black-box (i.e. your server) and telling you what they find and whether a fix is required (i.e. they have a level which defines whether a fix is mandatory to consider your box actually safe.)
The way it is done is fairly simple. You write a software that opens and connects to an HTTP port, you send a GET, POST, and other methods, and analyze the answer. The answer includes the headers and the content and, if you want to go to that level, the TCP/IP socket reaction (i.e. one test they like to do is to check whether TCP numbers are considered sequential, if so it makes it easier for a man in the middle to intercept future connections. i.e. they connect once, get number N, then know that the next number will be N + 1. It's a tad be more complicated, but it is one of the things that can allow you to piggyback on someone's else connection...)
In general, at least from what I've seen so far, it is done against all the CVE that exist. Unfortunately, for at least the few PCI compliance companies I've worked with, they do not seem to be proactive and tell you about things that you should do ahead of receiving a CVE... (although at least they do not limit themselves to just CVEs, they often use other sources too.)
I think "that guy from over there" gives you good examples. However, although Apache2 modules can be a liability, most vulnerabilities are in scripts that are added, such as PHP or Perl. Most of the attacks I can see on my websites check to see whether you are running Wordpress, Drupal, Joomla, phpBB... or one of my favorite: phpMyAdmin. Those are often way easier to penetrate than an Apache2 module.
However, either way the technique is the same.
Send some HTTP/1.0 or HTTP/1.1 GET to determine basic things that exist on that website, and move on from there. For example:
wget -S http://www.example.com/ >example.com 2>&1
And as you can see, we save all the output to
example.com. Then run various tests to see what headers are available, and possibly what's in the body (i.e. most commonly, what you find in the
<meta ...> tags,) you can check how long it took to get the page, whether you got a
Location: ..., etc. A simple example, is to use
sed to extract the Apache version is available.
By default an Apache2 server is likely to return something like this:
Server: Apache/2.4.18 (Ubuntu)
With tools such as
sed (if you know perl, much better...) you can:
sed -e '/^ Server: Apache\//!d' \
-e '/^ Server: Apache\// s/^ Server: Apache\/\([^\s]*\)\s.*/\1/' \
The result will be:
In most cases, if you are to repeat the work over and over again (i.e. automate the tests,) writing it as a script that you can re-run again and again is the best way to do it. (i.e. you run a test, see a problem, get the test to fail over it, ask the developer to fix the problem, re-run the test, if it still fails, just ask the developer again...) Obviously, if you do not trust your ability to write perfect regular expressions, you'd want to verify that your scripts do what they are expected to do, especially over time as other things change.
Actually, you probably want many scripts with a "driver script". i.e. script (1) can test feature A, then script (2) feature B, etc. Save various information and as you reach script (1001), you can check much more advanced features of, says, Wordpress because by now you know that this website runs that system and it is version 1.2.3 so you know that there is possibly a hack you can use to gain access of the computer and script (1001) is that running that hack to see whether it works or not.
And trying to hide the things hackers use to know what you are running is pretty much impossible. Being flagrant, as with the
Server: ... header in my example above, is certainly not a good idea, even if it is possible for a good hacker to determine what system your website is running anyway. But there are many things you just cannot hide. For example, a Drupal site always has a
/node page. Although you could hide that one, there are so many possibilities like that (a good one is
/cron.php under Drupal! You really cannot get rid of that one, although you could rename it...), that in the end, you just cannot hide them all and hackers will still be able to determine that you have a Drupal site and at least its major version and most of its modules. (i.e. so, really, big waste of time trying to hide anything. Better have the latest secure version...) Wordpress has the well know wp-admin and wp-include directories. If you get a 403 for wp-include, chances are that website is running Wordpress. If you get a 404 instead, chances are that this website is not running Wordpress.
Similarly, although I would say it is a bit more complicated, receiving various headers is not unlikely to reveal some part of the Apache2 server settings, probably not all of them, though. As mentioned in the other answer, by default Apache2 and PHP puts signatures in your headers, for example. If you turn on the language support for errors, then you'll get a form of Vary header...
Other modules may not output a specific header, but they will add various functionalities that you can test by accessing/triggering that functionality. If it happens, then it is installed. If it does not happen, it is not installed. As you know, there are quite a few modules, so it is quite a bit of a job to go through all of them. But I'll give a simple example: if you can connect to port 443 and a valid SSL connection ensue (whether verifiable or not), then one of the SSL modules is installed.
As a side note, CMS systems like Wordpress and Drupal also tweak the returned headers. So you have to be careful as both mechanisms may overlap. The priority for most headers is given to the scripts (a few headers are forcibly managed by Apache2 because it is in control of the corresponding features, such as the Keep-Alive feature, since Apache2 holds the client socket.)