Most efficient way to scan for webservers

Question

Lets say I want to scan a large amount of ip addresses for webservers running on port 80 (aware that there are other ports for webservers, ignoring for example).

What would be the most efficient way to do this?

Here are some options:

1) run a bash/python script that runs multiple instances of nmap simultaneously with only one port being scanned, command would look like this:

nmap --append-output -oG output.txt -p 80 -Pn -sS xxx.xxx.xxx.xxx (ip address) &

2) create a python script that attempts a GET request for each ip address (also running multiple instances simultaneously)

Essentially, my question(s) are: are any of these options the fastest way, or is there a faster way that I'm not thinking of, also, is nmap efficient compared to a python script or is there a lot of overhead in terms of requests, memory footprint etc.

Bonus Thought

An idea that crossed my mind was to run a bash/python script doing NSLLOKUP on each ip address, though this wouldn't give me webservers that don't have DNS configured, the plus side would be scanning without sending any packets into the network

Answer 1

There are tools made for large-scale scanning that have implemented optimizations exactly for solving this problem. One of the best known and most developed options is masscan. It can reach speeds of 10 MPPS (Million Packets per Second) and can scan the entire IPv4 internet in ~6 minutes. It even has flags that let you benchmark on your local network without spamming the whole internet.

Of course, that assumes an internet connection that can sustain that. If you're scanning from home, expect nowhere near that performance unless you have Gigabit internet.

Answer 2

Nmap itself can do everything you asked for, in an efficient way using simple NSE scripts.

• Scan large numbers of servers for port 80: nmap -p80 -PS80 --open
• Do a GET request for each root page: --script http-fetch --script-args 'maxpagecount=1,destination=/tmp/files . This unfortunately doesn't work well for multiple targets as of Nmap 7.60, but it wouldn't take much of a fix to make it do so. Other scripts you may be interested in are http-grep, or http-headers.
• NSLOOKUP on each address: this is unspecific, but Nmap can do:
  • Reverse DNS (PTR record) lookup, done by default unless you use -n. If this is all you want, use -sL for "list scan."
  • Forward lookup of hostnames/domains. Nmap does this by necessity, but if this is all you want, then use -sL -n.