Nmap not showing NSE script results

Question

When executing the command nmap -sn -Pn --script http-robots.txt nmap.org, I get no results back from the specified NSE script "http-robots.txt.nse"!

The script should show all the "disallow" entries in the "robots.txt" file on a given target domain, but nothing shows whatsoever no matter what domain I specify! Here's the output of the command:

Starting Nmap 7.31 ( https://nmap.org ) at 2016-11-06 18:01 EET
Nmap scan report for nmap.org (45.33.49.119)
Host is up.
Other addresses for nmap.org (not scanned): 2600:3c01::f03c:91ff:fe98:ff4e
rDNS record for 45.33.49.119: ack.nmap.org

Nmap done: 1 IP address (1 host up) scanned in 1.52 seconds

Any thoughts on what the problem might be?

Answer 1

Drop the -sn flag to let nmap do a port scan or specify a port.

The robots.txt is specific to a HTTP service running on a given port. Without checking ports, nmap is unable to identify HTTP services where it could apply the http-robots script.

From the docs:

With the -sn option it is possible to run a script scan without a port scan, only host discovery. In this case only host scripts will be eligible to run.

Drop the flag and you're fine:

$ nmap -Pn --script http-robots.txt.nse nmap.org                                                                                                                                                       

Starting Nmap 7.31 ( https://nmap.org ) at 2016-11-06 17:13 CET
Nmap scan report for nmap.org (45.33.49.119)
Host is up (0.21s latency).
Other addresses for nmap.org (not scanned): 2600:3c01::f03c:91ff:fe98:ff4e
rDNS record for 45.33.49.119: ack.nmap.org
Not shown: 991 filtered ports
PORT      STATE  SERVICE
22/tcp    open   ssh
25/tcp    open   smtp
70/tcp    closed gopher
80/tcp    open   http
113/tcp   closed ident
443/tcp   open   https
| http-robots.txt: 4 disallowed entries 
| /favicon/tiles/ /favicon/dim/ /favicon/dim-nogzip/ 
|_/mailman/
554/tcp   open   rtsp
7070/tcp  open   realserver
31337/tcp closed Elite

If you know which ports you're looking for you can skip the enumeration:

$ nmap -p 80,443 --script http-robots.txt.nse stackexchange.com