The Metasploit Framework is my go-to tool for pentest automation still to this day, however, I do like what I've seen of CORE INSIGHT and Immunity Security SWARM. There are a few tools such as Loki (or the older Yersinia tool), intrace, Chiron, mana-toolkit, mitmf, and Responder.py that must be run outside of the Metasploit framework, but so many things can be done inside msfconsole these days (`use kiwi' comes to mind). If you want to see some amazing resources, check out these two documents, the first which covers performing the full PTES using Metasploit and the second with msfconsole command inputs and outputs.
Much of the early work in network penetration testing is done with either nmap or unicornscan, although zmap and masscan have gained a lot of ground in recent years. In particular, dnmap is a sleek approach.
One of the reasons why nmap/dnmap have little competition is because of their solid inputs and outputs, and of course, as Tate Hansen mentions above, speed. In other ways, nmap is just a simple way of expressing what needs to be done.
Here are a few flags I enjoy with regards to nmap:
Slow scan (but not too slow), evades IDS, and gives the reason why the packets didn't make their destinations. Best performed one port at a time with data-length or string set when the destination protocol or port isn't known by nmap. Check scans.io and SHODAN for your target IP prefixes. Try to identify one or two ports you can leverage that are not web servers, but that have historically been vulnerable to remote code exploitation (N.B., you may need to correlate this data with cvedetails.com). You will notice I have chosen TCP port 623 (IPMI RMCP, an already slow-to-respond service that benefits from these slow scans) in the example, but you could easily change this to VNC on a non-standard port (although I don't recommend too too obvious ones such as SSH or RDP).
-T1 --max-retries 0 --randomize-hosts --reason -n -Pn -sT -p 623
HTTP or TLS targets will typically run the default ports, so I like to SYN scan them first before scanning other HTTP/TLS targets on non-default ports. The second command will be very slow, but it will likely go under the radar, allowing full pivot to the web layer undetected.
-T1 --max-retries 0 --randomize-hosts --script http-title --reason -n -Pn -r -p 80,443
-T1 --max-retries 0 --randomize-hosts --script +http-title --open -n -Pn -p 81,457,902,1100,1241,1944,2301,2375,2381,3010,3128,3790,4000-4002,4100,4567,4848,5000,5432,5801-5802,5986,6346-6347,7000-7002,8001,8008,8010,8040,8081,8085,8088-8091,8140,8200,8443,8500,8776,8880,8834,8888,8980,8999-9000,9060,9080,9084,9191,9292,9443,9990,30821
If everything is in order, switch to faster scans and check for IDS/IPS. You'll know something went wrong when all of your scans stop, but you can be a little more efficient about it by using nmap's traceroute flag (or combine it with the firewalk script), intrace, lft, or osstmm-afd to try and map out these systems. If you get blocked, you may need to switch the source public-IP (or LAN IP) you're coming from, so I suggest that you run most of these at Starbucks or off your primary, usual location (dnmap will likely also help here). Try not to hit any honeynets (e.g., Fortinet FortiGate, TrapX DeceptionGrid, etc) either, or at least identify them so that you can exclude them in future scans. The best way to avoid these traps is to leverage that scans.io and SHODAN data (and knowledge about the target environment, the kinds of technologies in use -- more useful when performing internal network scans), grab a scoreful of ports (3 nmap runs through 1 dnmap, each with 20-40 unique ports), and avoid the overly-shiny services that are potential honeypots, i.e., 21-23, 25, 53, 110, 135-139, 445, 1337, 1433, 1723, 3306, 3389, 5060-5061, 5800, 5900, 8080, 10000, 31337, and 44443 (N.B., DON'T scan these port numbers yet!).
-T2 --scan-delay 4 --max-retries 0 --randomize-hosts --open -n -Pn -p26,66,79,113,389,407,465,512-514,523-524,548,554,587,593,873,993,995,1026,1050,1080,1099,1158,1344,1352,1521,1604,1720
-T2 --scan-delay 1 --max-retries 0 --randomize-hosts --open -n -Pn -p2202,2302,2383,2628,2869,2947,3000,3031,3260,3478,3500,3632,3689,4369,5019,5040,5222,5353,5357,5432,5560,5631-5632,5666,5672,5679,5850
-T2 --max-retries 0 --randomize-hosts --open -n -Pn -p5920,5984-5985,6000-6005,6379,6481,6666,7019,7210,7634,8000,8009,8400,8834,9160,9999,11211,12000,17185,13722,19150,27017,30718,35871,49152,50000,50030,50060,50070,50075,50090,52822,52869,60010,60030,64623
Written below is the next level of nmap scanning (default speed, no randomizing because of the qscan latency checks), giving you lots of detail and potential for movement. Use mostly port numbers you know are open, although select one port that is potentially not open (especially ones in the past that had ambiguous results). Avoid using dnmap or other inefficiencies that will break the qscan script. TCP (typically HTTP or SSL/TLS) ports could be behind reverse-caching proxies, load balancers, redirected at the network level, or similar. The qscan script should help identify these situations, although you will likely want to supplement them with other tools such as webshot and halberd (or additional NSE scripts such as http-affiliate-id and http-favicon). In this phase, you can use either IP prefixes, hostnames, or a combination. I suggest using one strong tool here, such as blacksheepwall. Be careful when targeting hostnames because Passive DNS sensors will give away your location and/or intentions much more easily than you think, leading to blocks similar to IPS or WAF (and at this point you will need to be able to distinguish between those three, as well as other defensive countermeasures).
--script qscan --max-retries 7 -v -O --osscan-guess --max-os-tries 1 --reason -Pn --version-intensity 0 -sTV -p80
Keep testing all ports in the manner directly above until you've exhausted what you already know about the target network(s) and domains. If you have a target SSL/TLS port, the duplicates script will help you understand when a single host is speaking on multiple IPs or networks (typically multihomed).
--script qscan,duplicates,ssl-cert --max-retries 7 -v -O --osscan-guess --max-os-tries 1 --reason -Pn -sTV --version-all -p443
Follow this line of thinking with some single port checks using a bad IP/TCP checksum. Utilize your knowledge of the past and build on it. Before you proceed, make certain you have a good idea about the network paths and internals of every packet you send, and predict the responses from this point on out.
--script qscan --max-retries 7 --badsum -v -O --osscan-guess --max-os-tries 1 --reason -Pn --version-intensity 0 -sTV -p80
Scan a few UDP ports, but check with a few different tools to be sure. We aren't checking any low-range ports because of honeypots, intrusion detection, logging, or other inconsistencies at this time. If any UDP ports are open, this knowledge should supplement what we already know about the live TCP ports on this network. If, because of past knowledge about the target types (from OS or version detection) or target environment, you feel that you could gain more knowledge by scanning for IP protocols such as SCTP, ICMP, IGMP, or otherwise -- then, be careful, consider the consequences, and send out a few probes to verify or build on that knowledge. The last line of nmap flags below that scans all non-essential IP protocols could take days or weeks to complete, so I suggest identifying only a few ports instead of excluding only the obvious ones.
--max-rate 100 --max-retries 0 -v -O --osscan-guess --max-os-tries 1 --reason -Pn -sUV --version-all -p500,523,623,1604,1645,1812,5353,5632,6481,6502,10080,17185
unicornscan 10.0.0.0/24:500,523,1604,1701,1812,2000,3478,5353,5632,10080-10081 -Iv -mU
for i in ike db2 citrix net-support netop ; do udp-proto-scanner.pl -p $i 10.0.0.0/24 ; done
-T1 --max-retries 0 --randomize-hosts --reason -n -Pn -sO -p132
-T1 --max-retries 0 --randomize-hosts --reason -n -Pn -sY -p1167,1812,1813,2225,2427,2904,2905,2944,2945,3097,3565,3863,3864,3868,4739,4740,5090,5091,5672,5675,6704,6705,6706,7626,8471,9082,9084,9900,9901,9902,14001,20049,29118,29168,29169,36412,36422
-T1 --max-retries 0 --randomize-hosts --reason -n -Pn -sO -p1
-T1 --max-retries 0 --randomize-hosts --reason -n -sP -PP
-T1 --max-retries 0 --randomize-hosts --reason -n -Pn -sO -p2
-T1 --max-retries 0 --randomize-hosts --reason -n -sP -PM
-T1 --max-retries 0 --randomize-hosts --reason -n -Pn -sO --exclude-ports 1,2,6,17,132 --data-length 3
After this most-recent run, you must know nearly everything you can about each host, each port -- it should be obvious to you if there is a honeypot or IDS/IPS/IDP/etc. You'll be able to group them into categories even if not absolutely sure, e.g., "probably Cisco", "flavor of Unix", "definitely IP phones or softphones of some kind", or "unknown hosts but these match a certain profile even though they have differing ports open". You feel ready to start vulnerability analysis and exploitation against these hosts and their identified and/or unidentified ports.
You will need to add ports previously found to the following nmap port list so that you get a suitable Nmap XML file for importing into Metasploit or querying with metasploitHelper. Be sure to add TCP/SCTP ports to the "T:" section and UDP ports to the "U:" section.
If you find a good target, you may want to utilize custom Metasploit resources (or "rc") files that consist of notes on modules, their settings, and how to run them. You can find port-number specific rc files here, which can get you on the fast path to a shell. It may require some troubleshooting, but attacking the infrastructure this early on could indicate network-based IPS presence or lack thereof. It could also get your IP address blocked, so proceed with caution. If you don't feel brave enough to exploit anything yet, then you might want to run some of these port-specific NSE scripts, some of which have important script-args and other tuneables. If you already know for sure that a port is open, or even confident in the target OS/service, then you might as well do something to get more information about their runtime states. Your call, but at this stage in the testing automation you will definitely need to increase the information available to you for decision advantage.
Another possibility is to pivot to the web layer, which typically involves running Zapr through arachni's proxy after a little nmap and nikto action (N.B., you will want to modify your nikto.conf to modify the user agent to one of a normal web browser and perhaps comment out the two mutate lines). You will also want to discover more hostnames and IP prefixes at this time (domaincrawler.com, fierce, knock, dnsmap, and subbrute will help with subdomains and dnsrecon should handle all of the rest). If you have tons of targets then you may want to use webshot to drop images that you can sort through and triage your target selections. Zapr is very easy to run, but I'll leave an example arachni command-line example below for reference (but also be sure to check out how to optimize arachni). Before you run Zapr through arachni, you may want to customize your web layer scan with the results from some nmap http checks. If you definitely know there is an IPS or WAF in place, then you may want to switch tactics. First crawl SSL/TLS targets using ZAP with no attacks and then test for method interchange. Convert GET operations to POST and run wapiti in POST-only mode (wapiti has some checks for XXE that go beyond arachni, although arachni is much better at identifying file inclusions, particularly important when targeting PHP web apps).
-Pn -p 80 --version-all -sV --script "http-waf*",http-devframework,http-enum,http-vhosts -oG - | nikto.pl -h - -Tuning x04689c -D 1 -output nikto.xml
-Pn -p 443 --version-all -sV --script "http-waf*",http-devframework,http-enum,http-vhosts -oG - | nikto.pl -h - -ssl -Tuning x04689c -D 1 -output niktotls.xml
arachni --http-req-limit=50 -v --plugin=proxy --plugin=discovery --plugin=autothrottle --plugin=uniformity --plugin=timing_attacks --exclude-binaries --auto-redundant=2 --http-queue-size=8192 -g -p --modules=interesting_responses,common_files,form_upload,trainer,sqli*,no_sql*,xpath,path_traversal,file_inclusion,rfi,code*,os_cmd* --report=html:outfile=
Now is the time to engage any anti-IDS mechanisms (e.g., sniffjoke, fragroute, and Evader) or run any last-minute SYN-cookie, IDS, IPS, and WAF detection checks. Finally, go for it! No sense in scanning a single port or small set of ports at this point.
--script banner-plus --min-rate 450 --min-parallelism 20 --max-retries 5 --defeat-rst-ratelimit -n -Pn -p-
unicornscan 10.0.0.0/24:a -D -L 20 -r 450 -Iv -mU -w udp.pcap
The above rates (--min-rate in nmap and -r in unicornscan) are measured in packets-per second (pps) and can be modified up to 10000 when on a local network or other ideal conditions. There are patches to change nmap's scan rate dynamically (interactively) here (amazing when combined with tcptrace.org looking for retransmissions and modifying either the min-rate, max-rate, interactive equivalents, min-parallelism, min-hostgroup, or max-retries until maximum bandwidth is acheived). Some other suggestions involve using nping on the type of traffic you are targeting and setting a -T4 scan with twice the --initial-rtt-timeout and four times the –max-rtt-timeout based on some sort of average round-trip time result you see in nping (I don't recommend this approach but it is listed here for completeness, as well as insight into the rtt timers in the various nmap performance levels, seen with `nmap -d'). It is difficult to get nmap into best-fit approach for bulk SYN or UDP scanning, even when using pedantic min-hostgroup, min-parallelism, and min-rate parameters. This is why many people turn to unicornscan, zmap, or masscan. If your target networks and services have very reliable and consistent round-trip times in their server responses (or if you have the time and patience to figure out the network characteristics), nmap/dnmap may be your best bet for tooling consistency if nothing else.
You may need to verify new IPs, hostnames, or ports that you haven't seen in the past (such as the honeypot ports we avoided earlier, including SCTP ports 5060-5061 with -sY). Go back to the stage where you ran qscan and retry scenarios using the new information. Be sure to rerun metasploitHelper if applicable. Keep track of all of your information, perhaps by using Dradis, MagicTree, Lunarline VSC, Prolific Solutions proVM Auditor, Cisco Kvasir, or FishNet LAIR. These are better alternatives to Excel, Notepad, and vi because they are penetration-test data aware. Similar to earlier, you may find that the duplicates script helps identify multihomed hosts. Common scenarios include Windows machine with or without SSL/TLS -- or possibly a non-Windows machine running SSH, SSL/TLS, or both (although it is possible that a Windows host could be running SSH, but that's a severely strange corner case).
--script qscan,duplicates,nbtstat,ssl-cert --max-retries 7 -v -O --osscan-guess --max-os-tries 1 --reason -Pn -sTUV --version-all -pT:135,139,443,445,U:137
--script qscan,duplicates,nbtstat --max-retries 7 -v -O --osscan-guess --max-os-tries 1 --reason -Pn -sTUV --version-all -pT:135,139,445,U:137
--script qscan,duplicates,ssh-hostkey --max-retries 7 -v -O --osscan-guess --max-os-tries 1 --reason -Pn -sTV --version-all -p22
--script qscan,duplicates,ssh-hostkey,ssl-cert --max-retries 7 -v -O --osscan-guess --max-os-tries 1 --reason -Pn -sTV --version-all -p22,443
If you find any open ports with services (or apps of any kind) that require any form of authentication (HTTP, SSL/TLS, SMTP, POP3, IMAP, Oracle, DB2, MySQL, MS SQL, Cassandra, MongoDB, SVN, CVS, Git, SIP, VMware, et al), then you will want to read and configure the protocol-specific scripts, but generically your nmap/dnmap command-line arguments should look like:
--min-rate 100 --max-retries 5 -n -Pn --script brute,creds-summary --script-args brute.mode=pass,userdb=usernames.lst,passdb=passwords.lst,brute.firstOnly,brute.guesses=2 --version-all -sV (-sUV or -sUSV, if appropriate)
The above could be performed with dnmap using a different passwords.lst file on each server in order to increase attempts, to brute from different IP addresses, or a variety of other scenario-dependent options. If you had three servers for dnmap, as an example, you could set brute.delay to 3 seconds, 5 seconds, and 10 seconds -- noting the differences.
Finally, let's talk about how nmap/dnmap can really shine -- by bringing it all together after you know that SYN scanning works, you have UDP probes figured out (note that many of the below are missing in the nmap-payloads, nmap-service-probes, Unicornscan etc/payloads.conf, and udp-proto-scanner.conf files -- so you may have to create them manually!) and that IDS/IPS isn't going to be a bother for your automation. Let's say you aren't just on a coffee break, you can't take "just another nap", and your significant other demands some attention and a normal 8-hour sleep routine. Well, fire up nmap/dnmap this way and you'll get some great results for when you come back to the console!
Just a little effort with nmap can save you a lot of time if you engage with OpenVAS or Metasploit later on. I prefer to run nmap and OpenVAS from within the msfconsole so that all of that data makes it into the MSF database. An important lesson to learn about vulnerability scanners is that they miss a lot of real-world vulnerabilities. By running nmap/dnmap overnight in the manner described above you will get to see the potential for every single vulnerability path. Most of them won't work at first shot -- nmap/dnmap will tell you that they failed (but not the real "why", even if you think its reason is valid) -- so it is up to you to figure out if the vulnerability can lead to exploitation or not. Go back to the earlier discussion about port-number specific Metasploit and Nmap NSE files, it is even more relevant now. Know the internals of these scripts and how to troubleshoot each line of code that they consist of. A little offset, a small parameter, or a few combinatorial tweaks will grab that shell that every other penetration tester would have missed.