The Metasploit Framework is my go-to tool for pentest automation still to this day, however, I do like what I've seen of CORE INSIGHT and Immunity Security SWARM. There are a few tools such as Loki (or the older Yersinia tool), intrace, Chiron, mana-toolkit, mitmf, and Responder.py that must be run outside of the Metasploit framework, but so many things can be done inside msfconsole these days (`load kiwi' comes to mind). If you want to see some amazing resources, check out these two documents, the first which covers performing the full PTES using Metasploit and the second with msfconsole command inputs and outputs.
Much of the early work in network penetration testing is done with either nmap or unicornscan, although zmap and masscan have gained a lot of ground in recent years. In particular, dnmap is a sleek approach.
One of the reasons why nmap/dnmap have little competition is because of their solid inputs and outputs, and of course, as Tate Hansen mentions above, speed. In other ways, nmap is just a simple way of expressing what needs to be done.
Here are a few flags I enjoy with regards to nmap:
Slow scan (but not too slow), evades IDS, and gives the reason why the packets didn't make their destinations. Best performed one port at a time with data-length or string set when the destination protocol or port isn't known by nmap. Check scans.io and SHODAN for your target IP prefixes. Try to identify one or two ports you can leverage that are not web servers, but that have historically been vulnerable to remote code exploitation (N.B., you may need to correlate this data with cvedetails.com). You will notice I have chosen TCP port 623 (IPMI RMCP, an already slow-to-respond service that benefits from these slow scans) in the example, but you could easily change this (although I don't recommend too too obvious ones such as SSH or RDP).
-T1 --max-retries 0 --randomize-hosts --reason -n -Pn -sT -p 623
HTTP or TLS targets will typically run the default ports, so I like to SYN scan them first before scanning other HTTP/TLS targets on non-default ports. The second command will be very slow, but it will likely go under the radar, allowing full pivot to the web layer undetected. If you find that the web-related ports are open but the http-title script fails, then something is amiss -- potential honeypot warning! Look for correlation between an open port and a valid, corresponding HTML title element tag that matches the service name and purpose. If a particular IP always responds to any port, then use nmap's
--exclude <ip> directive or
--excludefile <filename> if you have a list of IPs (similar to nmap's -iL flag).
-T1 --max-retries 0 --randomize-hosts --script http-title --reason -n -Pn -r -p 80,443
-T1 --max-retries 0 --randomize-hosts --script +http-title --open -n -Pn -p 81,457,902,1100,1128,1241,1581,1944,2301,2375,2381,3010,3037,3128,3780,3790,3872,4000-4002,4100,4567,4848,5000,5250,5800-5802,5814,5986,6060,6405,7000-7002,7080,7181,7272,7443,7510,7770,7778,8000-8001,8008,8014,8028,8040,8081,8085,8088-8091,8095,8140,8161,8180,8200,8222,8300,8400,8443,8500,8776,8880,8834,8888,8980,8999-9000,9002,9060,9080,9084,9090,9191,9292,9443,9495,9990,9999,30821,50000
If everything is in order, switch to faster scans and check for IDS/IPS. You'll know something went wrong when all of your scans stop, but you can be a little more efficient about it by using nmap's traceroute flag (or combine it with the firewalk script), intrace, lft, ttl-mon.py, or osstmm-afd to try and map out these systems. Typically, firewalls that include IPS/IDP functionality, network-based IPS/IDP appliances, UTM, NGFW, and similar network-stack gateways will only sample the traffic to be analyzed every 8-11 seconds. Longer delays between port scans (e.g., the 15-second default in nmap T1 scanning, or by setting a specific 12-second delay in T2 scans) and the lack of parallelization are typically what make these slower scans sneak by intrusion detection of all kinds. Repeat traffic, such as that which is directed to a single port, could also cause intrusion detection bells to go off, identifying traffic as an attack or automatically rejecting it. During the sampling period, intrusion detection might also look for sequential ports. There are a variety of factors to try and identify or play with. Thus, the next nmap flags seen below should be modified to test for this series of IDS-identification strategies. If the first test case (T2 scan with 12-second delay, no retries, and no sequential ports) gets by but the second does not, then you will want to figure out why. Nmap's packet-trace and traceroute capabilities are highly suggested, but other tools such as hping3 or nping may come in useful to support this effort.
If you find your traffic blocked (e.g., responses completely stop), you may need to switch the source public-IP (or LAN IP) you're coming from, so I suggest that you run most of these at Starbucks or off your primary, usual location (dnmap will likely also help here). Try not to hit any honeynets (e.g., Fortinet FortiGate, TrapX DeceptionGrid, etc) either, or at least identify them so that you can exclude them in future scans. The best way to avoid these traps is to leverage that scans.io and SHODAN data (and knowledge about the target environment, the kinds of technologies in use -- more useful when performing internal network scans), grab a scoreful of ports (nmap or dnmap at increasing speeds, each with 20 or so unique ports), and avoid the overly-shiny services that are potential honeypots, i.e., 21-23, 25, 53, 110, 135-139, 445, 1337, 1433, 1723, 3306, 3389, 5060-5061, 5800, 5900, 8080, 10000, 31337, and 44443 (N.B., DON'T scan these port numbers yet!).
-T2 --scan-delay 12s --max-retries 0 --randomize-hosts --open -n -Pn -p26,66,79,113,389,407,465
-T2 --scan-delay 9s --max-retries 1 --randomize-hosts --open -n -Pn -p512-514,523-524
-T2 --max-retries 3 --randomize-hosts --open -n -Pn -p548,554,587,593,873,993,995,1026,1050,1080,1099,1158,1344,1352,1521,1604,1720,2202,2302
--max-rate 20 --max-parallelism 2 --max-retries 3 --randomize-hosts --open -n -Pn -p2383,2628,2869,2947,3000,3031,3260,3478,3500,3632,3689,4369,5019,5040,5222,5353,5357,5432,5560,5631-5632
--max-rate 40 --max-parallelism 4 --max-retries 3 --randomize-hosts --open -n -Pn -p5666,5672,5679,5850,5920,5984-5985,6000-6005,6379,6481,6666,7019,7210,7634
--max-rate 80 --max-parallelism 8 --max-retries 3 --randomize-hosts --open -n -Pn -p8000,8009-8010,8834,9160,9999,11211,12000,17185,13722,19150,27017,30718,35871,49152,50030,50060,50070,50075,50090,52822,52869,60010,60030,64623
Written below is the next level of nmap scanning (default speed, no randomizing because of the qscan latency checks), giving you lots of detail and potential for movement. Use mostly port numbers you know are open, although select one port that is potentially not open (especially ones in the past that had ambiguous results). Avoid using dnmap or other inefficiencies that will break the qscan script. TCP (typically HTTP or SSL/TLS) ports could be behind reverse-caching proxies, load balancers, redirected at the network level, or similar. The qscan script should help identify these situations, although you will likely want to supplement them with other tools such halberd (or additional NSE scripts such as http-affiliate-id and http-favicon). In this phase, you can use either IP prefixes, hostnames, or a combination. I suggest using one strong tool here, such as blacksheepwall. Be careful when targeting hostnames because Passive DNS sensors will give away your location and/or intentions much more easily than you think, leading to blocks similar to IPS or WAF (and at this point you will need to be able to distinguish between those three, as well as other defensive countermeasures).
--script qscan --max-retries 7 -v -O --osscan-guess --max-os-tries 1 --reason -Pn --version-intensity 0 -sV -p80
Keep testing all ports in the manner directly above until you've exhausted what you already know about the target network(s) and domains. If you have a target SSL/TLS port, the duplicates script will help you understand when a single host is speaking on multiple IPs or networks (typically multihomed).
--script qscan,duplicates,ssl-cert --max-retries 7 -v -O --osscan-guess --max-os-tries 1 --reason -Pn -sV --version-all -p443
Follow this line of thinking with some single port checks using a bad TCP checksum. Utilize your knowledge of the past and build on it. Before you proceed, make certain you have a good idea about the network paths and internals of every packet you send, and predict the responses from this point on out. If you want to build a little extra knowledge at this stage, you might want to use different tools, such as portbunny, masscan, zmap, or unicornscan (the -w file.pcap flag could be repurposed here to allow for packet capture analysis much like MSF's pcap_log, but I don't necessarily suggest either approach as cool as they initially sound). If you do have a way (preferably a separate, inline system so that the disk I/O, CPU, and memory usage doesn't affect your scans) to perform packet capture, typically you will want a tool such as netsniff-ng to save pcaps (or a different tool if you require pcapng) and then process the results with p0f and/or PRADS. Fingerprinting can get complex, and there are a few tools besides nmap that you will likely want to leverage at this stage. Targeting port-specific versions isn't necessary quite yet, although it can be accomplished here. I suggest going heavy on the OS-level fingerprinting with "just enough" service identification to correlate. There are old techniques (such as a fork of nmap called cron-os) that should be ported to a newer tool with a fp database refresh to modern platforms (which is why I suggested p0f and PRADS earlier -- they tend to keep up with the constant change of technologies). Rapid7 has a big-data project for service fingerprinting, recog. Nmap's service versioning is only average when compared to tools such as amap, and there are many point-solution tools such as fpdns, ntp-fingerprint.pl, info2cpe, et al. Vuln scanners such as Nessus incorporate hmap (an HTTP fingerprinter), but there are many tools in that space as well (httprint, httprecon). It's annoying that OS fingerprinting and service versioning are highly problematic. I suggest you do the best you can with the methods and tools I described.
--script qscan --max-retries 7 --badsum -v -O --osscan-guess --max-os-tries 1 --reason -Pn --version-intensity 0 -sV -p80
If, and when you do, run into any obvious IPS or other blocks, see the section far below which mentions sniffjoke and other advanced evasion techniques (AETs). If you can actually fingerprint the system(s) blocking your traffic then your best bet is to run a sniffjoke client/server pair in another similar environment where you have server access already (N.B., this could be from a previous pen test or could be a lab or mock environment). The above badsum trick should preempt most IPS systems. Other techniques we've been through may have also uncovered these blocking systems. If you find a good AET then you will likely want to run all of your previous port scans over again to see if you have similar results. If you feel that SYN traffic is being throttled (one way to check is to use nmap against a bunch of known-open ports with full TCP connects and the same ports with standard SYN-only), see the below section about SYN-cookie evasion as well as the even-further section on using tcptrace to detect such issues.
Scan a few UDP ports, but check with a few different tools to be sure. Learn how to import Nmap data to MSF in order to leverage hosts -R in modules like empty_udp (but modify the script to remove 1-1024,2049,5060,5061). We aren't checking any low-range ports because of honeypots, intrusion detection, logging, or other inconsistencies at this time. If any UDP ports are open, this knowledge should supplement what we already know about the live TCP ports on this network. If, because of past knowledge about the target types (from OS or version detection) or target environment, you feel that you could gain more knowledge by scanning for IP protocols such as SCTP, ICMP, IGMP, or otherwise -- then, be careful, consider the consequences, and send out a few probes to verify or build on that knowledge.
--max-rate 100 --max-retries 0 -v -O --osscan-guess --max-os-tries 1 --reason -Pn -sUV --version-all -p500,523,623,1604,1645,1812,5353,5632,6481,6502,10080,17185
unicornscan 10.0.0.0/24:500,523,1604,1701,1812,2000,3478,5353,5632,10080-10081 -Iv -mU
for i in ike db2 citrix net-support netop ; do udp-proto-scanner.pl -p $i 10.0.0.0/24 ; done
-T2 --scan-delay 1s --max-retries 0 --randomize-hosts --reason -n -Pn -sO -p132
--max-rate 400 --max-retries 0 --randomize-hosts --reason -n -Pn -sY -p1167,1812,1813,2225,2427,2904,2905,2944,2945,3097,3565,3863,3864,3868,4739,4740,5090,5091,5672,5675,6704,6705,6706,7626,8471,9082,9084,9900,9901,9902,14001,20049,29118,29168,29169,36412,36422
-T2 --scan-delay 1s --max-retries 0 --randomize-hosts --reason -n -Pn -sO -p1
-T2 --scan-delay 1s --max-retries 0 --randomize-hosts --reason -n -PM -sO -p1
-T2 --scan-delay 1s --max-retries 0 --randomize-hosts --reason -n -Pn -sO -p2
-T2 --scan-delay 1s --max-retries 0 --randomize-hosts --reason -n -PM -sO -p1
-T2 --scan-delay 1s --max-retries 0 --randomize-hosts --reason -n -Pn -sO -F --exclude-ports 1,2,6,17,132
-T2 --scan-delay 1s --max-retries 0 --randomize-hosts --reason -n -Pn -sO --exclude-ports 1,2,6,17,132 --data-length 3
To be absolutely pedantic, run the below on every active host that responded to anything from past scans.
-T2 --scan-delay 500ms --max-retries 9 -v -O --osscan-guess -n -PE (or -PP if successful and/or -PM if also successful) -sTUV -pT:(at least one closed port but also one open if possible),U:(as T:) --version-all
After this most-recent run, you must know nearly everything you can about each host, each port -- it should be obvious to you if there is a honeypot or IDS/IPS/IDP/etc. You'll be able to group them into categories even if not absolutely sure, e.g., "probably Cisco", "flavor of Unix", "definitely IP phones or softphones of some kind", or "unknown hosts but these match a certain profile even though they have differing ports open". You feel ready to start vulnerability analysis and exploitation against these hosts and their identified and/or unidentified ports.
You will need to add ports previously found to the following nmap port list so that you get a suitable Nmap XML file for importing into Metasploit or querying with metasploitHelper. Be sure to add TCP/SCTP ports to the "T:" section and UDP ports to the "U:" section.
--min-rate 100 --max-rate 400 --min-parallelism 16 --max-retries 9 --defeat-rst-ratelimit -sUS -n -Pn -pT:0,1,19,42,49,85,105,111,143,264,402,444,446,502,515,631,689,705,783,888,910,912,921,998,1000,1099,1211,1220,1533,1582,1617,1755,1811,1900,2000,2001,2067,2100,2103,2207,2362,2380,2525,2940,2947,2967,3050,3057,3200,3217,3299,3460,3465,3628,3690,3817,4322,4444,4659,4672,4679,5038,5051,5093,5168,5227,5466,5498,5554,5555,6050,6070,6080,6101,6106,6112,6503,6504,6542,6660,6661,6905,6988,7021,7071,7144,7414,7426,7579,7580,7777,7787,8020,8023,8030,8082,8087,8503,8787,8812,8899,9100,9200,9256,9390,9391,9788,9855,10001,10008,10050,10051,10202,10203,10616,10628,11000,11234,12174,12203,12221,12345,12397,12401,13364,13500,16102,18881,19810,20010,20031,20034,20101,20111,20171,20222,22222,23472,26000,26122,27000,27960,28784,30000,31001,32764,34205,34443,38080,38292,40007,41025,41080,41523,41524,44334,44818,46823,46824,50001,50013,55553,57772,62514,65535,U:19,42,49,69,111,161,631,1900,2049,2362,7777 -oX nmap_target1.xml
If you find a good target, you may want to utilize custom Metasploit resources (or "rc") files that consist of notes on modules, their settings, and how to run them. You can find port-number specific rc files here, which can get you on the fast path to a shell. It may require some troubleshooting, but attacking the infrastructure this early on could indicate IPS (network or host) presence or lack thereof. It could also get your IP address blocked, so proceed with caution. You also might want to run some of these port-specific NSE scripts, some of which have important script-args and other tuneables. If you already know for sure that a port is open, or even confident in the target OS/service, then you might as well do something to get more information about their runtime states. Your call, but at this stage in the testing automation you will definitely need to increase the information available to you for decision advantage.
Another possibility is to pivot to the web layer, which typically involves running arachni after a little nmap and nikto action (N.B., you will want to modify your nikto.conf to modify the user agent to one of a normal web browser and perhaps comment out the two mutate lines). You will also want to discover more hostnames and IP prefixes at this time (domaincrawler.com, fierce, knock, dnsmap, and subbrute will help with subdomains and dnsrecon should handle all of the rest). If you have tons of targets then you may want to use webshot to drop images that you can sort through and triage your target selections. An example arachni command-line example is listed below for reference (but also be sure to check out how to optimize arachni). Before you run arachni, you may want to customize your web layer scan with the results from some nmap http checks. If you definitely know there is an IPS or WAF in place, then you may want to switch tactics. arachni supports POST-only operations over SSL/TLS as one example.
-Pn -p 80 --version-all -sV --script "http-waf*",http-devframework,http-enum,http-vhosts -oG - | nikto.pl -h - -Tuning x04689c -D 1 -output nikto.xml
-Pn -p 443 --version-all -sV --script "http-waf*",http-devframework,http-enum,http-vhosts -oG - | nikto.pl -h - -ssl -Tuning x04689c -D 1 -output niktotls.xml
arachni --http-req-limit=50 -v --plugin=discovery --plugin=autothrottle --plugin=uniformity --plugin=timing_attacks --exclude-binaries --auto-redundant=2 --http-queue-size=8192 -g -p --modules=interesting_responses,common_files,form_upload,trainer,sqli*,no_sql*,xpath,path_traversal,file_inclusion,rfi,code*,os_cmd* --report=html:outfile=
Now is the time to engage any anti-IDS mechanisms (e.g., sniffjoke, fragroute, and Evader) or run any last-minute SYN-cookie, IDS, IPS, and WAF detection checks. Finally, go for it! No sense in scanning a single port or small set of ports at this point.
--script banner-plus --min-rate 450 --min-parallelism 20 --max-retries 5 --defeat-rst-ratelimit -n -Pn -p-
unicornscan 10.0.0.0/24:a -D -L 20 -r 450 -Iv -mU
unicornscan 10.0.0.0/24:9,42,49,67,88,135,139,162,213,259,260,407,445,464,514,517-523,546,631,657,826,829,1069,1194,1558,1645-1646,1900,1967,2055,2362,2427,2727,2746,3001,3283,3401,3544,4045,4500,4665,5060,5350,5351,5355,5405,5432,6481,6502,8905,8906,9999,17185,18233,26198,27444,32822-32823,34555,41250,47545,49152,49599,54321 -r 450 -Iv -mU
The above rates (--min-rate in nmap and -r in unicornscan) are measured in packets-per second (pps) and can be modified up to 10000 when on a local network or other ideal conditions. There are patches to change nmap's scan rate dynamically (interactively) here (amazing when combined with tcptrace.org looking for retransmissions and modifying either the min-rate, max-rate, interactive equivalents, min-parallelism, min-hostgroup, or max-retries until maximum bandwidth is acheived). It is difficult to get nmap into best-fit approach for bulk SYN or UDP scanning, even when using pedantic min-hostgroup, min-parallelism, and min-rate parameters. This is why many people turn to unicornscan, zmap, or masscan. If your target networks and services have very reliable and consistent round-trip times in their server responses (or if you have the time and patience to figure out the network characteristics), nmap/dnmap may be your best bet for tooling consistency if nothing else.
You may need to verify new IPs, hostnames, or ports that you haven't seen in the past (such as the honeypot ports we avoided earlier, including SCTP ports 5060-5061 with -sY). Go back to the stage where you ran qscan and retry scenarios using the new information. Continue to check for honeypots by using additional techniques such as Metasploit's detect_kippo module. Be sure to rerun metasploitHelper if applicable. Keep track of all of your information, perhaps by using Dradis, MagicTree, discover.sh XML to CSV conversions, Lunarline VSC, Prolific Solutions proVM Auditor, Cisco Kvasir, or FishNet LAIR. These are better alternatives to Notepad, etc because they are penetration-test data aware. Similar to earlier, you may find that the duplicates script helps identify multihomed hosts. Common scenarios include Windows machine with or without SSL/TLS -- or possibly a non-Windows machine running SSH, SSL/TLS, or both (although it is possible that a Windows host could be running SSH, but that's a severely strange corner case).
--script qscan,duplicates,nbtstat,ssl-cert --max-retries 7 -v -O --osscan-guess --max-os-tries 1 --reason -Pn -sSUV --version-all -pT:135,139,443,445,U:137
--script qscan,duplicates,nbtstat --max-retries 7 -v -O --osscan-guess --max-os-tries 1 --reason -Pn -sSUV --version-all -pT:135,139,445,U:137
--script qscan,duplicates,ssh-hostkey --max-retries 7 -v -O --osscan-guess --max-os-tries 1 --reason -Pn -sV --version-all -p22
--script qscan,duplicates,ssh-hostkey,ssl-cert --max-retries 7 -v -O --osscan-guess --max-os-tries 1 --reason -Pn -sV --version-all -p22,443
If you find any open ports with services (or apps of any kind) that require any form of authentication, then you will want to read and configure the protocol-specific scripts, but generically your nmap/dnmap command-line arguments should look like:
--min-rate 100 --max-retries 5 -n -Pn --script brute,creds-summary --script-args unsafe,brute.mode=pass,userdb=usernames.lst,passdb=passwords.lst,brute.firstOnly,brute.guesses=2 --version-all -sV (-sUV or -sUSV, if appropriate)
The above could be performed with dnmap using a different passwords.lst file on each server in order to increase attempts, to brute from different IP addresses, or a variety of other scenario-dependent options.
Finally, let's talk about how nmap/dnmap can really shine -- by bringing it all together after you know that SYN scanning works, you have UDP probes figured out (note that many of the below are missing in the nmap-payloads, nmap-service-probes, Unicornscan etc/payloads.conf, and udp-proto-scanner.conf files -- so you may have to create them manually!) and that IDS/IPS isn't going to be a bother for your automation. Let's say you aren't just on a coffee break, you can't take "just another nap", and your significant other demands some attention and a normal 8-hour sleep routine. Well, fire up nmap/dnmap this way and you'll get some great results for when you come back to the console!
Just a little effort with nmap can save you a lot of time if you engage with OpenVAS or Metasploit later on. I prefer to run nmap and OpenVAS from within the msfconsole so that all of that data makes it into the MSF database. One could also export tool data to Metasploit-consumable XML, or use a tool like nmap2nessus. An important lesson to learn about vulnerability scanners is that they miss a lot of real-world vulnerabilities. By running nmap/dnmap overnight in the manner described above you will get to see the potential for every single vulnerability path. Most of them won't work at first shot -- nmap/dnmap will tell you that they failed (but not the real "why", even if you think its reason is valid) -- so it is up to you to figure out if the vulnerability can lead to exploitation or not. Go back to the earlier discussion about port-number specific Metasploit and Nmap NSE files, it is even more relevant now. Know the internals of these scripts and how to troubleshoot each line of code that they consist of. A little offset, a small parameter, or a few combinatorial tweaks will grab that shell that every other penetration tester would have missed.