I am currently building a Python-based OSINT tool that allows a user to crawl a supplied domain for pages using traditional scraping/spider methods, but I also want to have the option to 'brute force' common pages for web applications and attempt to continue the crawl based on found results. For instance, I want the user to be able to supply a domain name (example.com) and then once it exhausts the links found on the main site, I want it to systematically try a predefined list of known/common pages that exist (example.com/wp-login.php, example.com/admin.php, etc...) and then use any found pages from that brute force to continue its crawl.
I know that NMap has something similar to this with the http-enum nse script. However, I took a look at the raw .nse file and it doesn't seem to be pulling from a list, rather its constructing the links in another way (not strong enough with nse to know how). Does anyone know if a list exists of common pages that I might be able to feed into my program? I feel like this would fall under "Google Dork" territory? Even something like the "top 100" would be good.
I know that this would be noisy but I plan to give users control over how aggressive they want the scan to be.
Thanks in advance!