I am currently building a Python-based OSINT tool that allows a user to crawl a supplied domain for pages using traditional scraping/spider methods, but I also want to have the option to 'brute force' common pages for web applications and attempt to continue the crawl based on found results. For instance, I want the user to be able to supply a domain name (example.com) and then once it exhausts the links found on the main site, I want it to systematically try a predefined list of known/common pages that exist (example.com/wp-login.php, example.com/admin.php, etc...) and then use any found pages from that brute force to continue its crawl.

I know that NMap has something similar to this with the http-enum nse script. However, I took a look at the raw .nse file and it doesn't seem to be pulling from a list, rather its constructing the links in another way (not strong enough with nse to know how). Does anyone know if a list exists of common pages that I might be able to feed into my program? I feel like this would fall under "Google Dork" territory? Even something like the "top 100" would be good.

I know that this would be noisy but I plan to give users control over how aggressive they want the scan to be.

Thanks in advance!


I believe the kind of tool you are looking for is called a "web fuzzer." A quick search turned up an open source tool called wfuzz, which contains a folder called wordlist which has lists that seem to be pretty close to what you describe.

  • Ah yes! This is exactly what I was looking for. It looks like they don't include extensions but I can task those on with the scripts. I like that they have a range of ones (for admin panels) vs just common endings. I may build out a "top 100" list from this and then allow the user to brute force (or in this case "fuzz") more possible pages. It might also be interesting to see if I can simply integrate this tool into my own. – Tobin Shields Aug 28 '18 at 18:11

Not the answer you're looking for? Browse other questions tagged or ask your own question.