What ways are there to populate web directory wordlists?

Question

Background: as a pentester I often just get IP addresses where websites are hosted. Since sometimes DNS is not available or the client wants a black box test, getting to the actual hosted application is difficult. One technique commonly used is Web directory brute forcing. Tools like dirb, dirbuster and so on use wordlists. But these lists rarely get updated and often fail me in corporate environments (think share point, owa, Lync/sfb...).

What efficient ways and techniques are there to create these lists beforehand? What ways are there to get e.g. a tomcat/iis/... server to spill it's secrets without wordlists when tools like DNS are not available?

Answer 1

You can use word lists in the Kali Linux /usr/share/wordlists. If you want to create specific word lists, you can use the Crunch.

Answer 2

Use different lists, There are tons of web-path wordlists for content discovery here around. Some code to quickly download and merge lists from a different location: https://github.com/ax/merge_webpath_list

The most useful web-path lists are already included in the directory sources.

Answer 3

It will probably depend on the tools you use. I personally have a custom path in which I put my wordlist.

Sometimes I use Burp's TurboIntruder to perform directory/file recon:

for line in open('/home/soufiane/wordlist/random_wordlist.txt'):
engine.queue(target.req, line.rstrip())

Sometimes I use dirsearch:

dirsearch -t 50 -utarget -w /home/soufiane/wordlist/random_wordlist.txt -H "authorization: Basic blah=" -e PHP,ASP,ASPX,JSP -x 301

And sometimes I use wfuzz:

wfuzz --hc 401 -c -z file,home/soufiane/wordlist/random_wordlist.txt -b SESSION=BLAH-b XSRF-TOKENBLAH -u https://target/**FUZZ**

The most important thing regarding your question is the wordlist itself. I'll in your case use Assetnote Wordlists! SecLists is awsome, but Assetnote Wordlists is much more adequate to your need:

aws s3 sync s3://assetnote-wordlists/data/ ./assetnote-wordlists --no-sign-request

Answer 4

I suggest you look into https://wordlists.assetnote.io/.

Some wordlists are generated using Commonspeak. Commonspeak is using BigQuery and certain Datasets (GitHub, HTTPArchive) to generate updodate wordlists.

More info on how Commonspeak works: https://pentester.io/commonspeak-bigquery-wordlists/