Finding all links under a website

Question

How can i find all directories and links under a website? Note that there is no link from home page to all other pages. For example if i have a domain like this users.company.com and each user will have a link like users.company.com/john, users.company.com/peter etc. But i don't know how many users are there what are all the links and i want to brute force to check all links. Is there any tool or website providing this service? If i want to write a python script to do this job, where can i get information to develop this in python?

Answer 1

There is a program called dir-buster developed by the OWASP project which i believe will do exactly what you want, to brute-force file and folder names

To quote the site:

DirBuster is a multi threaded java application designed to brute force directories and files names on web/application servers. Often is the case now of what looks like a web server in a state of default installation is actually not, and has pages and applications hidden within. DirBuster attempts to find these.

And you should be aware that:

Tools of this nature are often as only good as the directory and file list they come with. Just make sure that you've updated this list with any data you can to assist it in finding these folders.

Hope that helps answer your question!

Answer 2

Burp Spider, part of the Burp suite of tool has a useful spidering tool for identifying common files and directories of web applications. It's another useful option along with those that have already been suggested and from my experience is fairly simple to use.

You also might consider checking out this question which seem similar to yours:

Python Web Crawler has a good list going and it seemed like there were a lot of answers. A couple of the suggestions were Harvest Man and spider.py module.

There are also recipes for such tasks posted and there is at least one framework to assist in developing your own web crawler or scraper, called Scrapy.

Hope this helps.

Answer 3

I recommend using skipfish:

The snippet below is from ./skipfish/doc/dictionaries.txt (it is better to read the whole doc):

"The basic dictionary-dependent modes you should be aware of (in order of the associated request cost):

1) Orderly crawl with no DirBuster-like brute-force at all. In this mode, the scanner will not discover non-linked resources such as /admin, /index.php.old, etc:

$ ./skipfish -W- -L [...other options...]

This mode is very fast, but NOT recommended for general use because the lack of dictionary bruteforcing will limited the coverage. Use only where absolutely necessary.

2) Orderly scan with minimal extension brute-force. In this mode, the scanner will not discover resources such as /admin, but will discover cases such as /index.php.old (once index.php itself is spotted during an orderly crawl):

$ touch new_dict.wl
$ ./skipfish -S dictionaries/extensions-only.wl -W new_dict.wl -Y [...other options...]

This method is only slightly more request-intensive than #1, and therefore, is a marginally better alternative in cases where time is of essence. It's still not recommended for most uses. The cost is about 100 requests per fuzzed location.

3) Directory OR extension brute-force only. In this mode, the scanner will only try fuzzing the file name, or the extension, at any given time - but will not try every possible ${filename}.${extension} pair from the dictionary.

$ touch new_dict.wl
$ ./skipfish -S dictionaries/complete.wl -W new_dict.wl -Y [...other options...]

This method has a cost of about 2,000 requests per fuzzed location, and is recommended for rapid assessments, especially when working with slow servers or very large services.

4) Normal dictionary fuzzing. In this mode, every ${filename}.${extension} pair will be attempted. This mode is significantly slower, but offers superior coverage, and should be your starting point."

For additional dictionaries, check out fuzzdb. In particular, the files in:

fuzzdb-1.09/Discovery/FilenameBruteforce
fuzzdb-1.09/Discovery/PredictableRes

Another: Better Wordlists for Forced Browsing

Answer 4

What about site/inurl:users.company.com?