Take the 2-minute tour ×
Information Security Stack Exchange is a question and answer site for Information security professionals. It's 100% free, no registration required.

How can i find all directories and links under a website? Note that there is no link from home page to all other pages. For example if i have a domain like this users.company.com and each user will have a link like users.company.com/john, users.company.com/peter etc. But i don't know how many users are there what are all the links and i want to brute force to check all links. Is there any tool or website providing this service? If i want to write a python script to do this job, where can i get information to develop this in python?

share|improve this question
    
Sounds totally legit, to me, so i'll bite... You're looking to create a dictionary attack on a web server to expose ordinarily non-navigable user folders? –  BrianAdkins Jan 30 '13 at 6:45
add comment

4 Answers

I recommend using skipfish:

The snippet below is from ./skipfish/doc/dictionaries.txt (it is better to read the whole doc):

"The basic dictionary-dependent modes you should be aware of (in order of the associated request cost):

1) Orderly crawl with no DirBuster-like brute-force at all. In this mode, the scanner will not discover non-linked resources such as /admin, /index.php.old, etc:

$ ./skipfish -W- -L [...other options...]

This mode is very fast, but NOT recommended for general use because the lack of dictionary bruteforcing will limited the coverage. Use only where absolutely necessary.

2) Orderly scan with minimal extension brute-force. In this mode, the scanner will not discover resources such as /admin, but will discover cases such as /index.php.old (once index.php itself is spotted during an orderly crawl):

$ touch new_dict.wl
$ ./skipfish -S dictionaries/extensions-only.wl -W new_dict.wl -Y [...other options...]

This method is only slightly more request-intensive than #1, and therefore, is a marginally better alternative in cases where time is of essence. It's still not recommended for most uses. The cost is about 100 requests per fuzzed location.

3) Directory OR extension brute-force only. In this mode, the scanner will only try fuzzing the file name, or the extension, at any given time - but will not try every possible ${filename}.${extension} pair from the dictionary.

$ touch new_dict.wl
$ ./skipfish -S dictionaries/complete.wl -W new_dict.wl -Y [...other options...]

This method has a cost of about 2,000 requests per fuzzed location, and is recommended for rapid assessments, especially when working with slow servers or very large services.

4) Normal dictionary fuzzing. In this mode, every ${filename}.${extension} pair will be attempted. This mode is significantly slower, but offers superior coverage, and should be your starting point."

For additional dictionaries, check out fuzzdb. In particular, the files in:

fuzzdb-1.09/Discovery/FilenameBruteforce
fuzzdb-1.09/Discovery/PredictableRes

Another: Better Wordlists for Forced Browsing

share|improve this answer
add comment

Burp Spider, part of the Burp suite of tool has a useful spidering tool for identifying common files and directories of web applications. It's another useful option along with those that have already been suggested and from my experience is fairly simple to use.

You also might consider checking out this question which seem similar to yours:

Python Web Crawler has a good list going and it seemed like there were a lot of answers. A couple of the suggestions were Harvest Man and spider.py module.

There are also recipes for such tasks posted and there is at least one framework to assist in developing your own web crawler or scraper, called Scrapy.

Hope this helps.

share|improve this answer
add comment

What about site/inurl:users.company.com?

share|improve this answer
    
Google doesn't allow you to access every single link of the website through its search interface(I do believe that they have indexed everything that they can, but they won't show all of it to you due to security reasons). If there are a lot of links under that particular subdomain, it will list out a list of it but it won't be able to list all of it(hence, we need to bruteforce it or use other methods). In addition, if the robots.txt has disabled Google bot, then Google won't index it. –  wcypierre Jan 31 '13 at 15:44
add comment

There is a program called dir-buster developed by the OWASP project which i believe will do exactly what you want, to brute-force file and folder names

To quote the site:

DirBuster is a multi threaded java application designed to brute force directories and files names on web/application servers. Often is the case now of what looks like a web server in a state of default installation is actually not, and has pages and applications hidden within. DirBuster attempts to find these.

And you should be aware that:

Tools of this nature are often as only good as the directory and file list they come with. Just make sure that you've updated this list with any data you can to assist it in finding these folders.

Hope that helps answer your question!

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.