I am in the process of looking for a specific keyword on a particularly large website with lots of pages. I'm thinking I'd have better luck if I have a way to see the links within the website. If I want to see all links under a website, for example: company.com/1, company.com/2, company.com/3, etc. will DirBuster be able to pull this information? The website is static if that matters.


At its core, dirbuster takes in a list of common URLs, and tries them.

Also, while it's trying URLs from its wordlist, it has functionality to parse returned HTML content and add those URLs to the list.

DirBuster HTML parsing options

So to your question

I want to see all links under a website. Will DirBuster be able to pull this information?

If the website has HTML links to all its pages, then DirBuster might be able to figure that out, but otherwise dirbuster is only as good as the wordlist that you feed into it.

| improve this answer | |
  • I'm trying not to use a crawler for this purpose, because it would be too timely. Is there a way I could input what I'm looking for and perhaps DirBuster could find the link where the keyword is? The main idea is I'm trying to figure out how to find a keyword among a large number of non-indexed webpages within a website. – thenewcoder May 6 at 13:51
  • You should go play with DirBuster and see if you can get it to do what you want by changing the wordlist file that it is using for the scan. – Mike Ounsworth May 6 at 14:22
  • 1
    Sure thing, if it's okay, I will come back and let you all know how it goes. – thenewcoder May 7 at 14:50
  • @thenewcoder Another option could be a tool like OWASP ZAP; use its Spider crawler to index your site, and then ctrl+f through its history for your keyword. That would be a bit slower but not terrible, maybe hours? – Mike Ounsworth May 7 at 16:32
  • There are millions upon millions of pages. It would take a very long time. – thenewcoder May 10 at 22:23

Considering that the website is implemented as a normal informative "brochure" (you mentioned it is static) all the content should be linked, so what you need to do is to crawl all its content then search. There's numerous tools that can do this for you including wget or curl. A quick search for crawler or http mirror tools will give you alternatives.

dirbuster (and more modern derivatives) would be useful if you are looking for hidden or unlinked content. Files and directories with predictable names that can be guessed which are not linked anywhere on the website from its index page. It is used to find admin interfaces, configuration files, etc. It's not a crawler.

| improve this answer | |
  • To the best of your knowledge, which tool is the fastest? – thenewcoder Jun 1 at 18:41
  • Not sure, never benchmarked them. Although speed wouldn't be a major concern to me in this situation. Completeness and accuracy would. – Pedro Jun 1 at 19:31

I think you need httrack + grep, it's easy and fast.

You can download the entire accessible website following links dynamically with httrack (install with apt install webhttrack on a Debian system, else please refer to httrack download page). You'll have a copy of the website locally. So you can easily search on those local files with grep -rni 'your search here' from the root directory of the downloaded website.

You can hardly beat grep for quick searches (even though you have forks of grep in order to do so).

If httrack is not enough, you can mix a dirbuster / dirb / gobuster / patator / whatever with a wordlist to try to catch hidden endpoints, then re-run an httrack on them to complete the previously downloaded website and re-grep what you're looking for.

Have fun.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.