0

I am looking for a tool that generates a set of word lists based on the words it finds on the pages of a particular web domain. This question has an answer that generates permuted word lists starting from existing ones. Other tools such crunch in backtrack/kali has a number of modes of generation but does not take into account an existing word list.

How can I generate passwords based on words on the web domain?

2
  • Do you mean you would like to scrape the web site and generate a list of words from its content?
    – WoJ
    Feb 28 '16 at 9:40
  • Yes, based on some criteria for example word frequency.
    – Sebi
    Feb 28 '16 at 9:45
2

A Python solution

You can use Scrapy which will make most of the work for you. You then would just need to use Counter() to get the top words (if looking for frequency counts).

You could also use a more low-level approach with Beautiful Soup to get the top 5 words:

# coding=utf-8
import requests
import collections
from bs4 import BeautifulSoup

thesite = requests.get("http://www.lemonde.fr").text

soup = BeautifulSoup(thesite, 'html.parser')
thewords = soup.get_text().split()

print(collections.Counter(thewords).most_common(5))

Since the output is

[('de', 223), ('la', 154), (':', 123), ('{', 115), ('à', 84)]

you could look at setting a minimal length of a "word" (3 perhaps?)

UPDATE: the code for a sorted list of most common words with 3 or more letters

# coding=utf-8
import requests
import collections
from bs4 import BeautifulSoup
import operator

thesite = requests.get("http://www.lemonde.fr").text

soup = BeautifulSoup(thesite, 'html.parser')
thewords = soup.get_text().split()

# keep only words over 3 chars

thewords = {w: f for  w, f in collections.Counter(thewords).items() if len(w) > 3}
topwords = sorted(thewords.items(), key=operator.itemgetter(1), reverse=True)

print(topwords)
2

How about using a tool called CeWL - Custom Word List generator?

CeWL is a ruby app which spiders a given url to a specified depth, optionally following external links, and returns a list of words which can then be used for password crackers such as John the Ripper.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.