I am a researcher and want to test our phishing website identification tools, but we don't want to rely on any already-identified phishing sites. Instead, we want to find fresh potential phishing sites in real-time. My intention is similar to Crawling for Phishing Websites, but we do not work for any particular brand and don't have access to any webpage.

My question is what URLs/what type of URLs should we scan for? I don't think scanning against any list, such as Alexa top million, will give us any good result. Any guidance is appreciated!

1 Answer 1


Your best bet is Certificate Transparency logs. These can give you a stream of all HTTPS certificates in almost real-time as they are issued which you can then run your analysis on the domains they're valid for.

Here's an example of a project which does a very naive analysis on keywords in Certificate Transparency log domains: https://github.com/x0rz/phishing_catcher

Note: you'll be able to find a huge amount of phishing domains this way, but it does miss out one important category. Because this only gives you a stream of domain names, you won't easily be able to find phishing pages hosted on paths within those domains (e.g. someone's compromised WordPress site which is being abused to host phishing)

  • Thank you! Then how possible to find any website that is on a path of a good domain? I am curious how links such as khyberproton.com.pk/wp-admin/user/verification/Zim021 can ever be found ( just copied this link from phishtank.org)?
    – dovod74538
    Commented Jun 20, 2022 at 19:34
  • Phishing sites have a fundamental weakness: at some point a victim has to see them! Sites like your example can't be found proactively but can be detected reactively e.g. by an email filter that sees the link in a phishing email. Now that the link is known, it can be run through the same analysis as any other suspected malicious site. The difficultly of course is in obtaining this (fairly sensitive) feed of URLs. Commented Jun 21, 2022 at 10:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .