Is there a way to identify which popular sites will host arbitrary user-generated content?

I am doing some analysis to detect phishing sites. It would be nice to use the popularity of a website as a way to weed out sites that, most likely, aren't phishing sites. For instance, Google.com and Amazon.com are very unlikely to be hosting a phishing site. Thus, very popular sites might be considered trusted. However, popularity isn't a perfect proxy for trust, if some popular sites allow hosting arbitrary user content. For instance, Wordpress.com, Github.io, Googleusercontent.com, and Wix.com probably shouldn't be considered as trusted, because they host user-generated content and thus could be used to host a phishing attack.

Is there any way to obtain a list of such sites? I only care about this for the most popular sites (e.g., the Alexa top 1000 or so). Are there any website reputation or trust rating services that provide this information?


At the most basic level: no. Any site can choose to host any content, and have differing levels of sanitation of user generated content, and various levels of security. Even Google has services that could be used for phishing, no matter how challenging it may be to actually do so. Even if a site does not allow it, there are attacks, like DNS poisoning, SQL injection, escalation attacks, and so on, that might leave a normally secure site in a malicious state. Even NASA's own website was once hacked, and could have done a lot of damage if the attackers had intended to do so.

|improve this answer|||||
  • 1
    Thanks for your answer! I am fine with ignoring hacked sites and vulnerabilities in sites, and focusing only on sites that support hosting of user content as an intentional feature. I realize such a list won't be perfect. – D.W. Mar 21 '16 at 7:34
  • Fair enough. I'll have to delete this answer, and you might want to revise this question... At any rate, I wish you luck with this, because I'm pretty sure most free sites advertise themselves instead of each other. I've always managed to find these sites, but never a curated list of such sites. Your best bet might be an entry on Wikipedia. – phyrfox Mar 21 '16 at 7:41

Research to better understand, document, and protect against phishing is of course desirable, but I would echo the previous answer that at the domain level this is probably impossible without doing content analysis, because it amounts to proving a negative, complete information for which does not exist.

The presence or absence of user generated content in the content delivered by a business entails additional provisions in a privacy policy, specifically to cite laws that protect businesses against suits based on content created by users that does not represent the policy of the business.

It is probably possible to analyze a corpus of privacy policies and determine which ones suggest the presence of user-generated content, but that maps only to businesses, not to domains.

Generating the full listing of domains and subdomains in the possession of a business is an art, as that information is not expressly public and has to be guessed at or inferred. In the context of a specific domain, a given policy may apply to many domains in the possession of the business that is ultimately accountable for the policy, and not necessarily apply specifically to the domain under which it is hosted.

So one could say that there is UGC somewhere on these X domains owned by business Y, because the privacy policy said so, but there is no policy to dictate determine UGC is only on domain A and not on B.

Domains themselves represent trust boundaries with browsers, and for those domains that participate, there is some analysis possible to determine what additional resources are considered within the trust circle established by the domain. So for instance, many domains "trust" jquery or other third party javascript hosting sites- which means these sites become targets for hijack to distribute malware or to phish. But this itself does not map to the presence or absence of user generated content.

Many many sites of course traffic in ads, which are a major phishing and malware delivery vehicle. One may even exhaustively crawl a site to determine whether it serves ads, but if it determines a bot is visiting, it may decide not to deliver ad tags to that particular client because doing so will have an impact on metrics important to ad delivery and revenue.

Phishing sites can be hidden behind layers of URL shorteners, and these also can make decisions in real time about what link to deliver to what visitor, so even the presence of links in UGC is not necessarily a straightforward indicator.

Finally, in the context of popularity- google.com ABSOLUTELY can host phishing content, because user-generated content is returned in search results served under the google.com domain.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.