I recently had an issue where I released software which made use of a new URL and several of my customer sites complained that the software wasn't working within their wifi network. It turned out that their respective Firewall software products of choice (some were using BlueCoat, others ForcePoint, Barracude, etc) were all blocking my new URL because it was categorized as "New" or "Unknown" or "Untrusted" or something.

My question is - What is the authoritative source of these categorizations? They all seem to agree on my new URL's category, so are these products all using the same central authority (Google?) for categorization? Is there some authority or organization I could have reached out to to get my URL categorized correctly before releasing my software?

I don't want all of my customers to have to add firewall exceptions in cases like this.


As a former, long-time bluecoat customer, I can give you some insight.

Two main sources:

  1. Threat intelligence sources - many ingest the same sets of data regarding what's a malicious source and what is not. As a result, you're going to see a lot of overlap between products.

  2. Mass quantities of people - Bluecoat specifically has people whose sole job it is to review new sites and suggest categorizations, or to review suggested changes to those categorizations.

What your customers are doing - blocking "new" and "unclassified" sites is absolutely best practice. Given the rise of fast flux DNS, and domain generation algorithms, the number of legitimate new domains is an order of magnitude less than malicious or spammy ones.

I'd suggest simply setting your customers expectations and understanding the support models for these major vendors to ensure that your new sites can be opened up as soon as possible.

  • 1
    Thanks Chris. Is there anything I can do to expedite the categorization of my new sites? – CFL_Jeff Apr 23 '18 at 21:18

Answer to your 1st question: There are no central authority but they could refer to same sources of information like threat intelligence databases and they all follow systematic process for this categorization.

The process of categorization/classification of websites/web content could vary supplier to supplier but most of them follow something similar to below:

  • Skilled/Trained analysis of web content/sites - Human brain is deployed through rigorous review process.

  • Automated programs (scripts) - Automated scripts like web spiders/crawlers those are deployed to monitor the worldwide web to understand the new web urls and contents.

  • Threat Intelligent Sources - They rely on threat intelligent sources to identify the malicious index of the IPs/Domains/Etc.

  • Categorization databases - They source information from web content databases like https://incompass.netstar-inc.com/

  • Follow legislation guidelines - Certain categorization is applied based on law and legislation guidelines.

  • Manual submission by users - User feedback on wrong categorization will be taken as feedback and review will be conducted.

***These some of the steps and as indicated this would vary depend on the supplier.

Answer to your 2nd question: Now your 2nd question of how to categorize your website, there are several sources available where you will be able to submit your url with the suggested classification. Eg: https://incompass.netstar-inc.com/

And you can submit your URL to your customers web-filter vendor site as well for re-categorization. For example: http://www.barracudacentral.org/report https://fortiguard.com/faq/wfratingsubmit

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.