1

Real case: I am writing dictionary program and part of the data are URLs. When I set them myself, life is simple, because it is me, so I know what I do. But when I import dictionary data from somebody else those URLs put in the data are risk factor (URLs are part of the data, and in this case serve as references in sense "cat in English = kot in Polish, to learn more see this website").

So I am looking for a policy which balances security on one hand, and user friendliness on the other hand (consider importing 100,100 URLs and going one after another manually and tick whether they are trusted/verified or not).

Currently I came to such approach:

  • at start, at my computer, I can define trusted domains, like wikipedia, wiktionary, Cambridge Dictionary, etc.
  • then I import the dictionary my uncle Joe prepared (and imported from other sources),
  • all URLs in the imported dictionary are compared to trusted domains I defined at start -- if the domain matches, I set entire URL as trusted, if not -- it is not trusted,
  • for all not trusted URLs I will scan them through URL scanner, if the URL is legit, I set it as trusted, if not mark them as threat,
  • then when presenting web page using given URL -- if the URL is trusted I display regular (clickable) link, if not not trusted, the URL can be viewed, copied and pasted, but it would be not possible to click on it (it would be inactive/disabled link), for the known threat -- I am thinking (it has to be in some kind viewable to search for it and report this),
  • and if at some point later I check manually the URL is fine, I can tick it or I can add entire domain to trusted ones, so this would set trusted for a bunch of URLs as described above.

Do I miss some vector attack here, or this policy is sound for both sides?

Improve this question
12
  • 3
    Why define safe domains yourself? Automate the process of URL assessing using the many tools out there for the job.
    – schroeder ♦
    Commented Sep 29 at 16:09
  • 3
    What attacks are possible? What are you worried about? What happens if the domain is not safe? How are you defining "user friendly"? The only thing your design is doing is making the URL clickable. That's very little protection to begin with.
    – schroeder ♦
    Commented Sep 29 at 16:11
  • 1
    @schroeder User friendly meaning, you don't have to check every URL stored in the database one after another. What attacks are possible with URLs? Somebody set dictionary for circulation with quote from "Benjamin Franklin totally not malware" attaching to such title URL for his/her site with malware. That's for starter. As for the tools for URL verification, good point, thank you, this will come handy of course. As for the security measure -- this is why I am asking the question :-).
    – greenoldman
    Commented Sep 29 at 16:15
  • 3
    @greenoldman Even just for learning I would expect a certain level of trust. After all you don’t learn anything from a random heap of URLs. At some point you need curation, whether for security or other reasons. There are a lot of ways to get hurt by non-malicious content, be it is just from wasting a ton of time or learning something wrong. Can you define what you consider "secure" or better yet what you would consider "good" data?
    – MisterMiyagi
    Commented Sep 29 at 17:05
  • 2
    @greenoldman: Then why not restrict the references to a handful of sources that are known to be both non-malicious and accurate? If you don't know the URL, then don't show it at all -- for most people, not having a reference is probably preferable over ending up on a site that contains malicious, illegal or just nonsensical data. You can still save all user-provided URLs and occasionally check it's worth extendeding the whitlist of trusted URLs. For example, if you see lots of URLs pointing to some popular dictionary which you weren't aware of, then you could simply re-classify those URLs.
    – Ja1024
    Commented Sep 30 at 13:50

1 Answer 1

Reset to default
2

You should save all user-provided reference URLs in the database, but only display those whose target is known to be both non-malicious and accurate. For this purpose, create an (extensible) whitelist of trusted domains.

If you don't know the URL yet, then don't show it at all – for most people, not having a reference is likely preferable over ending up on a site that contains malicious, illegal or just nonsensical data. You should still save all user-provided URLs and check whether it's worth extendeding the whitlist of trusted domains. For example, if you see lots of URLs pointing to some popular dictionary which you weren't aware of, then you could simply re-classify those URLs, so that they will be shown from now on.

To make monitoring the submitted URLs easier, group and count them by domain. Then you can, for example, occasionally check the top 10.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .