Just like with any external resources, you're leaking a considerable amount of information to a third-party site if you're loading the external font in the background. These privacy concerns might be the main reason why they are whitelisting a few trusted sites.
Most notably, you're leaking the URL of the current site via the
Referer header. Imagine a password reset page with a secret reset token in the URL that loads an external font - the token would immediately be leaked to that third-party service. (E.g., this bug report refers to this exact problem.) Even with non-secret content in the URL they would be able to collect usage statistics of your site by recording page visits, user-agent strings, etc.
Another threat of third-party fonts would be defacement (although you don't risk script injection which you would with external JS). If the third-party site was compromised, an attacker could modify the font in a way that would deface your website by changing how the caracters of that font are displayed. But even if you don't expect the site to be compromised you have to rely on their general availability. So if that site isn't a reliable CDN, I would consider hosting the font myself.