Common URLs are for example
http://xxx,https://xxx,mailto:xxxso I could just whitelist those 3 types of URLs.
That's exactly what you should do. Whitelist these three protocols. You could optionally include ftp: for the rare cases someone submits an FTP link. You don't need to allow anything else if you don't have a special unusual use case in mind.
At the same time I'm worried, are there are potentially bad URLs?
There are pseudo-protocols that have side effects. javascript: URLs should definitely not be possible. The file: schema is dangerous as well, because local files are often more privileged than web content. On Firefox you especially don't want to allow chrome: and resource: URIs.
Since it's not possible to keep track of all the custom URI schemes for every possible browser you should whitelist a few plausible protocols, as you proposed yourself.