I think I'll call this quote, Stilez' Truism of Data Theft, if someone else didn't get there first about a hundred times over :) :
Data is not unimportant if it can be pivoted to gain access to something important.
That means your phone number isn't at all important - until I discover that it combines with other data or can be exploited, to get me something else that is.
Let's wander around the high-hanging fruit ( @Matthew1471 got the rest of it neatly!) in "black-hat" mindset:
I can ask my mate at the phone company, or a PI, for your account information, including IMEI number (or other phone identifiers) call records, direct debit/payment info, home and contact info, date of birth or PIN if applicable, email address, perhaps your login/password to the telco's "MyPhone" service that doesn't store it encrypted and which you reused on other valuable services (or gives me an idea what kind of p/ws you're likely to choose).
If I'm hostile (business opponent, secret vendetta, harasser) I could disrupt your calls or voicemail or arrange for it to go on a "stolen" list by faking a report in your name on a landline, perhaps at a known crucial time, change your forwarding or other settings, maybe even send you new APN settings or use phone# and IMEI etc to make your phone connect to a "fake" or "poisoned" mast near you for data and voice and grab your internet traffic (after all, the US police have technology of that kind so why can't anyone else? They've been fighting hard to stop details getting into courts, too).
If I know your phone # it gets easier to spoof you to others as well, or to send SMS that superficially look like they're from you.
If you share or link your phone account, or have a "family group account" of some kind, I now have your family and children's individual phone details too, and probably any data stored under these records for these people too.
From your calls and texts I have the phone numbers of your most contacted people (most likely the people you are closest to, such as immediate family), and your workplace and key work contacts.
I can send texts in your name or with your number to them or just research them as well, using you as a stepping stone (maybe it's not all about you, and I'm using your phone account as a way to get someone elses data or to trawl for anything useful?)
Your billings also give me times of calls, and periods of use/disuse, so I can start to learn your daily routine, places you might go, and any regular wake up/bed/"out of contact" routine that might suggest your home is empty or you're sleeping at certain times.
That's just for easy starters, and an accomplice in a telco isn't exactly unheard of or difficult if I want to get one badly enough. Lets find more fruit..... suppose that's a stepping stone. What might it be a stepping stone to?
I can probably learn, either from your phone-to-telco connections, phone records, or social engineering, a lot more about your life and activities, especially if I can also grab your data traffic by leveraging these.
I can look up your phone # (or any other data I got, such as date of birth, address, email, close contacts) and correlate it with other data not from your phone. Suppose your phone gives me your name, address, DOB and email, but I can search on a black market credit card database by any of those. Now I have your credit card number and expiry date to go with them.
I'm starting to be able to call your bank and card provider and in quite a few cases if their security is based on this kind of data, I might be able to authenticate by phone as you and make inquiries... or perhaps I have a mate who can get me your financial data, maybe your full credit rating and the detailed records which it contains about your financial matters, if I have this kind of data (it's probably going to be assumed that anyone knowing this much is likely to be asking legitimately).
Your handset number probably lets me find your main ISP connection that you use daily, either on the handset, or with the same provider, or at your home address or one of the numbers you call most often. Now I have your static IP (or at least, past dynamic IPs for your home) and in some cases that could mean a way to play with your router or connection, watch your traffic, grab your webcam feed or files using the access gained to your home network and PCs (easy prey now), poison your DNS, or inject malware into traffic too. Or maybe I just want to look up your IP online and find out for sure if you really are Jaded123 on that RPG or forum.
If I'm especially keen to target you, and have $100 spare, I can phone you up plausibly as a Telco representative, take you through the Telco security screen, and offer you a nice smartphone handset upgrade as a reward for being a customer for X months and anything else in your records (that I know all about and can discuss), and you'll probably never question it or wonder if it's genuinely sent by your telco, or discover the malware I put on it first.
All of these and many others are possible. Whether they are likely is different.
That's just what social engineering and ability to target the phone-to-mast connection can (in principle) get me, given just your phone number and a few contacts. We haven't even begun wondering what else someone can get, if they get your data traffic as a result. That could include your moment-by-moment location (from your GPS sent back unencrypted by some app you never thought about twice), calendar/notes/email app sync, and the rest. Current location within 5m leads to what - NFC skimming in a crowd of your payment app? "Accidental" meeting or avoidance? Who can tell.
So yeah.
Not to be scared or paranoid. But data leads to data. Often the initial data theft is a beach-head for an issue, not the actual issue.
A phone number looks innocuous. Usually it is. But - hopefully with luck very rarely - what it might lead to, might not always be innocuous. The first stopping point isn't always the final stopping point.....