I don't know your level of knowledge in this area, but I'd like to answer your question with a layman-level explanation of what an IP address is.
An IP address is very much like your postal address. If you want to correspond with someone through the postal system, you need their postal address and they need yours so you can send each other letters. Similarly, to exchange data or send messages online, your computer and the other computer or server must each know the IP address of the other.
When you type in a web address, your browser first corresponds with an IP address it already knows, which is the IP address of a DNS server. Google's DNS server has IP address '126.96.36.199', for example. It asks this server for the IP address of the server that hosts the website you are looking for. The DNS server gives it if it exists, and your computer then corresponds with that IP address to get the website data.
So there are many servers on out there that have known your IP address, just like any business you've sent letters to or received services from likely know your home address.
Your IP address is not a secret, or a password. It is the address used to do all legitimate internet transactions and services with you.
A person who wants to hurt or harass you can't cause you much trouble if they don't know where you live. If they don't have your home address they can't even find you to do anything. So getting your street address is the very first minimal prerequisite to messing with you or your home, and thus many people who are the victims of stalking or harassment try to conceal their home address, and would be distressed to learn that their stalker or angry ex has learned it.
Similarly a hacker can't do anything without knowing which computer to hack. If they don't know your IP address they can't even find you to begin their work. But it's really just the minimal first step. While not knowing the address prevents them from working, knowing the address doesn't really get them far. Like an angry ex who wants to steal things from your house, knowing where to go is just the first step. Once they find out where to go, they don't really have super secret info or some crazy advantage. Once they get there, they have no more power than any random person who walks by your house. They are still left with all the work of breaking in.
A hacker who knows your IP has not gained any special privileges or powers. A hacker who brags about knowing your IP is like a burglar bragging that they know your postal address. In fact, it's an even more ridiculous brag on the hacker's part, because a determined burglar can often break into a house, and even if they fail their attempts usually damage the house. But hacking is different. When a better physical home security is invented, only some homes get it, because the upgrade costs money to install. When new computer security is invented, most everyone gets it, because the software provider pushes the security patch to everyone's system automatically for free. So if your computer has the latest security updates, then it has the same security system that world-class experts have built to resist world-class hackers. The only disadvantage you have is if you have configured your system in a way that grants strangers access, like accidentally leaving your garage door unlocked (leaving your router set to use the manufacturer-default password) or if you didn't change your locks when you broke up with your ex (someone knows your iCloud password).
If you've got the latest security updates (you do update your computer right?) and an antivirus program, and you haven't left anything vulnerable (you don't give your passwords away or leave them as the default right?) then your hacker friend still faces 99.9% of the challenge of hacking after he learns your IP address. He's found your castle, but now has to attack it.
The one thing an amateur can do to a properly-configured system is called a DDOS attack. And it is basically abusing the legitimate internet protocols to overwhelm your system with spam messages. It's like someone sending you 10,000 junk letters every day, making it impossible for you to sort through and find your real mail.
So your IP address is like the street address of your home. And the hacker is like an angsty punk you met downtown shouting "I know where you live!". Sure buddy, plenty of people know where I live, that's why doors have locks.