I was wondering, What is the most damage someone can do with your IP address and what attacks can be carried out?
If you have dynamic IP address (using DHCP, PPPoE) you are just fine. Your address gets changed in a interval of some hours or you share the IP address with tens of hundreds of pepole.
If you don't have open forwarded ports there is a small risk (if your ports/apps aren't properly configured) this by default is off.
So no worries. If they have the power, they could DDoS the ISP and slow your internet or cut it off. But what point would it serve?