These two have said everything that needed to be said and very well, so I commend them for not blowing it out of proportion but also not blowing you off. However I would like to add one thing that should give you some piece of mind. They have the IP of your router, but not your personal device. You see, devices behind a router only have private IP addresses, and your router has a public IP. The difference?

• Private(aka local): A private IP is assigned to a device by a router and only routes on the local network. So this means you and I could have the same private IP but it doesn't matter because were on different networks. A person for some reason having your private IP means virtually nothing.
• Public(aka external): A public IP is assigned to a router by an ISP and is in fact routable over the internet. So a person having your public IP does somewhat mean something but not much, as said by these two, and for one other reason.

The simple explanation is that if someone like this goon tries to in someway interact with your router then that means the request is coming from the internet and not the internal network. Routers are known for being strict with internet sourced requests, and their firewalls tend to be very prone to blocking request over the internet if they seem even slightly fishy. Also as noted with the public/private difference, your devices are a lot safer behind the guard of your router. They would have to somehow brake past or take control of your router to access your personal devices, which is unlikely as noted by the first answerer saying this is not very different from empty threat spam emails.

They have the IP of your router, but not your personal device. You see, devices behind a router only have private IP addresses, and your router has a public IP. The difference?

• Private  (aka local): A private IP is assigned to a device by a router and only routes on the local network. So this means you and I could have the same private IP but it doesn't matter because were on different networks. A person for some reason having your private IP means virtually nothing.
• Public  (aka external): A public IP is assigned to a router by an ISP and is in fact routable over the internet. So a person having your public IP does somewhat mean something but not much, as said by these two, and for one other reason.

The simple explanation is that if someone like this goon tries to in some way interact with your router then that means the request is coming from the internet and not the internal network. Routers are known for being strict with internet-sourced requests, and their firewalls tend to be very prone to blocking request over the internet if they seem even slightly fishy. Also as noted with the public/private difference, your devices are a lot safer behind the guard of your router. They would have to somehow break past or take control of your router to access your personal devices, which is unlikely as noted by the first answerer saying this is not very different from empty threat spam emails.