To add my two cents: it depends.
Also, this sounds like the kind of process that could easily fail (i.e. you forget, fall asleep at the keyboard, or need to allow someone else to use the connection).
If you are chosing to unplug it when unused as opposed to making it resilient against attacks, then I think it is in fact a net negative.
If you are chosing to unplug it even though you have done all you can to make it resilient to attack, then that might be acceptable as a short-term stopgap measure while you figure out a more permanent solution.
But I don't think there is any case where removing ethernet cables is a good long-term solution.
Even if doing so would not generally cause other issues, it simply doesn't seem like a solid solution - it sounds like avoiding an issue rather than addressing it (I may very well be wrong).
A possibly more productive line of reasoning might involve considering what that router being compromised could lead to, and what counter-measures might apply.
I suspect your ISP modem is more likely to be compromised (e.g. ala mirai), which may or may not lead to your router also being compromised, but would in most cases be functionally equivalent - someone else potentially controls your routing and may be able to see unprotected traffic. So you sort of need to have a plan against that either way.
In that sense, unplugging the ISP modem might be more useful, but you remain exposed when it is plugged in - and I am not sure the reduction in attack surface (if restricting the amount of hours a device is online can be seen as reducing that) is really going to be effective. This is all the more so as ISP deploy large numbers of identical devices, so most likely, if your modem was 'seen' online, attackers would know what it would be vulnerable to - you just give them a bit less opportunity to launch the attacks that they know work against your ISP.
So if I was you, I would look into how having your network path compromised would affect you, and do something about that. It certainly doesn't hurt to keep an eye on your router's logging, notably, to (try and) detect signs of compromise, but I don't know that that is the primary risk I would be concerned about.