By just being connected to the Internet ther is risk of being vulnerated.
This risk can be mitigated with software updates and some security products (antimalware, firewall, intrusion detection systems, etc...).
On an ideal setting, the risk is very low. You would be running behind one or two routers on a NAT, with a fully updated operating system, no services open to the public network... and no port forwarding anyway.
To get to you on that situation, they need to compromise the routers first.
Then again if you are connected directly to the ISP (not behind a NAT nor a proxy) the risk is higher.
In theory the attacker will need an exploit for a vulnerability in your particular system. And the chances of an attacker picked at random knowning what vulnerabilities you have is low....
But! there are two scenarios in which this is not the case:
- If you are a target of a persistent treat, they will find a vulnerability sooner or later. When they do, it will be race to see if you patch it before they exploit it.
- If the attackers are looking for a random victim*, they will look for hosts that are vulnerable to the exploits they have, instead of looking for exploits for the vulnerabilities you have. And that search will probably be automated. Given enough time and enough exploits laying around, they will get you.
*: they may looking for hosts to add to a botnet, for example.
All of this is made worse if you have particulary vulnerable services installed... or simply poorly configured ones. For instance a web server, a database engine, or file sharing service can be targeted.
And finally you are hardly "doing nothing else". As you recon, you would download updates... perhaps your dns gets poisoned and it start directing you to a fake update server (that may even appear to have valid certificates and digital signatures due to a collision attack on the hash algorithms used to create those by the real author). And then you get infected from the updates.
Also, what else is installed on your machine? Do you have... I don't know... a Java updater? Flash updater? Chrome updater? Acrobat Reader updater? etc... any of those could be used to harm you.
Also, many user will sync files with a remote server, via Dropbox, OneDrive, Google Drive, etc...
Did you remember to disable remote assistance? Do you have a convenient Team Viewer service running all the time? How about that tool that comes with the drivers for your audio or video adapter that will look for drivers updates?
Edit: no, I'm not saying that you should not update. It is a higher risk to not update and be stuck with and old version for which there will be known exploits that will continue to work forever because you never update. It is better to take the risk of updating, even considering the occasional faults update form legitimate source.
Edit 2: yes, your router can be compromised. Heck, even "mine" probably is compromised and I don't even know. That is because "my" router is form the IPS and they don't give me full access to it. That's why I have a second router behind it, and this one is trully mine! I can set a more secure wifi on it, I can also see its activity logs, keep a backup of its system, and I can even update it's firmware (I did once already).
Edit 3: By the way, by just being connected to the Internet is how honeypots work. Of course honeypots are usually intentionally vulnerable. By keeping you machine updated you are mitigating the risk, but not eliminating it. There will always be a residual risk... the only way to actually delete the risk is to be disconnected.