maybe the blinking box was just a distraction.
Maybe not. Maybe the box was pre-scripted to do something malicious.
3 s is really a lot of time for a computer.
In that time, he definitely has gotten an impression of the IP addresses used (either by DHCP or by just sniffing an ARP request or couple dozen of these), has very likely been able to determine the private and public IP address of your router, and get an overview of active PC-type machines in your network (assuming they are still so noisy). It will have been enough time to open a port using UPnP to the inside and to phone home to some server to let that server know "where you've been".
You didn't clearly answer the question whether your router had default passwords when the person plugged in his box. If it did, yes, you might have been compromised. I'd ignore the people saying that they need to wait seconds for admin pages of their router to load – I don't know your router, so it might a) not be that crappy slow in the first place, b) have a management interface secured with the same password that is much faster to interact and c) to do something with the interface, you don't actually have to wait till all the login pages have completely loaded if you know what URLs you need to load with which parameters.
Now, default passwords aren't that helpful if the attacker doesn't know the type of device you have. But, again, 3s is pretty long, and that would be enough to figure out what device you use, and then try the default credentials. Also, there might have been prior knowledge – most people just have the router their ISP gives them, and it's pretty easy to guess people's ISP (or you can read it from their public IP address, and again, 3s is a long time if you're well-prepared).
What can be said about default credentials also applies to unpatched security problems with routers. I've personally connected to LANs of more than one router where the web interface was actually that buggy, it leaked configuration information that could be used to gain access on the LAN, even if you didn't have the credentials.
And, all that I'm saying here also applies to the other devices in the network. Maybe the attacker wasn't after the router at all, but after Windows machines with security holes – which might come in very handy, eg. as servers for illegal content.
Now, people nowadays seem to prefer to get victim machines/networks with much less risk – simply by sending around malware, hoping that one in ten thousand (millions?) emails leads to an infection, building a botnet.
So, I'd personally consider your network attacked, but it's a bit doubtful what the attack surface would be. I'd hence try not to be overly paranoid about that. That is, unless you have reasonable concern (not a conspiracy theory!) that someone would have enough political/financial/intelligence interest in getting access to your computers – for example, to exfiltrate secrets from your employer, who trusts you and your laptop — to actually send someone to take the risk to show up in person! Now, someone who's been hired to do so probably wouldn't need to ask about how much income you have, how many people live in your household etc.
However, restoring factory settings on your router, flashing a clean firmware image (or just buying a new one, if you have a cheap one), setting new, separate, unguessable passwords for router access and WiFi, scanning all your networked machines for viruses and being twice as careful as soon someone tries to convince you to hand over money or something similar based on "secrets" that e.g. only the tax office or your employer should have (but an attacker might have gotten access to on your computer) surely would not be too cautious. Good luck!
