Assuming that your side initiates every communication, then yes, blocking every connection to your machine from IP addresses you didn't connect to is a good idea.
But note that this is firewall basics. Most client-side firewalls don't allow any incoming connections initiated from a remote machine unless you specificallly allow them. They do more than you imagine, too - they keep track of active tcp sessions and only allow packets from ips that are in an active session with you. Once the session ends, packets from that IP will be blocked again.
If you're talking about a home network, then the firewall should run either on or right behind your internet router, eg the box providing you with internet access. If you don't allow any connection initiated from the outside, the devices on your home network are much better isolated from the internet.
Usually broadband internet providers give you a box which has such firewalling capabilities. They might or might not be enabled, but it's usually trivial to configure.