You are correct when you say an attacker spoofing IP may not receive traffic back, but they may not want to. They may want traffic sent to another IP address - possibly for denial of service attack on that IP.
Alternatively, there are attacks which just require the initial part of the handshake to take down your defences (you mentioned SYN flood)
Less of a direct risk, but also relevant is general traffic load. My take on this is to disallow as much as you can at the perimeter - this includes traffic types and ports you don't use, and also traffic which is effectively invalid. It is simple to do on most routers, and it means any deep inspection firewall has to trawl through less packets thus reducing the load.