There is a network that is protected with a firewall and only one IP is whitelisted.
Assuming the attacker knows the whitelisted IP, what would prevent an attacker from sending traffic via a spoofed IP to access the network?
I'm not entirely sure what you're asking.
If you're asking, "are there other controls that a defender can use to prevent inbound connections from spoofed IPs?", then yes; TTLs or other metadata of the traffic is one possible avenue, for example. Even the data content could be used(i.e. blocking unexpected data/formats).
If you're asking, "is there anything inherent in how networks work that would prevent this?", then no; spoofing a source IP doesn't affect delivery(unless you use a bogon/martian and happen to get it dropped by an intermediary system).
You won't get any data back from the attacked machine, and you will have a very hard time if you're using a protocol like TCP, where it wants to establish a connection via a handshake... but it's certainly possible.