There is a network that is protected with a firewall and only one IP is whitelisted.
Assuming the attacker knows the whitelisted IP, what would prevent an attacker from sending traffic via a spoofed IP to access the network?
Nothing prevents an attacker from getting at least one IP packet through to a destination IP address behind a firewall if the hacker spoof's a whitelisted IP address for the packet's source IP address.
SO
The attacker won't get any packets back, because any response IP packets will be routed to the spoofed IP address (where they might flood the network connection for the host with the spoofed IP address); and
Depending upon what kind of packets the attacker is sending using the spoofed IP address and the frequency of these spoofed packets, the firewall might apply security policies that either block or throttle this spoofed packet stream.
e.g. A SYN Flood is a situation where DDoS protection would throttle IP traffic from the spoofed IP address. see Cloudflare SYN Flood
Regardless of what the firewall's security policies are, at least one UDP/IP packet or at least one TCP/IP SYN packet would get through using the spoofed whitelisted IP address.
I'm not entirely sure what you're asking.
If you're asking, "are there other controls that a defender can use to prevent inbound connections from spoofed IPs?", then yes; TTLs or other metadata of the traffic is one possible avenue, for example. Even the data content could be used(i.e. blocking unexpected data/formats).
If you're asking, "is there anything inherent in how networks work that would prevent this?", then no; spoofing a source IP doesn't affect delivery(unless you use a bogon/martian and happen to get it dropped by an intermediary system).
You won't get any data back from the attacked machine, and you will have a very hard time if you're using a protocol like TCP, where it wants to establish a connection via a handshake... but it's certainly possible.