Assume a server and a client that are each on physically secure, uncompromised networks. However, between each network is an attacker that is capable of viewing traffic.
This basically means: "Assume A, the throw A out the window, and stop assuming it."
The traffic between the two devices is encrypted with 256-bit AES using a key shared by a physical side channel (thumb drive).
This makes the rest of your setup irrelevant.
The two device's firewalls are set to drop all traffic not from each other's IP address.
This is pointless and doesn't add much in terms of security. It is trivial to re-write the TCP header and spoof the source.
It seems that you are saying: "Point A and Point B use end-to-end encryption with a symmetric key that is known to A and B, and never transmitted in-band. If an attacker compromises [any portion] of the network between A and B, what can the attacker do versus what can they know?"
Assuming you're using AES256 to encipher your data prior to leaving point A, and point B decrypts it, then the middle man cannot do much.
What they can know - that data transactions and conversations are taking place between the two points along with other metadata.
What they can do - block, re-route, cache, and otherwise tamper with the traffic.
Since your setup does not mention authentication, only encryption, then a MiTM attack could (in this setup) decrypt your data, cache it, then re-encrypt it and send it on. Neither point A or B would be aware of it.
"How do they decrypt my data???" - if this is a state level actor with resources, we can assume they have the resources to social engineer themselves a copy of the key, or had malware pick up a copy of it when it was in a computer's USB drive, etc... ).
Your main problem is you are using encryption without authentication. Now, if you use encryption with message signing and authentication, if the MiTM is tampers with your traffic at all, both endpoints will know. (Well... could know depending on your implementation).