Can iptables filter out incoming datagrams that are IPSec-tunneled packets going to port 25?
Assuming that you mean "can host X, which terminates the IPSec tunnel, apply iptables filters to the de-tunneled traffic?" then the answer is yes.
Here's a Server Fault Q&A on this exact topic - With iptables, match packets arrived via IPSEC tunnel - which states
You need to use the policy module, and specify the ipsec policy, to match this traffic.
and gives as an example:
iptables -A INPUT -m policy --pol ipsec --dir in -p tcp --dport 12345 -j ACCEPT
Unfortunately, documentation is pretty sparse. Googling for it will find you things like this:
I'm looking for for in-depth documentation for the iptables "ipsec policy match" module. It might be just the right thing to solve my iptables/ipsec-problems; but the information given with "iptables -m policy -h" is not very intuitive.
(and of course some help for using it is found in the ensuing thread).