how IPS and IPSec work together in a host

Question

I want to know how they work together . for example , we implement IPSec tunnel between 2 routers and we establish Snort IPS on Router 1. when Router 1 get the packets , they're encrypted . Now which one examine them first ? if IPS investigate the packets , it can not detect malicious content . so do we lost IPS because of IPSec tunnel ? my questions are

• Which one examine input packets first ?
• Does IPS examine packet after decryption ? or before decryption ?

I think my problem is that i have no idea about packet flow when they are received on interface .

Answer 1

Linux Based Router

In Linux the key exchange (generally ISAKMP) is performed in user space, while the security policies, keys, and encryption/decryption are performed at the kernel level. When a packet arrives the kernel first processes this packet by checking to see if the SPI and IP match a security policy. If it does the packet is processed using the policy's stored keys, and the promoted up the network stack for processing. This is still performed at the kernel level.

Now where Snort sits in the network stack implementation is important. As it's a user space program it receives packets from the kernel. This is generally performed by the kernel's network filter. Then the packets from the kernel's netfilter get passed on to user space programs like snort [1] (search Snort).

What this means for you

1. Kernel sees encrypted packets
2. Kernel determines if it has a security policy matching these packets
3. If yes, kernel decrypts the packets and promotes them up the network stack
4. Netfilter processes them, and pushes the decrypted packets up to user space
5. Snort sees decrypted packets and processes them.

This isn't the full process, and I'm might have glossed over a bit of the Linux kernel network stack. But you really don't want those details anyway.

Or maybe you do...
Dirty Details on Linux Kernel Network Stack
Dirt Details on Netfilter
Filtering IPSec packets after decryption using iptables

An Interesting Side-note: If you're sniffing packets on Router 1 with Snort or even Wireshark. Ping through the VPN. From Router 1 you should see both the encrypted ESP packet coming in, and the decrypted response in Snort/Wireshark. As verification that it will in fact see both packets.