I have a specific environment in which TCP is not optimal due to its general assumptions, and thus needs to be improved. This small improvement has been done successfully. It operates at various layers, from 2 to 4, and needs read/write access to some information displayed in the TCP packets, such as the congestion window size. I was now wondering whether I could extend my proprietary protocol stack to operate with ipsec as well. The problem is that ipsec encrypts everything over ip, and I need rw access to some TCP values (not all of them). During the configuration time, I have root access to every component of the network (hosts, routers, ipsec implementations, ...), but during run time, only end hosts should be able to decrypt the whole payload. I am not a security expert, and I was wondering if you could help me to figure out what options were available for me to achieve that goal, without compromising too much the security (at least the payload).


  • I think that we'd need more details about your protocol to answer this. Commented Apr 5, 2012 at 18:33
  • You may have unintentionally used IPSec for something that another endpoint solution resolved already -- please provide more details about your modifications, etc. so we are not discussing hypotheticals... Commented Apr 6, 2012 at 7:25
  • Do you mean 'endpoint solution' as explained in is.gd/EX5GoL ? From what I understand, and although I'm not a security expert, this kind of security is about securing the host from unwanted access. What I want is to protect the information being transfered from unwanted eavesdroppers and unwanted modification, at the tcp level, regardless of the application using it. I don't know of any solutions besides ipsec providing this.
    – bob
    Commented Apr 6, 2012 at 9:25
  • What kind of detail do you need ? I don't know how much more specific I can be, without giving the whole protocol, which would be quite long, and I'm allowed to anyway. Let me just add an example: tcp considers any untransmitted packet to be caused by congestion. In my environment, many packets fail to reach their destination for other reasons. I hence needed to slightly modify the protocol stack so that the throughput would not be cut in half when the lost of a packet is not due to a congestion issue. This requires accessing and modifying the tcp window in some cases.
    – bob
    Commented Apr 11, 2012 at 8:41
  • In other words, I would like a solution that performs as well as ipsec as far as security is concerned, without impairing the enhancements of the TCP protocol made by the routers. Is that possible ?
    – bob
    Commented Apr 11, 2012 at 8:44

1 Answer 1


Are you sure you need link-layer encryption? For most purposes, application-layer encryption is sufficient.

If you really need link-layer encryption, your best bet is probably going to be to tunnel your traffic over some secure tunnel (OpenVPN, SSH, IPSec, etc.). While it is possible to design an IPSec-like protection into your protocol, that is a non-trivial task that requires competent cryptographers and will require looking at the detailed specifics of your protocol. It's not something we can do on this site.

If you don't really need link-layer encryption, I think it's going to be a lot easier to deploy application-layer encryption. Basically, you encrypt the traffic that is sent over your protocol. If your protocol guarantees reliable in-order delivery, use TLS. If does not, I suggest using DTLS. These protocols have been carefully vetted, many implementations are available, and it should be pretty straightforward to layer them on top of your protocol without fear that you will introduce some crazy subtle security bug in the crypto.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .