3

If I remember correctly applying ESP after AH is not doable (using transport mode), but I can't find any conclusive answer why. I assume it has something to do with ESP having an impact on the IP Header, but the only field - I assume - that gets changed is the "proto" field (AH -> ESP), which I do not know if AH authenticates. I'm talking about the following setup:

[IPHDR][Payload]
Apply AH ---> [IPHDR][AH-H][Payload]
Apply ESP ---> [IPHDR][ESP-H][AH][Payload][ESP-Trail][ESP-Auth]

3

From a technical and theoretical stand point there is nothing that prevents you from applying ESP after AH in transport mode. The proto field is restored again when ESP is removed. So unless other parts of the IP header that AH protected were modified on the way (e.g. the source IP due to a NAT) this should work fine.

But since AH is used to protect the integrity of the transmitted data you usually want to do that last. And if you use ESP with both encryption and authentication you'd do that work twice. Therefore, RFC 2401 contained the following text (it's not part of RFC 4301 anymore because the SA bundle requirement was removed):

For transport mode SAs, only one ordering of security protocols seems appropriate. AH is applied to both the upper layer protocols and (parts of) the IP header. Thus if AH is used in a transport mode, in conjunction with ESP, AH SHOULD appear as the first header after IP, prior to the appearance of ESP. In that context, AH is applied to the ciphertext output of ESP. In contrast, for tunnel mode SAs, one can imagine uses for various orderings of AH and ESP.

Also, doing authentication before encryption (i.e. if you use AH in confidentiality-only ESP) might not provide the same protection as doing it the other way around. As RFC 4303 puts it in its introduction:

Using encryption without a strong integrity mechanism on top of it (either in ESP or separately via AH) may render the confidentiality service insecure against some forms of active attacks. Moreover, an underlying integrity service, such as AH, applied before encryption does not necessarily protect the encryption-only confidentiality against active attackers [Kra01].

Where [Kra01] refers to the paper "The Order of Encryption and Authentication for Protecting Communications (Or: How Secure Is SSL?)" by Hugo Krawczyk, which argues that under certain circumstances an active attacker might be able to break the encryption in the integrity-before-encryption scheme.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.