Calculating the Integrity Check Value (ICV)

The Authentication Data fields in the AH and ESP Headers are variable-length fields, each of which contains an Integrity Check Value (ICV). The field is variable length to accommodate variations from ICV algorithms, and the length is specified by the selected function. This is an optional field: It is included only when an authentication service is in use for the SA that corresponds to the header, and information about the ICV function in use is maintained along with the rest of the SA data.

The ICV calculation is a bit tricky in that some of the data being authenticated may be modified en route, such as IP header hop counts. According to RFC 2402 the AH ICV is computed on the IP header fields that either don't change in transit or whose values on arrival can be predicted, the AH header itself (though the Authentication Data field is set to zero for the calculation), and the upper-level protocol data that is being authenticated (this is assumed to be unchanged in transit).

The ESP ICV, according to RFC 2406, is computed on the entire ESP packet, excluding the Authentication Data field. This includes the SPI, Sequence Number, Payload Data, Padding (if present), Pad Length, and Next Header; the last four fields will be in ciphertext form, since encryption is performed prior to authentication.

These are the suggested algorithms for ICV.

Message Authentication Codes (MACs), the results of which are then encrypted with an appropriate symmetric encryption algorithm (for example, AES)

Secure hash functions, such as MD5 or SHA-1 (an updated version of SHA)

To comply with the standard, implementations must support MD5 and SHA-1 keyed hashing, at least.