H2EZ: HTTP/2 Header Compression

[TODO: A significant portion of text the 'introduction' and 'security considerations' sections, is completely lifted from the HPACK Internet Draft; re-write or give credit somehow.] In HTTP/1.1 (see ), header fields are encoded without any form of compression. As web pages have grown to include dozens to hundreds of requests, the redundant header fields in these requests now measurably increase latency and unnecessarily consume bandwidth (see and ). SPDY initially addressed this redundancy by compressing header fields using the DEFLATE format , which proved very effective at efficiently representing the redundant header fields. However, that approach exposed a security risk as demonstrated by the CRIME attack (see ). A key observation of the CRIME and BREACH attacks is that they exploited the DEFLATE algorithm as described in RFC 1951 , not the DEFLATE format (also described in RFC 1951). EZFLATE , is a token-based DEFLATE compression algorithm for secure compression within encrypted communication channels. The key feature of the EZFLATE algorithm (and primary difference to the algorithm described in RFC 1951), is that LZ77 <length, backward distance> tuples may only reference a duplicated token (octet string) occurring in a previous block, up to 32K input bytes before, if the current token exactly matches in length and octet values. As shown in Section 3.1, "Take a Bite out of CRIME", of , this forces potential CRIME-like attackers to guess n-character secrets n characters at a time rather than one character at a time, making the search space no worse than a full brute-force attack. This document defines 1) the formatting of HTTP/2 header fields, and 2) how EZFLATE is used to compress HTTP/2 header fields. Finally, this document also includes implementation recommendations.

In HTTP/1.1 headers are defined as field-name, field-value pairs (see Section 3.2, "Header Fields" of ). The field-name and field-value are separated from each other by a single ':' character followed by optional whitespace. Each pair is separated by the two octet sequence CRLF (carriage return, line feed). The end of the header block is marked by an empty line followed by CRLF. In HTTP/2, the headers are still field-name, field-value pairs except that each are length-prefixed octet strings to simplify parsing. In other words, the field-name and field-value are separately length-prefixed. In this way no separator is necessary between the field-name and field-value and no separator is necessary between consecutive pairs. The end of the header block is naturally defined as the end of a HTTP/2 HEADERS or PUSH_PROMISE frame with the END_HEADERS flag set . Each field-name and field-value are encoded as an ASN.1 octet string . This simply means the octets which compose a field-name or field-value are length-prefixed with a variable-length integer which indicates the exact number of octets composing the original field-name or field-value. For ASN.1 encoded octet-strings, the length prefix is composed of at least a single octet, and optionally N additional octets, where N is exactly specified by the first octet as follows. If the length of the octets to be length-prefixed is less than 0x80 (128), then only a single octet is required for the length prefix and so the length-prefix octet is encoded as exactly the length (big endian) of the octets to be length-prefixed. If the length of the octets to be length-prefixed is greater than or equal to 0x80 (128), then the most significant bit of the first length-prefix octet is set (i.e. 0x80), and the remaining 7 bits specify the number of additional octets N (big endian) which will be added to encode the length, as an unsigned integer (big endian), of the octets to be length-prefixed .

The following are examples of computing the length prefix for converting various octet strings to ASN.1 octet strings.

The ASCII string "accept-encoding" is to be converted to an ASN.1 octet string. The length of the string is 15 octets. Since 15 is less than 128, the length prefix is simply a single octet with the value 15 (0x0f).

Hex dump of the resulting ASN.1 octet string:

The ASCII string "gzip;q=1.0, identity;q=0.5, *;q=0" is to be converted to an ASN.1 octet string. The length of the string is 33 octets. Since 33 is less than 128, the length prefix is simply a single octet with the value 33 (0x21).

Hex dump of the resulting ASN.1 octet string:

The string of octets 0x00 0x01 ... 0x7d 0x7f is to be converted to an ASN.1 octet string. The number of octets is 128. Since 128 equals 128, the first octet of the length prefix has the most significant bit set to indicate the length prefix is longer than one octet. The remaining seven bits encode the number of additional octets necessary to represent the value 128 as a big endian unsigned integer. Since encoding 128 requires one octet, the seven remaining bits encode the value 1 (0x01). The second octet encodes the value 128 (0x80).

Hex dump of the resulting ASN.1 octet string: ?@ABCDEFGHIJKLM 0000050 4e4f 5051 5253 5455 | 5657 5859 5a5b 5c5d | NOPQRSTUVWXYZ[\] 0000060 5e5f 6061 6263 6465 | 6667 6869 6a6b 6c6d | ^_`abcdefghijklm 0000070 6e6f 7071 7273 7475 | 7677 7879 7a7b 7c7d | nopqrstuvwxyz{|} 0000080 7e7f | | ~.]]>

The string of octets 0x00 0x01 ... 0xfe 0xff 0x00 0x01 is to be converted to an ASN.1 octet string. The number of octets is 258. Since 258 is greater than 128, the first octet of the length prefix has the most significant bit set to indicate the length prefix is longer than one octet. The remaining seven bits encode the number of additional octets necessary to represent the value 258 as a big endian unsigned integer. Since encoding 258 requires two octets, the seven remaining bits encode the value 2 (0x02). The second and third octets encode the value 258 (0x01 0x02).

Hex dump of the resulting ASN.1 octet string: ?@ABCDEFGHIJKL 000050 4d4e 4f50 5152 5354 | 5556 5758 595a 5b5c | MNOPQRSTUVWXYZ[\ 000060 5d5e 5f60 6162 6364 | 6566 6768 696a 6b6c | ]^_`abcdefghijkl 000070 6d6e 6f70 7172 7374 | 7576 7778 797a 7b7c | mnopqrstuvwxyz{| 000080 7d7e 7f80 8182 8384 | 8586 8788 898a 8b8c | ................ 000090 8d8e 8f90 9192 9394 | 9596 9798 999a 9b9c | ................ 0000a0 9d9e 9fa0 a1a2 a3a4 | a5a6 a7a8 a9aa abac | ................ 0000b0 adae afb0 b1b2 b3b4 | b5b6 b7b8 b9ba bbbc | ................ 0000c0 bdbe bfc0 c1c2 c3c4 | c5c6 c7c8 c9ca cbcc | ................ 0000d0 cdce cfd0 d1d2 d3d4 | d5d6 d7d8 d9da dbdc | ................ 0000e0 ddde dfe0 e1e2 e3e4 | e5e6 e7e8 e9ea ebec | ................ 0000f0 edee eff0 f1f2 f3f4 | f5f6 f7f8 f9fa fbfc | ................ 000100 fdfe ff00 01 | | ................]]>

EZFLATE is itself agnostic to tokenization methods and delimiter sets. Appropriate tokenization rules and delimiter sets depend on the type of data to be compressed. For HTTP/2 header data, each field-name and field-value are tokenized separately.

Since header field names are typically static, it is not beneficial to tokenize field names. Therefore field names are fed to the EZFLATE compressor as a single token, including the ASN.1 length prefix.

Field values, on the other hand, have varying values of staticity ranging from very static (e.g. :scheme, user-agent) to semi-static (e.g. accept, accept-encoding) to very dynamic (e.g. :path, date). Field values MAY be tokenized with following set of delimiters:

{ ' ', '\t', ';', ',' } Additionally, the set of headers { ":path", "location", "content-location" } MAY instead be tokenized with the following set of delimiters:

Special token delimiters for header values containing a path: { '/', '?', '&', '=' } The ASN.1 length prefix MAY be treated as a delimiter and therefore treated as a separate token. Finally, note that per Section 3.2 of , a sequence of consecutive delimiters are to be treated as a single unit (e.g. as one token). For examle, see and , where the consecutive delimiter characters ',' and ' ', are treated as a single token ", ".

The following is an example of converting an HTTP header name-value pair into ASN.1 length prefixed octet strings (see and then tokenizing the octet string based on the tokenization rules. The following header is to be encoded:

First the field-name and field-value are converted to ASN.1 octet strings by adding the appropriate length prefixes (see and ).

Hex dump of the header converted to ASN.1 octet strings: Next, the length-prefixed header name is taken as a single token.

Hex dump of the header name token: Finally, the length-prefixed header value is tokenized according to the rules in . The following is a hex dump of each of the eleven tokens.

Token 1:

Token 2:

Token 3:

Token 4:

Token 5:

Token 6:

Token 7:

Token 8:

Token 9:

Token 10:

Token 11:

The DEFLATE format has a fixed maximum length of 258 octets for matches (see Section 3.2.5 of ). As such, tokens with length L greater than 258 octets SHOULD be sub-tokenized into sub-tokens as follows. The number of sub-tokens N MUST be N = (L + 257) / 258 (integer division). If N > 2, the first N - 2 sub-tokens MUST be 258 octets in length. In all cases, the final two sub-tokens MUST have length greater than or equal to 129 (258 / 2). Let R be the remaining length for the final two tokens. The length of the first remaining token is L[N - 1] = R - R / 2 and the length of the second remaining token is L[N] = R / 2 (where the index is 1..N). Alternatively, tokens with length L greater than 258 octets may simply be emitted as literals. Consider the following examples:

A token T, 1292 octets in length, is to be sub-tokenized.

Thus there are six sub-tokens. The first four sub-tokens are 258 octets in length. The final two sub-tokens are 130 octets in length.

A token T, 516 octets in length, is to be sub-tokenized.

Thus there are two sub-tokens. The two sub-tokens are 258 octets in length.

A token T, 259 octets in length, is to be sub-tokenized.

Thus there are two sub-tokens. The length of the first sub-token is 130 octets. The length of the second sub-token is 129 octets.

An EZFLATE compressor can act as an oracle to an attacker probing the compression context. However, the attacker can only find out if the guess of the entire token is correct or not. Still, an attacker could take advantage of this limited information for breaking low-entropy secrets using a brute-force attack. A server usually has some protections against such brute-force attack. Here, the attack would target the client, where it would be harder to detect. The attack would be even more dangerous if the attacker is able to prevent the traffic generated by its brute-force attack from reaching the server. To offer protection against such type of attacks, and endpoint MUST use non-compressed DEFLATE blocks (see Section 3.2.4 of ) to prevent the compression of any header field whose value contains a secret which could be put at risk by a brute-force attack.

Roberto Peon and Hervé Ruellan.