]> Mozilla

mt@lowentropy.net

HTTP next generation unicorn sparkling distributed ledger This describes a content-coding for HTTP that enables progressive processing of content, without sacrificing integrity. About This Document The latest revision of this draft can be found at . Status information for this document may be found at . Discussion of this document takes place on the HTTP mailing list (), which is archived at . Source for this draft and an issue tracker can be found at .

Introduction Hash-based integrity is a powerful tool for ensuring that content matches expectations. Hashes are often the basis of digital signature schemes that can authenticate content, or at least provide verifiable proof of authorization. The main challenge with hashes is that the entirety of a piece of content needs to be acquired in order to validate the hash. This document describes a method of hash generation that can be verified progressively. This enables systems that depend on progressive processing, such as streaming processing of a long sequence of records, or performance-sensitive applications like HTML rendering. The way this works is by splitting a document into chunks, or chiclets, each comprising a length, some number of bytes of content, and a hash for the next chiclet. This is shown in .

Overview of Format ]]>

The hash of the entirety of the content can thereby be represented by the hash of the first chiclet. This enables progressive processing of content, providing strong integrity for all content bytes, provided that each chiclet is only released after being validated; see . That can reduce latency -- or at least perceived latency -- for applications that are sensitive to delays. The cost of this scheme is that there is a small amount of overhead. Additional bytes for the hash and length fields are interspersed with content bytes. The relative cost of that overhead depends on the size of chiclets; an average chunk size of 4k adds an 0.83% overhead. Constructing content in this form requires that the entirety of the content is available; see .

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. This document uses the varint encoding and presentation language developed for QUIC .

Chained Hash Integrity Format Any continuous stream of bytes can be turned into a stream of variable-length chiclets. This format allows for progressive validation of the integrity of content.

Format Description The format comprises zero or more non-final chiclets and a final chiclet (which can be the first). Content bytes are therefore interspersed with framing and integrity bytes, increasing the size of the content. A non-final chiclet comprises a non-zero length (for framing), between 1 and 2⁶¹-1 bytes (inclusive) of content (see ), and a 32 byte hash (that is, integrity bytes). Lengths are encoded using the variable-length integer format from . The hash is the output of the SHA-256 algorithm that has processed input of the chiclet that follows. The final chiclet includes a zero-valued length field, any number of bytes of content up to 2⁶¹ bytes, and no hash. The final chiclet ends when content ends, which means that this format can only be used where there is a distinct end to the protected content. That might mean the end of a file, the end of a stream, or the end of an HTTP message; see . The hash value that is computed from the very first chiclet is not written to the output. Instead, it needs to be carried outside the content; see for one method of indicating this value. The format is illustrated in .

Layout of Content Encoding

The format uses SHA-256 for integrity ; see for details.

Consumption and Validation An entity that receives a stream of bytes in this format will need a hash value for the first chiclet in order to validate the content. That value might be provided via some alternative communication channel; one method for carrying this hash value in HTTP fields is defined in . Without the hash of the first chiclet, it is not possible to validate the integrity of the content. Content can still be processed by removing the length and hash bytes. The process for progressively validating and releasing content bytes starts with an expected hash value, then proceeds as follows:

1. Initialize a new instance of SHA-256 .
2. Read a variable-length integer. Update the hash instance with each byte as it is read. Abort with an error if the stream ends before reading a complete value.
3. Read the indicated number of bytes, plus 32, from the stream. If the length value is 0, read to the end of the stream instead. Update the hash instance with each byte as it is read. Abort with an error if the stream ends before reading this amount.
4. Finalize the hash and generate hash output. If the hash output is not identical to the expected value, abort with an error.
5. Release the content bytes.
6. If the length value is non-zero, replace the expected hash value with the last 32 bytes read from the stream and return to step 1. Otherwise, end the process successfully.

This process ensures that released bytes are all integrity-protected, according to the value that was provided. To account for variations in how chiclets are constructed, multiple hash values might be provided. An entity that wishes to validate against any value from a set of provided hash values matches the hash of the first chiclet against the provided set. Once the first chiclet is processed, subsequent chiclets are validated against the single hash from the preceding chiclet.

Construction In order to produce hashes, content needs to be processed in reverse order. Content is split into multiple pieces, which are turned into chiclets. These chiclets are hashed, starting from the last, placing the resulting hash from each chiclet at the end of the immediately preceding chiclet. When constructing content, if the content is processed in a forward direction, chiclets might be written with placeholders values for hashes. Hashes cannot be added until all subsequent chiclets - and their hashes - are computed. The construction of chiclets can operate on fixed-sized pieces, or application needs can drive how content is split. For instance, an image that can be displayed at progressively-higher levels of quality can have certain points at which significant quality improvements can be realized; aligning chiclets to those points might enable better incremental processing. The fact that content needs to be processed in reverse order makes this format ill-suited to situations where content of arbitrary length is streamed. It is also poorly suited to use with dynamically-generated content, especially where content is latency sensitive, because it requires two processing passes to produce the format. It is therefore best suited to static resources where the format can be pregenerated and saved.

Length Constraints An implementation that processes input needs to hold an entire chiclet in its memory before releasing it. Implementations that accept untrusted inputs might refuse to support large chiclets to protect against memory exhaustion attacks. Implementations of this specification MUST support chiclets with length values of up to 16383 bytes (2¹⁴-1). This corresponds to a total chiclet length of 16417. Larger chiclets can be used with prior arrangement. The absolute maximum length of a chiclet is determined by the length encoding in SHA-2, which is expressed in bits as a 64-bit value. Consequently, the maximum length is 2⁶¹-1 bytes, not the 2⁶²-1 bytes of content that can be expressed in a QUIC variable-length integer. For a chiclet of that size, the length and hash further reduce the available content length by an additional 40 bytes to 2⁶¹-41; the final chiclet might reduce that by as little as one byte to 2⁶¹-2 bytes of content. Of course, there is little value in having such large chiclets, which directly act against the goal of incremental processing. An abusive implementation might also seek to generate wasteful work on receipients by constructing many small chiclets. While small chiclets might be necessary for some applications, very small lengths can add significant amounts of overhead. Recipients therefore need to look for abusive patterns, such as receiving a large number of chiclets below a given threshold, rather than have a strict lower bound on chiclet length.

HTTP Usage The use of progressive validation is well-suited to HTTP applications that rely on the streaming and incremental processing of message content. A content-coding is defined to describe content that can be progressively decoded; see . A digest format is defined to allow the hash of the first chiclet to be signaled; see .

Content-Coding The chi content coding indicates that the content of the request is a sequence of chiclets, following the construction defined in .

Digest Format The chi hash algorithm label identifies the use of the format described in when carried in the Content-Digest or Repr-Digest fields that are defined in . The chi digest format is only useful in combination with a very specific chi content-coding; see . Any change to the length of chiclets in that coding will invalidate the value of the digest format. This also means that use with Repr-Digest might not be useful if the length of chiclets could change between different encodings of the same representation. The value associated with the chi hash algorithm is a 32-byte binary value, represented as a structured field Byte Sequence; see Sections and of for details. Unlike other hash algorithms, the chi hash algorithm value can only be used during the removal of the chi content coding. It therefore provides integrity for content, but not necessarily the exact sequence of bytes that appear in the message. The same sequence of bytes can have multiple valid expressions of the chi content coding; or, another content coding could be applied after the chi coding. Note that the Content-Digest and Repr-Digest formats do not allow for multiple values of the chi hash algorithm. Only the last value is visible to a recipient; see .

Example This example shows content of 47 bytes, split into chiclets of 16 bytes. This chiclet size is far smaller than would be used in practice, but it allows the example to show multiple chiclets. The resulting 3 chiclets are shown in , with the hash of each, starting from the last.

Example Chiclets

The hash of the first chiclet can be carried in the Content-Digest field, as shown in the HTTP response in .

Example Response Header

The 114 bytes of response content are shown in in hexadecimal.

Example Encoded Content

Empty content is encoded as a single final chiclet, comprising a single byte with a value of 0. The chi digest of empty content is therefore the SHA-256 hash of that one byte, as shown in .

Fields for Empty Content

Note that these examples use the minimal encoding for the variable-length integers. Longer encodings for those integers would produce different digest values.

Security Considerations The security properties of this feature is intended to be broadly equivalent to simple hashing of content, with the added condition that a recipient can use parts of content prior to receiving every byte. The integrity property depends on the hash function chosen (), but exposes recipients to a risk of truncation (). There is also a resource exhaustion exposure, particularly for recipients with limited resources ().

Hash Choice This document relies on the second pre-image resistance property of SHA-256 . If a recipient holds a SHA-256 hash for some content, an attacker that wishes to substitute content needs to find a second plaintext that will produce the same hash. This document does not define the use of an alternative hash function. In the event that attacks on the second pre-image resistance of SHA-256 are found, new labels can be defined in a new specification. The definition of those labels could reuse the construction in this document with a different hash function.

Truncation The use of a streaming format means that content might be released to an application for processing, with no guarantee that subsequent chiclets will be delivered. Any application that uses this format therefore needs to be robust in the event that content is truncated. The final chiclet contains a zero-length prefix. The prefix of every chiclet is hashed, but the hashing of a zero length ensures that the final chiclet is easily distinguished from any non-final chiclets.

Chiclet Size The requirement to hold an entire chiclet until it is validated creates a denial of service risk; see for discussion and mitigations.

Non-Identifying Integrity The use of a cryptographic hash function provides pre-image and collision resistance, but not uniqueness. The same sequence of bytes can produce many different integrity values. As a result, the hash of the first chiclet cannot act as an identifier for content.

IANA Considerations TODO: register both chi things.

References Normative References In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. This document defines the core of the QUIC transport protocol. QUIC provides applications with flow-controlled streams for structured communication, low-latency connection establishment, and network path migration. QUIC includes security measures that ensure confidentiality, integrity, and availability in a range of deployment circumstances. Accompanying documents describe the integration of TLS for key negotiation, loss detection, and an exemplary congestion control algorithm. Federal Information Processing Standard, FIPS The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document describes the overall architecture of HTTP, establishes common terminology, and defines aspects of the protocol that are shared by all versions. In this definition are core protocol elements, extensibility mechanisms, and the "http" and "https" Uniform Resource Identifier (URI) schemes. This document updates RFC 3864 and obsoletes RFCs 2818, 7231, 7232, 7233, 7235, 7538, 7615, 7694, and portions of 7230. This document defines HTTP fields that support integrity digests. The Content-Digest field can be used for the integrity of HTTP message content. The Repr-Digest field can be used for the integrity of HTTP representations. Want-Content-Digest and Want-Repr-Digest can be used to indicate a sender's interest and preferences for receiving the respective Integrity fields. This document obsoletes RFC 3230 and the Digest and Want-Digest HTTP fields. Informative References This document describes a set of data types and associated algorithms that are intended to make it easier and safer to define and handle HTTP header and trailer fields, known as "Structured Fields", "Structured Headers", or "Structured Trailers". It is intended for use by specifications of new HTTP fields. This document obsoletes RFC 8941.

Acknowledgments was a co-author on a previous iteration of this work.