Principles for Web Traffic Management by Intermediaries

Principles The following principles should be followed by web intermediaries to balance\ security requirements with web openness:

Intermediaries must classify traffic by specific behaviors, not by client identity
Intermediaries must offer options other than block or allow
Intermediaries must favor open protocols for traffic management decisions
Intermediaries must prefer active signals over passive signals
Intermediaries must document where their services diverge from published standards.

Behavior-based classification Intermediaries must classify traffic by specific behaviors, not by client identity. Classifications based on inferred client types (such as "bot" or "automated") are unreliable as the distinction between human and automated web browsing continues to blur. Modern web applications routinely use automation, and browsers themselves perform automated requests for prefetching, security scanning, and other legitimate purposes. Human users can be represented by software, including automated clients (). Representing a user implies neither unconditional access nor an override of site policy. Behavior-based classification focuses on observable actions that may indicate abuse, such as excessive request rates, unusual access patterns, or failure to respect machine-readable signals such as caching directives and, for crawlers, the Robots Exclusion Protocol . This approach enables more precise enforcement while avoiding false positives based on client characteristics. Policies applied to every client must be general, such as not causing adverse impact on the site. More specific expectations apply according to the purpose a client declares, such as for crawlers.

Graduated enforcement Intermediaries must allow traffic unless restrictions are required by the policy of the operator they act for (client or server), or necessary to protect availability (). They must make the policies they apply and their effects clear to that operator, and offer options other than block or allow. Binary blocking decisions create unnecessary friction for legitimate users while failing to address the spectrum of potentially unwanted behavior. Graduated responses allow intermediaries to apply appropriate restrictions based on confidence levels and threat severity. Intermediaries should narrowly target restrictions to the relevant behavior, minimizing effects on unrelated users sharing infrastructure, applying the scope and granularity considerations in and . Effective responses include rate limiting (HTTP 429 status code), requests to retry after temporary overload (HTTP 503 with Retry-After), content modification per site policy (for instance, a summary or lower-quality variant), and verification challenges. Rate limiting implementations SHOULD follow emerging standards such as for consistent client communication. These responses can be applied progressively based on behavior patterns rather than defaulting to complete blocking for ambiguous cases. Challenges and content modification can prevent legitimate clients from processing a response. Where site policy otherwise permits access, intermediaries should offer a documented recovery or alternative access path that does not depend on human interaction or a particular browser's capabilities. Such a path does not guarantee access and remains subject to the availability protections in .

Protocol neutrality Intermediaries must favor open protocols for traffic management decisions and communicate responses using standard HTTP semantics. Traffic management decisions should be based on standardized mechanisms that any client can implement, rather than proprietary systems or curated lists. This allows legitimate automated clients to identify themselves appropriately and follow established conventions for access control. When responding to blocked or restricted requests, intermediaries MUST use appropriate HTTP status codes: 403 Forbidden for policy violations and 503 Service Unavailable for temporary overload or maintenance, as defined in , and 429 Too Many Requests for rate limiting, as defined in . Response headers SHOULD provide actionable information about restrictions and recovery paths. For 429 and 503 responses, intermediaries SHOULD include Retry-After () when they can estimate when the client should retry. Open protocols enable website owners to express their preferences consistently across different intermediaries, rather than managing separate configurations for each vendor's proprietary system.

Active signals over passive fingerprints Intermediaries must prefer active signals over passive signals. Passive fingerprinting techniques identify users without their participation, and create false positives for users sharing IP addresses, using privacy tools, or connecting from networks with unusual characteristics. These techniques also degrade over time as clients change behavior or implement countermeasures. Active signals provide assurance about the asserted property, because the client or its operator positively asserts a property that the intermediary can verify. Examples include HTTP Message Signatures profiles such as , Privacy Pass tokens , published IP ranges as recommended by , and forward-confirmed reverse DNS. Note that what differs is not necessarily the data itself. An observed IP address is a passive signal, while the same address published by its operator is an active signal. Authentication and privacy-preserving tokens can improve traffic management, but do not by themselves resolve how legitimate clients without recognized credentials or supported challenge capabilities are treated. Absence of an active signal does not itself establish abuse.

Implementation transparency Intermediaries must document where their services diverge from published standards. Intermediaries that develop or promote standards operate multiple services that may implement those standards differently. An intermediary that promotes a standard should disclose where its services do not follow it. While documenting every deviation may not be practical, intermediaries should prioritize disclosure in areas where they have established published guidance. Where safe and feasible, intermediaries should distinguish explicit site-policy denial from suspected abuse or temporary overload, and provide operators enough information to investigate unintended blocking. This need not disclose detection rules or sensitive signals.