]> University of Oslo

PO Box 1080 Blindern 0316 Oslo Norway michawe@ifi.uio.no http://welzl.at/

University of Washington

185 Stevens Way Seattle, WA 98195 United States tomh@tomh.org

University of Aberdeen

Fraser Noble Building Aberdeen, AB24 3UE UK gorry@erg.abdn.ac.uk https://www.erg.abdn.ac.uk/

National Institute of Technology Karnataka

P. O. Srinivasnagar, Surathkal Mangalore, Karnataka - 575025 India tahiliani@nitk.edu.in https://tahiliani.in/

Transport Congestion Control Working Group This document specifies how transport protocols increase their congestion window when the sender is rate-limited, and updates RFCs 4341, 5681, 9002, 9260, and 9438. Such a limitation can be caused by the sending application not supplying data or by receiver flow control. About This Document The latest revision of this draft can be found at . Status information for this document may be found at . Discussion of this document takes place on the Congestion Control Working Group mailing list (), which is archived at . Subscribe at . Source for this draft and an issue tracker can be found at .

Introduction A sender of a congestion controlled transport protocol becomes "rate-limited" when it does not send any data even though the congestion control rules would allow it to transmit data. This could occur because the application has not provided sufficient data to fully utilize the congestion window (cwnd). It could also occur because the receiver has limited the sender using flow control (e.g., by the advertised TCP receiver window (rwnd) or by the connection or stream flow credit in QUIC). Current RFCs specifying congestion control algorithms diverge regarding the rules for increasing the cwnd when the sender is rate-limited. This document provides a uniform behavior in , and specifies updates to RFCs 4341, 5681, 9002, 9260, and 9438 in . Congestion Window Validation (CWV) provides an experimental specification defining how to manage a cwnd that has become larger than the current flight size, and how to respond to detected congestion when this is the case. In contrast, this present document concerns the increase in cwnd when a sender is rate-limited. These two topics are distinct, but are related, because both describe the management of the cwnd when a sender does not fully utilize the current cwnd. An appendix provides an example of how rate-limited increase can play out.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Terminology This document uses the terms defined in . Additionally, the following are defined:

• cwnd-limited: A flow that has sent the maximum number of segments permitted by the cwnd, where the application utilises the allowed sending rate (based on the definition for TCP in Section 3 of ).
• rate-limited: a flow that is either idle or sends at a rate less than the maximum permitted by the cwnd. (Note: Section 3 of provides a more specific definition applied to the reduction of the cwnd when a sender remains rate-limited for a period of time).
• initcwnd: The initial value of the congestion window, also known as the "initial window" ("IW" in ).
• maxFS: the largest value of FlightSize since the last time that cwnd was decreased. If cwnd has never been decreased, maxFS is the maximum value of FlightSize since the start of the data transfer, and at least as large as initcwnd.

Rate-Limited Increase When a sender using a congestion control algorithm increases cwnd, the following "Rate-Limited Increase" rules apply:

• The sender MUST initialise the maxFS parameter to initcwnd when the congestion control algorithm is started. Thereafter, when the FlightSize is updated, the sender also updates the maxFS:

• Upon a reduction of cwnd (for any reason), maxFS MUST be reset to zero. This ensures that maxFS is reinitialized using the first FlightSize measurement taken after the cwnd reduction.
• When increasing cwnd, the sender MUST cap cwnd to become no larger than limit(maxFS).

The function limit() returns the maximum cwnd value the congestion control algorithm would yield by increasing for all ACKs that would be produced by successfully transmitting one window of size maxFS. For example, for Slow Start, as specified in , limit(maxFS)=2*maxFS, such that equation 2 in becomes: where cwnd and SMSS follow their definitions in and N is the number of previously unacknowledged bytes acknowledged in the incoming ACK. Similarly, with Rate-Limited Increase applied in Congestion Avoidance, limit(maxFS)=SMSS+maxFS, such that equation 3 in becomes: where cwnd and SMSS follow their definitions in . NOTE: This specification defines the method used to increase the cwnd for a rate-limited sender. Without a way to reduce cwnd when the transport sender becomes rate-limited, maxFS can stay valid for a long time, possibly not reflecting the reality of the end-to-end Internet path in use. This can be remedied by "Congestion Window Validation" in , which also defines a "pipeACK" variable that measures the recently acknowledged size of the network pipe when the sender was rate-limited.

Example The working of Rate-Limited Increase can be illustrated by showing the increase of cwnd in two scenarios: when the growth of cwnd is unconstrained, and when the rate-limited sender is constrained by Rate-Limited Increase. For simplicity, this example accounts for the cwnd in TCP segments (or QUIC packets), rather than bytes. In both cases, this assumes the initial cwnd (initcwnd) = 10 segments, as defined for TCP in and QUIC in , a single connection begins with Slow Start, the sender transmits a total of 14 segments but pauses after transmitting 10 segments and resumes the transmission for the remaining 4 segments afterward, no packets are lost, and an ACK is sent for every packet.

Unconstrained sender Initially, cwnd = initcwnd. Therefore, using initcwnd = 10 segments, the sender transmits 10 segments and pauses. Since the sender is in the Slow Start phase, the arrival of each ACK for the 10 sent segments increases the cwnd by 1 segment, resulting in the cwnd increasing to 20 segments. Subsequently, after the pause, the sender transmits 4 segments and pauses again. As a consequence, the arrival of 4 ACKs results in cwnd further increasing to 24 segments, even though the sender is rate-limited (i.e., has never sent more than 10 segments per round-trip time (RTT)).

Sender constrained by Rate-Limited Increase Initially, cwnd = initcwnd. Therefore, using initcwnd = 10 segments, the sender transmits 10 segments and pauses; note that FlightSize and maxFS are both 10 segments at this point. Since the sender is in the Slow Start phase, the arrival of each ACK for the 10 sent segments increases the cwnd by 1 segment, resulting in the cwnd increasing to 20 segments. Subsequently, when the sender resumes and transmits 4 new segments, Rate-Limited Increase constrains the growth of the cwnd because FlightSize < cwnd and therefore this caps the cwnd to be no larger than limit(maxFS) = 2maxFS = 210 segments = 20 segments.

Discussion If the sending rate is less than the rate permitted by the cwnd for multiple RTTs, limited either by the sending application or by the receiver-advertised window, a continuous increase in the cwnd would cause a mismatch between the cwnd and the capacity that the path supports (i.e., over-estimating the capacity). Such unlimited growth in the cwnd is therefore disallowed. However, in most common congestion control algorithms, in the absence of an indication of congestion, a cwnd that has been fully utilized during an RTT (where a sender was cwnd-limited) permits the cwnd to be increased during the immediately following RTT. This increase is allowed by Rate-Limited Increase.

Rate-based congestion control The present document updates congestion control specifications that use a cwnd to limit the number of unacknowledged bytes (or packets) that a sender is allowed to emit. Use of a cwnd variable to control sending rate is not the only mechanism available and not the only mechanism that is used in practice. Congestion control algorithms can also constrain data transmission by explicitly calculating the sending rate over some time interval, by "pacing" packets (injecting pauses in between their transmission) or via combinations of the above (e.g., BBR combines these three methods ). The guiding principle behind Rate-Limited Increase applies to all congestion control algorithms: in the absence of a congestion indication, a sender is allowed to increase its rate from the amount of data that it has transmitted during the previous RTT (this holds irrespective of whether the sender is rate-limited or not). Rate-based congestion control algorithms MUST ensure their maximum sustained rate is limited to a value that is not greater than would have been permitted by the Rate-Limited Increase method defined in .

Pacing Pacing mechanisms seek to avoid the negative impacts associated with "bursts" (flights of packets transmitted back-to-back). Rate-Limited Increase introduces a limit using "maxFS", which is based on the number of bytes in flight during a previous RTT; thus, as long as the number of bytes in flight per RTT is unaffected by pacing, Rate-Limited Increase does not constrain the use of pacing mechanisms.

Updates to RFCs 4341, 5681, 9002, 9260, and 9438

RFC 4341: Profile for Datagram Congestion Control Protocol (DCCP) Congestion Control ID 2: TCP-like Congestion Control According to , a DCCP CCID specifying TCP-like behavior is allowed to grow the cwnd without limit during an uncongested period when it sends at a rate unconstrained by the current cwnd. This document updates by adding the text in to specify how the cwnd is increased when the sender is rate-limited.

RFC 5681: TCP Congestion Control specifies no limit on the cwnd growth in the standard TCP behavior when a TCP sender is unable to send at the maximum rate allowed by the cwnd. This document updates by adding the text in to specify how the cwnd is increased when the sender is rate-limited.

RFC 9002: QUIC Loss Detection and Congestion Control states:

• "When bytes in flight is smaller than the congestion window and sending is not pacing limited, the congestion window is underutilized. This can happen due to insufficient application data or flow control limits. When this occurs, the congestion window SHOULD NOT be increased in either slow start or congestion avoidance."

This limits the cwnd growth in accordance with Rate-Limited Increase, but it is more conservative. This document updates by replacing the final sentence of the quoted text with the text in to specify how the cwnd is increased when the sender is rate-limited.

RFC 9260: Stream Control Transmission Protocol states:

• "When cwnd is less than or equal to ssthresh, an SCTP endpoint MUST use the slow-start algorithm to increase cwnd only if the current congestion window is being fully utilized and the data sender is not in Fast Recovery. Only when these two conditions are met can the cwnd be increased; otherwise, the cwnd MUST NOT be increased."

The quoted statement from limits cwnd growth in accordance with Rate-Limited Increase, but it is more conservative. only discusses Slow Start; Congestion Avoidance is discussed in and is addressed separately below. This document updates as follows, to specify when the cwnd is increased when the sender is rate-limited:

• In the first sentence of the quoted text, the words "the current congestion window is being fully utilized and" are removed, so that the use of the slow-start algorithm to increase the cwnd is conditioned only on the data sender not being in Fast Recovery.
• The final sentence of the quoted text is replaced with the text in .
• In the sentence that immediately follows the quoted text ("If these conditions are met, then cwnd MUST be increased by, at most, the lesser of ..."), the words "these conditions are" are replaced with "this condition is".

For Congestion Avoidance, recommends that "SCTP SHOULD increment cwnd by PMDCS once per RTT when the sender has cwnd or more bytes of data outstanding for the corresponding transport address", and further states:

• "SCTP MUST NOT increment cwnd by more than PMDCS per RTT."

These statements also limit cwnd growth in accordance with Rate-Limited Increase, but they are more conservative. This document updates by adding, after the statement "SCTP MUST NOT increment cwnd by more than PMDCS per RTT.", the text in . The SHOULD-level recommendation remains in place: a sender that follows it will not increase the cwnd while it is rate-limited. Based on the explanations in the present document, a sender can instead increase the cwnd while it is rate-limited, provided that the cwnd never exceeds limit(maxFS), as specified in . This ensures that the update applies to both Slow Start and Congestion Avoidance.

RFC 9438: CUBIC for Fast and Long-Distance Networks states:

• "CUBIC does not increase its congestion window if a flow is application limited".

This limits the cwnd growth in accordance with Rate-Limited Increase, but it is more conservative. This document updates by replacing the quoted text with the text in to specify how the cwnd is increased when the sender is rate-limited. The last sentence of regarding and inclusion of application-limited periods is unchanged by this document.

Security Considerations While congestion control designs could result in unwanted competing traffic, they do not directly result in new security considerations. The security considerations are the same as for other congestion control methods. Such methods rely on the receiver appropriately acknowledging receipt of data. The ability of an on-path or off-path attacker to influence congestion control depends upon the security properties of the transport protocol being used. Transport protocols that provide authentication (including those using encryption), or are carried over protocols that provide authentication, can protect their congestion control algorithm from network attack. This is orthogonal to the specification of congestion control rules.

IANA Considerations This document requests no IANA action.

References Normative References In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. This document defines TCP's four intertwined congestion control algorithms: slow start, congestion avoidance, fast retransmit, and fast recovery. In addition, the document specifies how TCP should begin transmission after a relatively long idle period, as well as discussing various acknowledgment generation methods. This document obsoletes RFC 2581. [STANDARDS-TRACK] This document describes loss detection and congestion control mechanisms for QUIC. This document contains the profile for Congestion Control Identifier 2 (CCID 2), TCP-like Congestion Control, in the Datagram Congestion Control Protocol (DCCP). CCID 2 should be used by senders who would like to take advantage of the available bandwidth in an environment with rapidly changing conditions, and who are able to adapt to the abrupt changes in the congestion window typical of TCP's Additive Increase Multiplicative Decrease (AIMD) congestion control. [STANDARDS-TRACK] This document describes the Stream Control Transmission Protocol (SCTP) and obsoletes RFC 4960. It incorporates the specification of the chunk flags registry from RFC 6096 and the specification of the I bit of DATA chunks from RFC 7053. Therefore, RFCs 6096 and 7053 are also obsoleted by this document. In addition, RFCs 4460 and 8540, which describe errata for SCTP, are obsoleted by this document. SCTP was originally designed to transport Public Switched Telephone Network (PSTN) signaling messages over IP networks. It is also suited to be used for other applications, for example, WebRTC. SCTP is a reliable transport protocol operating on top of a connectionless packet network, such as IP. It offers the following services to its users: The design of SCTP includes appropriate congestion avoidance behavior and resistance to flooding and masquerade attacks. CUBIC is a standard TCP congestion control algorithm that uses a cubic function instead of a linear congestion window increase function to improve scalability and stability over fast and long-distance networks. CUBIC has been adopted as the default TCP congestion control algorithm by the Linux, Windows, and Apple stacks. This document updates the specification of CUBIC to include algorithmic improvements based on these implementations and recent academic work. Based on the extensive deployment experience with CUBIC, this document also moves the specification to the Standards Track and obsoletes RFC 8312. This document also updates RFC 5681, to allow for CUBIC's occasionally more aggressive sending behavior. Informative References This document provides a mechanism to address issues that arise when TCP is used for traffic that exhibits periods where the sending rate is limited by the application rather than the congestion window. It provides an experimental update to TCP that allows a TCP sender to restart quickly following a rate-limited interval. This method is expected to benefit applications that send rate-limited traffic using TCP while also providing an appropriate response if congestion is experienced. This document also evaluates the Experimental specification of TCP Congestion Window Validation (CWV) defined in RFC 2861 and concludes that RFC 2861 sought to address important issues but failed to deliver a widely used solution. This document therefore reclassifies the status of RFC 2861 from Experimental to Historic. This document obsoletes RFC 2861. This document proposes an experiment to increase the permitted TCP initial window (IW) from between 2 and 4 segments, as specified in RFC 3390, to 10 segments with a fallback to the existing recommendation when performance issues are detected. It discusses the motivation behind the increase, the advantages and disadvantages of the higher initial window, and presents results from several large-scale experiments showing that the higher initial window improves the overall performance of many web services without resulting in a congestion collapse. The document closes with a discussion of usage and deployment for further experimental purposes recommended by the IETF TCP Maintenance and Minor Extensions (TCPM) working group. Google Google Meta This document specifies the BBR congestion control algorithm. BBR ("Bottleneck Bandwidth and Round-trip propagation time") uses recent measurements of a transport connection's delivery rate, round-trip time, and packet loss rate to build an explicit model of the network path. BBR then uses this model to control both how fast it sends data and the maximum volume of data it allows in flight in the network at any time. Relative to loss-based congestion control algorithms such as Reno [RFC5681] or CUBIC [RFC9438], BBR offers substantially higher throughput for bottlenecks with shallow buffers or random losses, and substantially lower queueing delays for bottlenecks with deep buffers (avoiding "bufferbloat"). BBR can be implemented in any transport protocol that supports packet-delivery acknowledgment. Thus far, open source implementations are available for TCP [RFC9293] and QUIC [RFC9000]. This document specifies version 3 of the BBR algorithm, BBRv3.

An Example Using cwnd Represented in Bytes The following informative example is provided for a sender that maintains the cwnd in bytes. 36 packets (or segments in the case of TCP) are sent in this example over four rounds of transmission. This shows the initial growth of the cwnd by a rate-limited sender, followed by a transmission that uses the full available cwnd. The SMSS (QUIC max_datagram_size)=1000 bytes. N is the number of previously unacknowledged bytes in a received acknowledgement. For simplicity, in this example the receiver sends an ACK for each received packet. The initial sender state is: The network path's bandwidth-delay product is such that, throughout this example, all packets in each round are sent before an ACK is received for the first packet in a round. One ACK is generated for each received packet. Round 1, the sender has 4000B to send in 4 packets (1000B); cwnd=10000 Received 4 ACKs (each N=1000); maxFS=10000; cwnd_new += N; cwnd = min(cwnd_new, 2*maxFS) Note: This round maxFS was not increased and cwnd was increased. Round 2, the sender has 8000B to send in 8 packets (1000B), cwnd=14000 Received 8 ACKs (N=1000); maxFS=10000; cwnd_new += min(N, SMSS); cwnd = min(cwnd_new, 2*maxFS) Note: This round maxFS was not increased and cwnd was limited to 2*maxFS. Round 3, the sender has 4000B to send in 4 packets (1000B), cwnd=20000 Received 4 ACKs (N=1000); maxFS=10000; cwnd_new += min(N, SMSS); cwnd = min(cwnd_new, 2*maxFS) Note: This round maxFS was not increased and cwnd was not increased. Round 4, the sender has 20000B to send in 20 packets (1000B), cwnd=20000 Received 20 ACKs (N=1000); maxFS=20000; cwnd_new += N; cwnd = min(cwnd_new, 2*maxFS) Note: In this round, maxFS increased and therefore cwnd increased to 2*maxFS.

Change Log

• -00 was the first individual submission for feedback by CCWG.
• -01 includes editorial improvements
  • Removes application interaction with QUIC pacing, since pacing might be within the QUIC stack.
  • Adds explicit mention of DCCP/CCID2.
  • Adds this change log.
• -02 addresses comments from IETF-119
  • Discusses rate-based controls and pacing.
  • Trims the list of possible RFCs to update.
  • Some editorial fixes: "congestion control algorithm" instead of "mechanism" for consistency with RFC5033.bis; earlier definition of maxFS; explicit mention of RFCs to update in abstract.
• -03 addresses comments from IETF-120
  • Introduces a third rule, with MAY, that avoids having an unvalidated long-lived maxFS (using pipeACK from RFC 7661).
  • Changes "inc" to "limit" and adapts the wording of rule 2 to make it clearer (thanks to Neal Cardwell).
  • Appendix: updates ns-3 in line with the recent implementation.
  • Appendix: makes the RFC 9002 text clearer and shorter.
• draft-ietf-ccwg-ratelimited-increase-00
  • adds Mohit Tahiliani as a co-author
  • refines the "rule" text (shorter, clearer)
  • adds an example
• draft-ietf-ccwg-ratelimited-increase-01
  • Clarified what we mean with an RTT
  • rephrased example regarding initcwnd, citing RFCs 6928 and 9002
  • removed the too vague rule 1 and made rule 2 (now rule 1) a MUST
• draft-ietf-ccwg-ratelimited-increase-02
  • Improved the last sentence of section 3.1.2.
  • Removed a confusing and unnecessary sentence about pacing (as suggested at IETF-123).
• draft-ietf-ccwg-ratelimited-increase-03
  • The editors checked rule 2, and found that rule 1 was sufficient, and did not depend on the ordering of rules in newCWV (RFC7661), hence rule 2 was finally removed.
  • Cleaned language and improved text explaining how this complements RFC7661.
  • Checked/updated definitions.
  • Added an example with cwnd in bytes.
• draft-ietf-ccwg-ratelimited-increase-04
  • Repeated definitions from RFC 7661 instead of just pointing at the RFC
  • Made RFC 7661 informational
• draft-ietf-ccwg-ratelimited-increase-05
  • Rephrased references to RFC 7661
  • Nits
• draft-ietf-ccwg-ratelimited-increase-06
  • Changed some TCP-specific language so it applies to QUIC too
  • RFC 4341 added to list of updated RFCs in the abstract
  • Nits
• draft-ietf-ccwg-ratelimited-increase-07
  • Updated this list
  • Made RFC 4341 reference normative
  • Fixed Updates list to match boilerplate
  • Moved some of appendix B to the main text as a new section that (only) defines the RFC updates.
  • Updated description lines in appendix A
• draft-ietf-ccwg-ratelimited-increase-08
  • Addressed a comment from Eric Vyncke about the paragraph on rate-based cc. algs.
• draft-ietf-ccwg-ratelimited-increase-09
  • Corrections and additions proposed by Mahesh Jethanandani
• draft-ietf-ccwg-ratelimited-increase-10
  • Corrections to accounting of bytes in example in the Appendix

Acknowledgments The authors would like to thank Neal Cardwell and Martin Duke for suggesting improvements to this document. Thanks are due to the various IETF reviewers, and especially to Lars Eggert and Mahesh Jethanandani.