| Internet-Draft | Enhanced ECMP for AI Network | October 2026 |
| Cheng & Lin | Expires 1 May 2027 | [Page] |
In AI training scenarios, the current mainstream load balancing technology is per-flow ECMP. However, hash collision issues lead to imbalanced traffic distribution, adversely affecting application performance.¶
To address this problem, this document proposes an enhanced ECMP method that resolves load imbalance caused by hash collisions. The proposed solution effectively improves load balancing efficiency, reduces network congestion, and enhances overall network performance.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 1 May 2027.¶
Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
Currently, there are two granularities for network load balancing: per-flow ECMP and per-packet forwarding.¶
As illustrated in Figure 1, the per-flow ECMP method employs flow characteristic-based hashing (typically using the five-tuple) to distribute traffic across multiple ECMP paths. This approach works effectively in environments with numerous small flows and absence of elephant flows. Its primary advantage is the elimination of packet reordering issues.¶
However, this method presents limitations when dealing with either: a limited number of flows, or the presence of elephant flows.¶
In such cases, five-tuple-based hashing may lead to hash collisions, causing disproportionate mapping of oversized flows to the same path. This results in suboptimal load balancing performance.¶
ECMP
Flow 1 Hash 1 +------+
Flow 4 +--------- |if 1 |-----
Flow 7 +------+
| |-----
Flow 2 Hash 2 +------+
Flow 5 +-------- |if 2 |-----
Flow 8 +------+
| |-----
Flow 3 Hash 3 +------+
Flow 6 +------- |if 3 |-----
... +------+
|... |-----
+------+
The other approach is per-packet forwarding. This method applies hashing to each individual packet, distributing traffic across different ECMP paths, as illustrated in Figure 2. Theoretically, it achieves optimal load-balancing granularity. However, it introduces severe packet reordering within the same flow, necessitating additional mechanisms (e.g., reordering buffers or sequence tracking) to handle out-of-order delivery. This imposes higher demands on network infrastructure.¶
Packet 1 Hash 1 +------+
Packet 4 +--------- |if 1 |-----
Packet 7 +------+
| |-----
Packet 2 Hash 2 +------+
Packet 5 +-------- |if2 |-----
Packet 8 +------+
| |-----
Packet 3 Hash 3 +------+
Packet 6 +------- |if3 |-----
... +------+
|... |-----
+------+
In AI training scenarios, the current mainstream load balancing technology is per-flow ECMP. However, hash collision issues lead to imbalanced traffic distribution, adversely affecting application performance.¶
To address this problem, this document proposes an enhanced ECMP method that resolves load imbalance caused by hash collisions. The proposed solution effectively improves load balancing efficiency, reduces network congestion, and enhances overall network performance.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
+---------+ +---------+
| R11 | | R12 |
+-#--#-#--+ +#---#--#-+
| | | | | |
| | | | | |
| | +-----------------------------)-+ | |
| | | | | |
| | +---------------------------+ | | |
| | | | | |
| +---)----------+ +------------)-+ |
| | | | | |
+-#------#+ +-#-----#-+ +--#----#-+
| R21 | | R22 | | R23 |
+-#------#+ +-#------#+ +-#------#+
| | | | | |
+-#+ +-#+ +-#+ +-#+ +-#+ +-#+
|H1| |H2| |H3| |H4| |H5| |H6|
+--+ +--+ +--+ +--+ +--+ +--+
Due to the unique traffic patterns in AI training networks - characterized by a limited number of flows - achieving balanced load distribution becomes challenging. Traditional flow-based load balancing strategies often result in uneven traffic distribution, potentially leading to network congestion. While packet-based approaches can mitigate this imbalance to some degree, they introduce packet reordering issues as flow packets may traverse different paths, requiring additional network-level reordering mechanisms.¶
This document proposes two enhanced ECMP methods to address the load imbalance issue in AI training networks and improve the overall network performance.¶
Group the ingress interfaces for traffic, assign an ECMP number to the interfaces within the same group, and then perform ECMP hashing based on this ECMP number. This method is suitable when the forwarding traffic size for each ingress interface is roughly the same.¶
+-------+
ingress-interface 1 -> group-id 1 -> ECMP Index 1--> |if 1 |
+-------+
ingress-interface 2 -> group-id 1 -> ECMP Index 2--> |if 2 |
+-------+
ingress-interface 3 -> group-id 1 -> ECMP Index 3--> |if 3 |
+-------+
ingress-interface 4 -> group-id 1 -> ECMP Index 4--> |if 4 |
+-------+
ingress-interface 5 -> group-id 2 -> ECMP Index 1--> |if 1 |
+-------+
ingress-interface 6 -> group-id 2 -> ECMP Index 2--> |if 2 |
+-------+
ingress-interface 7 -> group-id 2 -> ECMP Index 3--> |if 3 |
+-------+
ingress-interface 8 -> group-id 2 -> ECMP Index 4--> |if 4 |
+-------+
As shown in Figure 4 above, the eight ingress interfaces are divided into two groups, with four interfaces in each group.¶
Within each group:¶
For traffic entering through the four ingress interfaces in Group 1: Different egress interfaces are selected for forwarding (four distinct paths).¶
Similarly, for traffic entering through Group 2's four ingress interfaces: Different egress interfaces are selected for forwarding (four distinct paths).¶
+-----+ +----+ Egress
|Spine|-------|Leaf|----Host
Ingress / +-----+ / \ +----+
+----+ / +-----+/ \+----+ Egress
Host ----|Leaf|/----|Spine|------|Leaf|----Host
+----+\ +-----+\ / +----+
\ +-----+ \ / +----+ Egress
\ |Spine|------|Leaf|----Host
+-----+ +----+
As shown in the figure, the source HOST connects to the source Leaf via an Ingress interface, while the destination HOST connects to the destination Leaf through an Egress interface. Multiple ECMP (Equal-Cost Multi-Path) links exist between the Leaf switches and multiple Spine devices.¶
To improve load-balancing distribution uniformity, the ECMP interfaces connecting to multiple Spine devices are grouped on the source Leaf. This grouping SHOULD be configured on the Leaf device. For example, if there are 128 equal-cost links between the source Leaf and Spine devices, they can be divided into groups of 4 interfaces each (Group 1: interfaces 1-4; Group 2: interfaces 5-8, etc.).¶
For traffic load balancing, flows MUST first be mapped to specific groups based on their location information (flows with the same location information are assigned to the same group), and then hash-based load balancing is performed within each group.¶
The location information used for group mapping can be either the source's Ingress interface or the destination's Egress interface.¶
By implementing fine-grained grouping of ECMP interfaces, this solution achieves more uniform traffic load distribution, thereby addressing current issues of imbalanced load sharing and flow collisions.¶
Mapping to <------Route(with egress group-id)
+-------+Group ID +-----+ +------+
|Ingress|-------------|Spine|-----|Egress|------egress
|Leaf | | | |Leaf |
+-------+ +-----+ +------+
Ecmp Group--
|--Sub Group-id 1 ---------(if-1,if-2, if-3,if-4)
|--Sub Group-id 2 ---------(if-5,if-6, if-7,if-8)
...
First, as shown in Figure 6, when the Egress Leaf advertises a route, it MUST carry the egress group index, which can be composed of the local device's router-id and group-id (see Section 4 for details).¶
On the Ingress Leaf, the ECMP egress interfaces toward the Spine devices MUST be grouped (as illustrated in Figure 7).¶
When the Ingress Leaf receives a route, it MUST extract the remote egress group index carried in the route. It then maps this remote egress group index to a local ECMP subgroup index, effectively directing traffic to the corresponding subset of interfaces for forwarding.¶
This ensures that flows destined for different remote addresses are load-balanced across different ECMP subgroups, improving distribution granularity. Refer to Figure 8 for details.¶
+=========+=============+=================+======================+ | Dest |Remote | Local Index | ECMP interfaces | | | Attribute | | | +=========+=============+=================+======================+ | route-1 |Egress Group |Local ECMP |(if-1,if-2, if-3,if-4)| | |Index 1 |Sub Group Index 1| | +=========+=============+=================+======================+ | route-2 |Egress Group |Local ECMP |(if-5,if-6, if-7,if-8)| | |Index 2 |Sub Group Index 2| | +=========+=============+=================+======================+ | ... | ... | ... | ... | +=========+=============+=================+======================+
To support the enhanced ECMP mechanism based on egress grouping described in Section 3.2, the route advertised by the Egress Leaf MUST carry an egress group index, which is composed of the local device's router-id and group-id. The encoding of this information is defined as a new BGP Extended Community (BGP_EXT-COMM-ECMP-GROUPID) in [I-D.cheng-idr-ecmp-groupid].¶
On the Ingress Leaf, when a route carrying this extended community is received, the remote egress group index is mapped to a local ECMP subgroup index as described in Section 3.2, directing traffic to the corresponding subset of ECMP interfaces for forwarding.¶
This document has no IANA actions. The registration request for BGP_EXT-COMM-ECMP-GROUPID is defined in [I-D.cheng-idr-ecmp-groupid].¶