]> Cisco

djsmith@cisco.com

Cisco

sekrier@cisco.com

CoreWeave

sgaw@coreweave.com

AT&T

im8327@att.com

HPE Juniper Networking

krzysztof.szarkowicz@hpe.com

Routing Inter-Domain Routing BGP-4 BGP routing RR route reflector In BGP networks, Route Reflectors (RRs) simplify the provisioning of the Internal BGP (IBGP) control plane and mitigate the scaling challenges of maintaining a full mesh of IBGP sessions among border routers. Because of these operational benefits, RRs are widely deployed in modern networks utilizing BGP. This document describes common RR design options, their associated trade-offs, and deployment best practices based on industry operational experience. Operators should consider these suggestions when designing and operating BGP networks because BGP control plane stability is essential to overall network stability. Status information for this document may be found at . Discussion of this document takes place on the Inter-Domain Routing Working Group mailing list (), which is archived at . Subscribe at . Source for this draft and an issue tracker can be found at .

Introduction In BGP networks , Route Reflectors (RRs) simplify the provisioning of the Internal BGP (IBGP) control plane and mitigate the scaling challenges of maintaining a full mesh of IBGP sessions among border routers. Because of these operational benefits, RRs are widely deployed in modern networks utilizing BGP. This document describes common RR design options, their associated trade-offs, and deployment best practices based on industry operational experience. Operators should consider these suggestions when designing and operating BGP networks because BGP control plane stability is essential to overall network stability.

Performance and Scale Considerations BGP RR performance and scale depends on a variety of factors including but not limited to: Available system resources: CPU performance, memory speed and quantity available. Number of BGP sessions, associated timers and update-groups. BGP address families configured , . Number of routes and associated paths for each BGP address family enabled as well as route/path churn. Whether the RR operates in the BGP control plane only or also in the IP forwarding plane. BGP capabilities enabled including but not limited to, for example, inbound/outbound route policies (simple versus complex), Extended Message support , Add-Paths , Graceful Restart , Optimal Route Reflection , Route Target Constraint , automatic detection and isolation of slow peers, and RIB/FIB filtering. Transport layer optimizations including but not limited to Maximum Segment Size (MSS), window size, Selective Acknowledgement (SACK) , and TCP Path MTU discovery , as well as TCP-AO session authentication . Network latency to and from clients and non-client peers. BGP implementation and performance of the network operating system itself. Operators should consider all these factors as part of their RR deployment. Further, since these factors vary widely among network operators, specific BGP RR performance and scale guidelines are outside the scope of this document. Operational best practices related to the management, monitoring and maintenance of RRs are also out of scope.

Cluster Design Considerations The placement of an RR within the Interior Gateway Protocol (IGP) topology is an important design consideration. The resulting network latency can affect TCP performance and, consequently, BGP session performance. In addition, when the IGP metric to the BGP NEXT_HOP is considered during best path selection, an RR’s location within the IGP topology can influence the path it selects. RRs have therefore traditionally been deployed in close IGP proximity to their clients. This approach reduces latency and helps ensure that the RR and its clients have similar IGP metrics to each NEXT_HOP. Although latency remains an important consideration, mechanisms such as BGP Add-Paths and BGP Optimal Route Reflection (ORR) can mitigate path hiding and improve the paths advertised to RR clients. See for additional details.

Redundancy BGP RR redundancy enhances network stability by eliminating single points of failure and protecting against RR-level outages. Note, however, that excessive RR redundancy can be detrimental. This is because the more RRs a client (e.g., border router, PE router) peers with, the more IBGP sessions, updates, and paths the client must carry and process in its BGP table. If we consider two RR clusters (1 and 2), as shown in Figure 1, the RR clients (C-2A, C-2B, C-2C) in cluster 2 must process 50% more sessions, updates, and paths compared to cluster 1 clients. For example, if each of the RRs advertised the same 1.5 million (1.5M) routes to the clients, then the BGP tables on cluster 1 clients would carry 3M paths, while the BGP tables on cluster 2 clients would carry 4.5M paths. That is, the incremental overhead of BGP paths, sessions, and updates increases linearly with every redundant RR in the cluster.

Further, when a route is no longer available and needs to be withdrawn, an RR client cannot withdraw a route learned from its RRs until it receives a withdrawal from each of the RRs. As a result, increasing the number of redundant RRs could also lead to slower BGP convergence. Again, the incremental overhead of BGP sessions, updates, and paths increases linearly with every redundant RR in the cluster. Hence, for optimal redundancy and scaling, it is suggested that clients peer with at least two redundant RRs, but no more than four total - typically structured as two clusters, each containing a pair of redundant RRs. Also, depending on the network services and scale, different RR clusters may be suggested for different BGP address families. See below for more discussion on this topic. Consequently, this general guideline — limiting clients to peering with at most four RRs — applies per BGP address family.

CLUSTER_ID Configuration Redundant RRs within a cluster are not required to share the same CLUSTER_ID. When redundant RRs within a cluster use different CLUSTER_IDs, they treat each other as non-client IBGP peers, processing and accepting each other’s updates. Conversely, if they share the same CLUSTER_ID, they continue to exchange routes but will ignore reflected routes from one another to avoid routing information loops, as defined in . Consequently, there is no benefit to IBGP peering between redundant RRs sharing a CLUSTER_ID, unless either of the RRs have EBGP sessions or redistribute local IGP, static, or connected routes into BGP. This is because the CLUSTER_ID is not prepended to such non-IBGP learned routes as these are not reflected routes per . As a result, these non-reflected routes would be exchanged between redundant RRs configured with the same CLUSTER_ID but not discarded. Although using a shared CLUSTER_ID reduces resource consumption and update processing on redundant RRs within a cluster, it can potentially limit resiliency in the event of simultaneous BGP session failures involving multiple RRs and clients within the cluster. Therefore, it is suggested that redundant RRs within a cluster peer with each other using unique CLUSTER_IDs.

Cluster Sizing In addition to managing the number of redundant RRs per cluster, it's also important to maintain moderate RR cluster sizes (i.e., the number of client sessions per cluster) to limit the blast radius in the event of an RR cluster failure. Accordingly, BGP networks often require multiple RR clusters - not only for fault containment but also for horizontal scaling (sessions, path selection), geo-redundancy, and improved BGP best path selection.

Multi-Cluster Designs While the BGP control plane is essential to network stability, the RR design is essential to BGP control plane stability. In general, there are three (3) common multi-cluster RR designs: Full mesh Multi-tier Multi-plane Each of these designs has its own set of trade-offs which are described below. These are also not the only design options, as hybrid deployments combining these approaches are also possible.

Full Mesh A full mesh, multi-cluster RR design employs a full IBGP peering mesh between all RRs in all clusters as shown in Figure 2. This is the most common and widely deployed multi-cluster RR design, as it effectively balances scalability and fault isolation while optimizing the distribution of BGP routes and paths within the network. It achieves this by minimizing the number of IBGP hops that updates traverse, which leads to improved BGP performance at scale.

However, for massive-scale BGP deployments with many RR clusters, the IBGP full mesh between RRs becomes a scalability bottleneck. This is due to high RR session counts and the provisioning complexity involved when adding new RR clusters to the topology. For example, introducing a new RR requires configuring new IBGP sessions on all existing RRs, which increases overhead and BGP session density. This N^2 scaling challenge can also degrade BGP update processing and network convergence, much like a full IBGP mesh between a large number of border routers without RRs.

Multi-Tier A multi-tier, multi-cluster RR design is a hierarchical architecture involving multiple tiers of RRs, as illustrated in Figure 3. This design is intended for massively large-scale BGP deployments with many RR clusters.

At the lowest level, border routers remain clients of their assigned RR clusters, similar to the full mesh multi-cluster design described above. In this multi-tier model, the RRs peering with border routers are referred to as Tier 2 RRs. Instead of a full IBGP mesh between them, Tier 2 RRs are deployed as clients of Tier 1 RRs, which are themselves fully meshed. While this design reduces IBGP session density on RRs and simplifies their provisioning, it may adversely affect BGP convergence time given the incremental RR hops within the IBGP control plane. Furthermore, a failure of a Tier 1 RR cluster typically results in a large blast radius.

Multi-Plane A multi-plane, multi-cluster RR design comprises multiple independent RR planes (e.g., RED and BLUE). Within each plane, clusters can be fully meshed or multi-tier as described above. Note that while this document uses a two-plane model for illustration in Figure 4, the architecture is not limited to two planes. As shown in Figure 4, RRs within the RED plane (RR-R1 through RR-R6) are fully meshed as are RRs in the BLUE plane (RR-B1 through RR-B6). In a two-plane design, RR clients (typically border routers) peer with the redundant RRs of one specific cluster in each plane, resulting in four IBGP sessions per client. Multi-plane RR designs can be used in different ways, and each has a different failure behavior. For example, if multi-plane is used for increased redundancy, each plane should carry enough routing information to maintain service if another plane fails. In this deployment model, the failure of one plane should reduce redundancy but not remove required IP reachability. Further, a multi-plane design used for redundancy also facilitates software or hardware upgrades. Because the planes operate independently and provide mutual redundancy, they can be maintained or upgraded separately without disrupting IP reachability. Alternatively, if planes are used for horizontal scaling, different routes or paths may be distributed across different planes. In this deployment model, the failure of one plane may remove the routes or paths carried only by that plane.

Address Family and Control Plane Segmentation BGP supports multiple types of Network Layer Reachability Information (NLRI), identified by Address Family Identifier (AFI) and Subsequent Address Family Identifier (SAFI) values . An RR plane can carry multiple AFI/SAFI combinations over a common set of IBGP sessions, or different sets of AFI/SAFIs can be assigned to separate RR planes. The decision to separate address families depends on their scale, update characteristics, policies, client populations, availability requirements, and implementation-specific resource constraints. Separation can reduce contention for processing and memory resources while limiting the impact of failures or abnormal update activity. Resource and node-level isolation are most effective when the address families are assigned to separate RR processes or nodes. Separate sessions terminating on the same RR process provide session-level isolation but continue to share the underlying processing and memory resources. Segmentation also introduces operational costs. Each additional RR plane can require more RR instances, client sessions, loopback addresses, configuration, monitoring, and maintenance procedures. Excessive segmentation can therefore increase complexity and introduce additional failure modes. Address families with similar operational roles, policies, client populations, and availability requirements can generally share an RR plane unless scale, implementation constraints, or isolation requirements justify separation. One common deployment model groups address families into Internet, VPN service, and transport control planes: An Internet control plane carries global IPv4 and IPv6 reachability. This can include IPv4 unicast, IPv6 unicast, and 6PE routes when 6PE is used to provide global IPv6 reachability. A VPN service control plane carries routes used to provide L3VPN, L2VPN, and multicast VPN services. This can include VPN-IPv4, VPN-IPv6, MVPN, EVPN, and other L2VPN routes. These address families commonly use Route Distinguishers, Route Targets, and Route Target Constraints , and often have similar policy and operational requirements. Consequently, L3VPN, EVPN and L2VPN address families can generally share a common VPN service RR plane. Separate RR planes can be used when differences in scale, update characteristics, administrative boundaries, client populations, or fault-isolation requirements justify the additional complexity. A transport control plane carries routes used to establish or select transport connectivity. This can include BGP IPv4 labeled-unicast (BGP-LU) and BGP Color-Aware Routing (BGP CAR) . Because BGP-LU and BGP CAR have a common operational role and generally serve similar client populations, they can share a common transport RR plane. This avoids unnecessary RR infrastructure and IBGP sessions while allowing the associated transport address families to be operated together. Separate planes can be used when their scale, update characteristics, client populations, administrative boundaries, or isolation requirements differ significantly. These groupings describe operational roles rather than protocol requirements. For example, 6PE can be assigned to the Internet RR plane when it provides global IPv6 reachability, even though it uses IPv6 labeled-unicast routes. Similarly, BGP-LU and BGP CAR can be assigned to the transport RR plane because their function is to distribute transport reachability. Operators should consider the address families carried by each plane and the expected effect of a plane failure. Internet and VPN address families can also use different policies and BGP mechanisms. Add-Paths (primary/backup, n-way, all) and Optimal Route Reflection can be used to advertise additional or more appropriate paths to RR clients. These mechanisms are not limited to Internet address families and can also be applied to VPN address families when supported and operationally appropriate. VPN address families commonly use Route Target Constraints to limit the distribution of routes to client and non-client peers that have expressed interest in the associated Route Targets. Route Distinguishers allow otherwise identical VPN prefixes to be represented as distinct NLRI, but do not by themselves guarantee path diversity. In addition, RRs configured for BGP-LU or BGP CAR and deployed in the MPLS forwarding plane as part of a seamless MPLS architecture require next-hop-self configuration and MPLS label allocation. BGP Link-State (BGP-LS) distributes link-state and traffic-engineering information to consumers such as controllers and path computation applications. Although BGP-LS NLRI can be propagated through RRs, its purpose, update characteristics, and consumer population differ from those of Internet, VPN, and transport reachability routes. A dedicated BGP-LS RR plane can provide additional resource and operational isolation when BGP-LS carries a large volume of information, experiences substantial update activity, or serves a distinct set of consumers. In smaller deployments, BGP-LS can share infrastructure with another RR plane when the resulting scale and failure coupling are acceptable. The appropriate degree of segmentation is deployment-specific. A practical model uses separate RR planes for Internet reachability, VPN services, and transport reachability, with a dedicated BGP-LS plane where its scale or consumer population warrants additional isolation. Under this model, L3VPN and L2VPN address families share the VPN service RR plane, while BGP-LU and BGP CAR share the transport RR plane. Further separation is based on measured scale, update behavior, implementation limits, administrative boundaries, or explicit isolation requirements rather than solely on AFI/SAFI distinctions. It is also worth noting that separating address families into Internet, VPN service, and transport control planes increases the number of RR loopback addresses carried in the link-state IGP.

Policy and Update-Group Considerations specifies the handling of path attributes when an RR reflects a route. In general, an RR does not modify the NEXT_HOP, AS_PATH, LOCAL_PREF, or MED attributes solely as a consequence of route reflection. An RR also preserves an existing ORIGINATOR_ID and prepends its local CLUSTER_ID to the CLUSTER_LIST. If no CLUSTER_LIST is present, the RR creates one containing its local CLUSTER_ID. The ORIGINATOR_ID and CLUSTER_LIST provide route-reflection loop detection and therefore require particular care. Operators can apply policy at an RR when required by the network design. Examples include setting or removing communities , controlling route distribution, changing route preference, selecting which paths are advertised, and modifying NEXT_HOP when the RR has an appropriate forwarding role. Such policies are not inherently incompatible with route reflection, but their effects can extend to many clients and can change path selection, route visibility, or forwarding behavior across a large portion of the network. An operator modifying attributes that affect best path selection should evaluate the policy across the complete reflection topology rather than only on an individual session. Inconsistent policies among redundant RRs can cause clients to receive different paths and can complicate convergence and troubleshooting. Policies should therefore be deterministic, under both normal and failure conditions, and applied consistently across redundant RRs unless intentional differences are part of the design. An RR should not remove or rewrite ORIGINATOR_ID or CLUSTER_LIST information in a way that defeats the loop-detection procedures defined by . Policies affecting AS_PATH or other attributes used for loop prevention also require careful validation. When an RR must modify NEXT_HOP because it participates in the forwarding plane, operators should follow the procedures and deployment considerations in . Inter-AS Option C, described in Section 10 of , is one example of a design in which an RR can have both route-reflection and forwarding responsibilities. Policy design also affects RR scale. BGP implementations commonly group peers that have equivalent outbound advertisements so that an update can be generated once and replicated to multiple peers. These groups are often referred to as update-groups, although their exact behavior is implementation-specific. Per-peer policies, different advertised capabilities, and other session-specific behavior can divide clients into additional update-groups. This can increase update generation, memory consumption, CPU utilization, and convergence time. Operators should therefore use common outbound policies across clients where operational requirements permit. Policy variation should normally be applied at the largest practical grouping, such as by service, region, role, or client class, rather than through unique policy for each peer. Where per-client policy is required, operators should account for its effect on update-group formation and RR capacity. Inbound and outbound filtering can provide important operational benefits and should not be avoided solely to reduce processing overhead. For example, filtering can enforce route-distribution boundaries, prevent unintended route propagation, attach policy metadata, and protect the network from invalid or excessive advertisements. The objective is to avoid unnecessary policy complexity while retaining controls required for routing correctness, security, and service operation. Centralized policy is an important benefit of an RR architecture because it allows operators to influence route distribution and path selection from a limited number of locations. This benefit should be balanced against the size of the policy configuration, the number of resulting update-groups, the expected update rate, and the convergence objectives of the deployment. Operators should validate this balance through implementation-specific scale testing and monitor update-group counts, policy-processing load, queue depth, memory utilization, and update-generation time where those measurements are available.

Slow Peer Considerations It is common for RRs to dynamically establish BGP update-groups across IBGP peers that share identical outbound routing policies , including instances where no outbound policies are applied. This improves RR performance, as an RR formats a single update advertised to all peers within an update-group rather than advertising unique updates for each individual peer. This also enhances overall BGP control plane performance. However, a slow peer within an update-group that cannot keep up with the rate at which an RR generates BGP update messages can adversely affect other members of the group. A single slow client can force a RR to throttle, delaying updates to other peers in the same group. This degrades BGP convergence and may increase traffic disruption during BGP convergence events. It is suggested to isolate known slow clients, such as legacy routers with limited CPU and memory resources, into their own IBGP update-group. This isolation can be performed manually via static configuration or, if supported by the RR software, dynamically. Dynamic isolation involves the RR automatically detecting and moving slow peers out of an update-group so as not to adversely impact other members. This dynamic behavior can also return previous slow peers back into an update-group after their BGP processing is caught up. However, for permanently known slow peers it is suggested to statically assign them to dedicated update-groups. This approach helps to prevent any adverse impact on other peers and minimizes the RR churn associated with the automatic detection, isolation, and reintegration of slow peers. The procedure to identify slow peers - a process that can be automated - involves monitoring the BGP table version and BGP output queue of the peers in each update-group. This information should then be used to verify whether the BGP table version of the peers ever catches up with the main BGP table version or whether it is always behind. Similarly, check whether any peers have a very high output queue value for a long duration. If so, it is likely that these are slow peers and should be isolated.

Path Selection Considerations By default, a BGP speaker selects a single best path for each destination and advertises only that path to a peer, subject to policy. When acting as an RR, the speaker generally performs this selection from its own perspective. Paths not selected by the RR are hidden from its clients unless another mechanism causes those paths to be advertised. Path hiding can occur when the path selected by an RR differs from the path that a client would have selected had the client received all available paths. This is particularly relevant when the IGP metric to the BGP next hop is considered during best path selection, because the RR ordinarily evaluates that metric from its own location rather than from the location of each client. Depending on the topology and routing policies, path hiding can result in suboptimal routing, reduce the paths available during a failure, and contribute to the route oscillations described in . Several mechanisms can reduce path hiding. The appropriate mechanism depends on the address family, the required number of paths, the topology, and the scaling characteristics of the deployment.

Route Distinguisher Assignment In BGP L3VPN deployments , operators commonly assign distinct Route Distinguishers (RDs) to routes originated by different PEs (Provider Edge routers). EVPN deployments similarly use RDs to distinguish otherwise identical EVPN NLRI originated from different PEs or EVPN instances. Because the RD is part of the NLRI, routes with different RDs are treated as distinct destinations by BGP and are subject to separate best path decisions. Using distinct RDs can preserve multiple VPN or EVPN routes through the RR system without requiring Add-Paths for those distinct NLRI. After import, a receiving PE can evaluate the eligible routes for the corresponding service destination. This can improve route visibility and preserve useful path diversity. A unique RD does not, by itself, guarantee forwarding-path diversity, optimal routing, or load balancing. The resulting behavior also depends on route import policy, NEXT_HOP reachability, path attributes, the client’s best path and multi-path procedures, and the underlying topology. Operators should therefore assign RDs according to the service design and the path diversity that needs to be preserved. RD-based path preservation does not apply to routes in the global routing table carried using IPv4 unicast, IPv6 unicast, or 6PE, because those routes do not use an RD as part of their NLRI.

Add-Paths Historically, some networks used policy to cause redundant RRs to select and advertise different paths. A client connected to multiple RRs could consequently receive more than one candidate path. This approach depends on deliberate policy differences between RRs and can increase operational complexity. BGP Add-Paths provides a protocol mechanism for advertising multiple paths for the same NLRI over a BGP session. Depending on the path-selection and advertisement mode implemented and configured, an RR can advertise a best path and one or more additional paths, such as backup, diverse, n-way, or all eligible paths. This reduces the need to rely exclusively on policy differences between redundant RRs to expose multiple paths. When an RR advertises suitable candidate paths, a client can perform best path selection from its own perspective, including the use of its own IGP metrics where applicable. The benefit depends on which paths the RR selects for advertisement. For example, advertising only the RR’s best path and a backup path does not necessarily expose the path that is best from every client’s location. Add-Paths can improve path visibility, convergence, and resilience, and can be used as part of a solution for the route oscillation conditions discussed in and . Its use does not, however, guarantee optimal routing or eliminate every form of path hiding or route oscillation. Add-Paths can also increase the number of paths retained, processed, and advertised by BGP speakers. This can increase memory consumption, update generation, policy processing, and convergence workload on both RRs and clients. Operators should select an advertisement mode that provides the required path diversity without distributing unnecessary paths, then validate its impact against the deployment’s scale and convergence objectives.

Optimal Route Reflection Optimal Route Reflection (ORR) addresses cases in which the path selected from the RR’s location is not the path that would be preferred from a client’s location. ORR allows an RR to select and advertise a path using an IGP perspective associated with a root client or group of clients rather than the RR’s own IGP location. ORR can reduce IGP metric-based path hiding and allows RRs to be placed independently of the clients for which they select paths. It does not necessarily produce a globally optimal forwarding path, because selection remains subject to the paths available to the RR, the configured ORR perspective, BGP policy, path attributes, and implementation behavior. ORR and Add-Paths are complementary. ORR selects a path based on the root client’s or client group’s location, rather than the RR’s location. In contrast, Add-Paths advertises multiple candidate paths, allowing the client to perform its own selection. A deployment can use either mechanism or both, depending on its path selection and scaling requirements. ORR requires sufficient IGP topology information to compute path metrics from the perspective of root clients or client groups. An RR can obtain this information by participating in the relevant IGP or by using BGP-LS . Maintaining multiple ORR perspectives and performing path selection for different root clients or client groups increases RR processing and state. Operators should minimize the number of distinct ORR perspectives by grouping clients with equivalent topologies, then evaluate the resulting impact on CPU, memory, and convergence.

Protocol NEXT_HOP Resolution The need for Protocol NEXT_HOP (PNH) resolution depends on the AFI/SAFI and the paths being processed. For address families carrying reachability information, an RR needs to determine whether a BGP NEXT_HOP is resolvable and, when applicable, obtain the IGP metric to be used during best path selection. The exact PNH behavior depends on the specific address family. Some address families do not use the PNH in the same manner as unicast reachability routes. For example, BGP-LS distributes topology information rather than forwarding reachability, and Route Target Constraint routes control VPN route distribution. FlowSpec NLRI describes traffic-matching and action rules rather than conventional destination reachability. However, FlowSpec validation and route acceptance can still depend on information from the unicast routing system. Implementations can provide mechanisms to avoid or reduce PNH resolution for address families or routes for which it is not required. Where supported, operators can leverage these mechanisms to reduce processing overhead, provided they first verify the protocol procedures, validation rules, policy dependencies, and implementation details for the target AFI/SAFI.

Deployment Guidance No single path selection mechanism is appropriate for every address family or deployment. Operators should consider: Distinct RDs where multiple VPN or EVPN routes need to remain distinct NLRI. Add-Paths where clients benefit from receiving multiple candidate paths. ORR when path advertisement must be constrained and path selection must align with client-specific or client-group topological proximity, rather than the RR’s local perspective. Multiple of the above mechanisms where their benefits are complementary. The selection should account for route scale, required path diversity, convergence objectives, RR and client capacity, topology, and operational complexity. Operators should also consider failure scenarios, as a mechanism that provides optimal paths during steady state may fail to offer sufficient alternatives during convergence.

Route Target Constraint Considerations BGP Route Target Constraint (RTC) can significantly reduce the volume of VPN updates an RR advertises to edge routers (PEs). With RTC, it is suggested that RRs advertise an RT-filter default to clients, while clients send specific RT-filters to the RR, rather than the reverse. This is particularly important in multi-tier RR designs: upper-tier RRs should advertise an RT-filter default to lower-tier RRs, while lower-tier RRs send specific RT-filters to the upper-tier RRs. Failure to do so may cause clients and lower-tier RRs to receive all VPN routes, potentially exceeding their scale capacity. Further, RTC is most effective when activated end-to-end to build precise outbound route filters, including those for single-domain and multi-domain network-based VPNs. This also includes multi-tier, multi-cluster RR designs so that Tier 1 RRs only send the required VPN prefixes to Tier 2 RRs, drastically reducing BGP VPN processing and overhead. Operators should also be aware of known issues associated with RTC in specific scenarios. These are documented in and , both of which also propose solutions.

EVPN Route Target Considerations To optimize NLRI distribution via RR design, special attention is suggested for EVPN. in Section 7.6 defines the ES-Import Route Target (ES-Import-RT) extended community. The ES-Import-RT, together with RTC, enables optimized route distribution of Ethernet Segment (ES) routes. Namely, from an EVPN architectural point of view, it is sufficient that only PEs connected to the same multi-homed ES receive the ES route specific to a given multi-homed ES. All other PEs — those not connected to a given multi-homed ES — do not require visibility of that specific ES route. To achieve the desired behavior, the value of the ES-Import-RT extended community is derived automatically from the Ethernet Segment Identifier (ESI) by encoding the high-order 6-octet portion of the 9-octet ESI value. For example, as illustrated in Figure 5, the octets depicted as 'XX' are used as the value of the ES-Import-RT.

When RTC is enabled, the RT-filter advertised by a PE (which is an RR client) is also automatically updated to signal interest to the RR in receiving NLRIs (here, ES routes) with a specific ES-Import-RT (here, with the value XX:XX:XX:XX:XX:XX). In large-scale deployments with a large number of multi-homed Ethernet segments, this can significantly reduce the number of ES routes that are sent to RR clients. In order, however, to take advantage of this optimized distribution of ES routes, ESI values must be assigned in such a way that they are differentiated in the high-order 6-octet portion of the 9-octet ESI value. While this happens automatically for ESI types 1, 2, and 3, care must be taken with ESI type 0 (manual allocation). Figure 6 depicts an ESI type 0 value allocation scheme that does not result in an optimized distribution of ES routes through RRs.

In this ESI Type 0 value allocation scheme, the high-order 6-octet portion of the 9-octet ESI value is the same for all ESIs (00:00:00:00:00:00). Therefore, all PEs (all RR clients) update the RT-filters they advertise to the RR with an RT equal to 00:00:00:00:00:00, resulting in all PEs (RR clients) receiving ES routes for all Ethernet segments. Figure 7 depicts a different allocation scheme for ESI Type 0 values. In this allocation scheme, the values are differentiated in the high-order 6-octet portion of the 9-octet ESI value.

Such an allocation scheme results in different ES-Import-RTs (and, as a consequence, different RT-filters) for each group of PEs connected to different Ethernet segments. Therefore, the ES route pertaining to ESI-1, based on the RT-filters received by the RR from PEs connected to the multi-homed Ethernet segment with ESI-1, is sent by the RR only to PEs (RR clients) connected to that particular Ethernet segment. It is suggested that operators consider adopting this allocation scheme to optimize ES route distribution, thereby reducing both the volume of routes propagated to RR clients and the associated control plane overhead.

Transport Optimizations

TCP Path MTU Discovery To improve BGP control plane performance, it is suggested to enable TCP Path MTU Discovery (PMTUD) on both RRs and clients; this allows TCP to select an appropriate Maximum Segment Size (MSS) for each BGP session. In this way, BGP sessions use the largest packet size that does not require IP fragmentation anywhere along the path to client and non-client IBGP peers.

Ethernet MTU The Ethernet maximum transmission unit (MTU) is the size of the largest frame, minus the 4-byte frame check sequence (FCS), that can be transmitted on a Ethernet network. Every physical network along the path to the destination of a packet can have a different MTU. The default Ethernet MTU (Maxiumum Transmission Unit) value is 1514 bytes for standard Ethernet frames and 1518 bytes for 802.1Q tagged frames. These numbers exclude the 4-byte frame check sequence (FCS). When the MTU size is larger, more data can be sent in fewer number of packets. As each packet contains a header, a small number of packets means less header data to be transmitted. Thus, a larger size MTU helps in transmitting more actual data faster and efficiently. To ensure the RR is not the limiting factor related to PMTUD and BGP update packet sizes, it is suggested to enable Jumbo frame support on RR interfaces with the largest packet size value available.

Miscellaneous Considerations

BGP Router ID To prevent IBGP session instability caused by changes in the BGP router ID when a physical interface's state changes, it is suggested to statically configure the BGP router ID using the IP address of a virtual loopback interface. This approach is preferred because loopback interfaces are more stable than physical interfaces, which may go down or change state, causing the router ID to change and potentially leading to BGP session resets. This IBGP best practice applies to both RRs and clients.

BGP RIB/FIB Filtering For RRs operating exclusively in the control plane, configuring RIB/FIB filtering policies to suppress BGP route installation is suggested. Restricting these tables on non-forwarding devices: Minimizes RIB and FIB resource consumption. Eliminates unnecessary RIB and FIB processing overhead. Improves an RR's overall efficiency and scalability.

BGP Error Handling specified critical updates to BGP error handling procedures. Ensure these procedures are enabled on RRs (and all other BGP speakers) if they are not already active by default.

Diffserv QoS Considerations Because BGP control plane stability is fundamental to overall network stability, it is suggested that BGP packets be prioritized using Diffserv QoS policies. This approach helps to prevent BGP packet loss and minimize queuing delays during congestion events, thereby ensuring session stability and avoiding the traffic disruption that occurs if BGP sessions fail.

Form Factor Considerations Traditionally, RRs were deployed using physical routing devices. With the advent of Network Function Virtualization (NFV), RRs that operate exclusively in the control plane (i.e., control-plane-only) are now commonly deployed as Virtual Network Functions (VNFs) or virtual RRs. Virtual RRs offer superior scalability and convergence performance compared to physical RRs by providing substantially more CPU performance, memory speed and quantity available for BGP protocol processing. Consequently, deploying control-plane-only RRs as virtual RRs is suggested. In contrast, RRs that participate in the IP forwarding plane are typically deployed as physical RRs, unless a virtual RR can satisfy the required forwarding capacity. It is also worth noting that virtual RRs rely on independent compute, hypervisor, storage, and virtual networking infrastructure; however, vendors offer integrated appliances that bundle these components. If a virtual implementation is not feasible for operational or administrative reasons, a physical RR remains a viable alternative; however, the operator must account for the difference in control plane scale.

BGP Non-Stop Routing/Forwarding A virtual RR is deployed on a compute server which has no concept of redundant route processors. A physical RR can be configured with redundant route processors and leverage this redundancy to enable BGP Non-Stop Routing (NSR). NSR preserves BGP session and control plane state during a route processor failover, ensuring that IBGP clients and non-client peers do not detect the switchover and eliminating the need for session resets, BGP Graceful Restart, or its helper mode. While physical RRs with redundant route processors uniquely support NSR, they generally offer significantly lower BGP scaling compared to virtual RRs as described earlier. Consequently, deploying multiple redundant virtual RRs is suggested for networks with high BGP control plane scale.

BGP Graceful Restart enables a BGP router to gracefully restart its BGP routing process when necessary (e.g., due to a memory leak) and preserve its BGP forwarding state to avoid traffic loss. Redundant route processors are not required, however, sessions with BGP peers must be enabled for the BGP graceful restart capability. Otherwise, the forwarding state of BGP route paths learned via a session without graceful restart enabled is not preserved during a graceful restart of the local BGP routing process. Control-plane-only RRs are not in the traffic forwarding plane, so they do not benefit from having their sessions enabled for graceful restart. Similarly, RR clients typically have multiple copies of reflected routes from multiple redundant RRs and, as a result, incur no traffic loss when their local BGP routing process restarts. Consequently, there is no tangible benefit to using graceful restart on sessions between RRs and their clients. Graceful restart is suggested for BGP routers participating in the traffic forwarding plane that lack BGP NSR and do not have access to redundant copies of BGP route paths from multiple peers.

Multi-hop BFD for IBGP Multi-hop BFD is not suggested for IBGP sessions between control-plane-only RRs and their clients. Deploying BFD in this context exposes the BGP control plane to unnecessary instability, as it may trigger session resets due to minor, transient connectivity issues that TCP would otherwise handle gracefully. A multi-hop BFD-triggered reset on a session carrying millions of routes, for example, creates significant churn, even when the underlying network topology remains stable. Furthermore, using multi-hop BFD to accelerate failure detection within the BGP control plane is counterproductive; since clients typically receive redundant path information from multiple RRs, the loss of a single RR session does not impact the client's traffic forwarding plane.

Security Considerations This document introduces no new security considerations to the BGP protocol itself. It describes deployment considerations and best practices that BGP network operators should consider or adopt when deploying RRs to maximize BGP control plane performance, scalability and stability. While general BGP operational security guidance is provided in , several RR considerations with security relevance are described below.

BGP Session Authentication BGP session cryptographic authentication is suggested to ensure that BGP updates are only accepted from trusted, verified peers, which helps to reduce the risk of unauthorized route injection and session hijacking. MD5 is the legacy method for BGP authentication, as it has been deprecated for some time now, given it is considered cryptographically weak by modern standards. TCP-AO is the modern standard and suggested method for BGP session security as it provides stronger cryptographic algorithms and improved security features.

BGP Session Prefix Limits Configuring BGP session prefix limits is an important measure to protect a network from an EBGP peer unexpectedly advertising an excessive number of routes, such as the full Internet routing table. BGP session prefix limits enforce a maximum receive prefix count from a neighbor for each address-family, helping safeguard a network from out-of-resource (OOR) conditions which could lead to traffic disruption. In the event a session prefix limit is hit, BGP implementations typically provide several actions in response: Bring down session Restart session Discard extra paths SYSLOG warning Beyond generating a SYSLOG warning, it is suggested to implement one of the listed actions for EBGP sessions based on your specific network requirements. BGP session prefix limits can be configured on IBGP sessions as well; however, if configured for IBGP, it is suggested to use only a SYSLOG warning without taking further action. This approach helps monitor prefix counts while avoiding IBGP session resets or disruptions, which could result in a more significant detrimental impact.

Control Plane Protection It is common for modern IP routers to support control plane protection , including but not limited to control plane policing. Typically enabled by default on modern IP routers, this capability helps protect a router from excessive control plane traffic by classifying and policing different packet types, thereby preventing potential denial-of-service conditions. With regard to RRs, it is important for large-scale deployments to ensure that BGP control plane policer rates are optimally configured to prevent BGP packet drops, which could adversely affect both BGP stability and overall network stability.

IANA Considerations This document has no IANA actions.

This document discusses the Border Gateway Protocol (BGP), which is an inter-Autonomous System routing protocol. The primary function of a BGP speaking system is to exchange network reachability information with other BGP systems. This network reachability information includes information on the list of Autonomous Systems (ASes) that reachability information traverses. This information is sufficient for constructing a graph of AS connectivity for this reachability from which routing loops may be pruned, and, at the AS level, some policy decisions may be enforced. BGP-4 provides a set of mechanisms for supporting Classless Inter-Domain Routing (CIDR). These mechanisms include support for advertising a set of destinations as an IP prefix, and eliminating the concept of network "class" within BGP. BGP-4 also introduces mechanisms that allow aggregation of routes, including aggregation of AS paths. This document obsoletes RFC 1771. [STANDARDS-TRACK] The Border Gateway Protocol (BGP) is an inter-autonomous system routing protocol designed for TCP/IP internets. Typically, all BGP speakers within a single AS must be fully meshed so that any external routing information must be re-distributed to all other routers within that Autonomous System (AS). This represents a serious scaling problem that has been well documented with several alternatives proposed. This document describes the use and design of a method known as "route reflection" to alleviate the need for "full mesh" Internal BGP (IBGP). This document obsoletes RFC 2796 and RFC 1966. [STANDARDS-TRACK] This memo describes a technique for dynamically discovering the maximum transmission unit (MTU) of an arbitrary internet path. It specifies a small change to the way routers generate one type of ICMP message. For a path that passes through a router that has not been so changed, this technique might not discover the correct Path MTU, but it will always choose a Path MTU as accurate as, and in many cases more accurate than, the Path MTU that would be chosen by current practice. [STANDARDS-TRACK] This document describes an extension to BGP which may be used to pass additional information to both neighboring and remote BGP peers. [STANDARDS-TRACK] This document defines an architecture for implementing scalable service differentiation in the Internet. This memo provides information for the Internet community. This memo proposes an implementation of SACK and discusses its performance and related issues. [STANDARDS-TRACK] This document describes a method by which a Service Provider may use an IP backbone to provide IP Virtual Private Networks (VPNs) for its customers. This method uses a "peer model", in which the customers' edge routers (CE routers) send their routes to the Service Provider's edge routers (PE routers); there is no "overlay" visible to the customer's routing algorithm, and CE routers at different sites do not peer with each other. Data packets are tunneled through the backbone, so that the core routers do not need to know the VPN routes. [STANDARDS-TRACK] This document defines Multi-Protocol BGP (MP-BGP) procedures that allow BGP speakers to exchange Route Target reachability information. This information can be used to build a route distribution graph in order to limit the propagation of Virtual Private Network (VPN) Network Layer Reachability Information (NLRI) between different autonomous systems or distinct clusters of the same autonomous system. This document updates RFC 4364. [STANDARDS-TRACK] This document describes a mechanism for BGP that would help minimize the negative effects on routing caused by BGP restart. An End-of-RIB marker is specified and can be used to convey routing convergence information. A new BGP capability, termed "Graceful Restart Capability", is defined that would allow a BGP speaker to express its ability to preserve forwarding state during BGP restart. Finally, procedures are outlined for temporarily retaining routing information across a TCP session termination/re-establishment. The mechanisms described in this document are applicable to all routers, both those with the ability to preserve forwarding state during BGP restart and those without (although the latter need to implement only a subset of the mechanisms described in this document). [STANDARDS-TRACK] This document explains how to interconnect IPv6 islands over a Multiprotocol Label Switching (MPLS)-enabled IPv4 cloud. This approach relies on IPv6 Provider Edge routers (6PE), which are Dual Stack in order to connect to IPv6 islands and to the MPLS core, which is only required to run IPv4 MPLS. The 6PE routers exchange the IPv6 reachability information transparently over the core using the Multiprotocol Border Gateway Protocol (MP-BGP) over IPv4. In doing so, the BGP Next Hop field is used to convey the IPv4 address of the 6PE router so that dynamically established IPv4-signaled MPLS Label Switched Paths (LSPs) can be used without explicit tunnel configuration. [STANDARDS-TRACK] This document specifies the TCP Authentication Option (TCP-AO), which obsoletes the TCP MD5 Signature option of RFC 2385 (TCP MD5). TCP-AO specifies the use of stronger Message Authentication Codes (MACs), protects against replays even for long-lived TCP connections, and provides more details on the association of security with TCP connections than TCP MD5. TCP-AO is compatible with either a static Master Key Tuple (MKT) configuration or an external, out-of-band MKT management mechanism; in either case, TCP-AO also protects connections when using the same MKT across repeated instances of a connection, using traffic keys derived from the MKT, and coordinates MKT changes between endpoints. The result is intended to support current infrastructure uses of TCP MD5, such as to protect long-lived connections (as used, e.g., in BGP and LDP), and to support a larger set of MACs with minimal other system and operational changes. TCP-AO uses a different option identifier than TCP MD5, even though TCP-AO and TCP MD5 are never permitted to be used simultaneously. TCP-AO supports IPv6, and is fully compatible with the proposed requirements for the replacement of TCP MD5. [STANDARDS-TRACK] This memo provides a method for protecting a router's control plane from undesired or malicious traffic. In this approach, all legitimate router control plane traffic is identified. Once legitimate traffic has been identified, a filter is deployed in the router's forwarding plane. That filter prevents traffic not specifically identified as legitimate from reaching the router's control plane, or rate-limits such traffic to an acceptable level. Note that the filters described in this memo are applied only to traffic that is destined for the router, and not to all traffic that is passing through the router. This document is not an Internet Standards Track specification; it is published for informational purposes. This document describes procedures for BGP MPLS-based Ethernet VPNs (EVPN). The procedures described here meet the requirements specified in RFC 7209 -- "Requirements for Ethernet VPN (EVPN)". The Border Gateway Protocol (BGP) is the protocol almost exclusively used in the Internet to exchange routing information between network domains. Due to this central nature, it is important to understand the security measures that can and should be deployed to prevent accidental or intentional routing disturbances. This document describes measures to protect the BGP sessions itself such as Time to Live (TTL), the TCP Authentication Option (TCP-AO), and control-plane filtering. It also describes measures to better control the flow of routing information, using prefix filtering and automation of prefix filters, max-prefix filtering, Autonomous System (AS) path filtering, route flap dampening, and BGP community scrubbing. According to the base BGP specification, a BGP speaker that receives an UPDATE message containing a malformed attribute is required to reset the session over which the offending attribute was received. This behavior is undesirable because a session reset would impact not only routes with the offending attribute but also other valid routes exchanged over the session. This document partially revises the error handling for UPDATE messages and provides guidelines for the authors of documents defining new attributes. Finally, it revises the error handling procedures for a number of existing attributes. This document updates error handling for RFCs 1997, 4271, 4360, 4456, 4760, 5543, 5701, and 6368. This document defines a BGP extension that allows the advertisement of multiple paths for the same address prefix without the new paths implicitly replacing any previous ones. The essence of the extension is that each path is identified by a Path Identifier in addition to the address prefix. Some network operators build and operate data centers that support over one hundred thousand servers. In this document, such data centers are referred to as "large-scale" to differentiate them from smaller infrastructures. Environments of this scale have a unique set of network requirements with an emphasis on operational simplicity and network stability. This document summarizes operational experience in designing and operating large-scale data centers using BGP as the only routing protocol. The intent is to report on a proven and stable routing design that could be leveraged by others in the industry. Routing information reduction by BGP Route Reflection or Confederation can result in persistent internal BGP route oscillations with certain routing setups and network topologies. This document specifies two sets of additional paths that can be used to eliminate these route oscillations in a network. This document describes Path MTU Discovery (PMTUD) for IP version 6. It is largely derived from RFC 1191, which describes Path MTU Discovery for IP version 4. It obsoletes RFC 1981. This document specifies a set of procedures for using BGP to advertise that a specified router has bound a specified MPLS label (or a specified sequence of MPLS labels organized as a contiguous part of a label stack) to a specified address prefix. This can be done by sending a BGP UPDATE message whose Network Layer Reachability Information field contains both the prefix and the MPLS label(s) and whose Next Hop field identifies the node at which said prefix is bound to said label(s). This document obsoletes RFC 3107. The BGP specification (RFC 4271) mandates a maximum BGP message size of 4,096 octets. As BGP is extended to support new Address Family Identifiers (AFIs), Subsequent AFIs (SAFIs), and other features, there is a need to extend the maximum message size beyond 4,096 octets. This document updates the BGP specification by extending the maximum message size from 4,096 octets to 65,535 octets for all messages except for OPEN and KEEPALIVE messages. This document defines a Border Gateway Protocol Network Layer Reachability Information (BGP NLRI) encoding format that can be used to distribute (intra-domain and inter-domain) traffic Flow Specifications for IPv4 unicast and IPv4 BGP/MPLS VPN services. This allows the routing system to propagate information regarding more specific components of the traffic aggregate defined by an IP destination prefix. It also specifies BGP Extended Community encoding formats, which can be used to propagate Traffic Filtering Actions along with the Flow Specification NLRI. Those Traffic Filtering Actions encode actions a routing system can take if the packet matches the Flow Specification. This document obsoletes both RFC 5575 and RFC 7674. This document defines an extension to BGP route reflectors. On route reflectors, BGP route selection is modified in order to choose the best route from the standpoint of their clients, rather than from the standpoint of the route reflectors themselves. Depending on the scaling and precision requirements, route selection can be specific for one client, common for a set of clients, or common for all clients of a route reflector. This solution is particularly applicable in deployments using centralized route reflectors, where choosing the best route based on the route reflector's IGP location is suboptimal. This facilitates, for example, a "best exit point" policy ("hot potato routing"). The solution relies upon all route reflectors learning all paths that are eligible for consideration. BGP route selection is performed in the route reflectors based on the IGP cost from configured locations in the link-state IGP. This document specifies the Transmission Control Protocol (TCP). TCP is an important transport-layer protocol in the Internet protocol stack, and it has continuously evolved over decades of use and growth of the Internet. Over this time, a number of changes have been made to TCP as it was specified in RFC 793, though these have only been documented in a piecemeal fashion. This document collects and brings those changes together with the protocol specification from RFC 793. This document obsoletes RFC 793, as well as RFCs 879, 2873, 6093, 6429, 6528, and 6691 that updated parts of RFC 793. It updates RFCs 1011 and 1122, and it should be considered as a replacement for the portions of those documents dealing with TCP requirements. It also updates RFC 5961 by adding a small clarification in reset handling while in the SYN-RECEIVED state. The TCP header control bits from RFC 793 have also been updated based on RFC 3168. In many environments, a component external to a network is called upon to perform computations based on the network topology and the current state of the connections within the network, including Traffic Engineering (TE) information. This is information typically distributed by IGP routing protocols within the network. This document describes a mechanism by which link-state and TE information can be collected from networks and shared with external components using the BGP routing protocol. This is achieved using a BGP Network Layer Reachability Information (NLRI) encoding format. The mechanism applies to physical and virtual (e.g., tunnel) IGP links. The mechanism described is subject to policy control. Applications of this technique include Application-Layer Traffic Optimization (ALTO) servers and Path Computation Elements (PCEs). This document obsoletes RFC 7752 by completely replacing that document. It makes some small changes and clarifications to the previous specification. This document also obsoletes RFC 9029 by incorporating the updates that it made to RFC 7752. This document describes a BGP-based routing solution to establish end-to-end intent-aware paths across a multi-domain transport network. The transport network can span multiple service provider and customer network domains. The BGP intent-aware paths can be used to steer traffic flows for service routes that need a specific intent. This solution is called BGP Color-Aware Routing (BGP CAR). This document describes the routing framework and BGP extensions to enable intent-aware routing using the BGP CAR solution. The solution defines two new BGP SAFIs (BGP CAR SAFI and BGP VPN CAR SAFI) for IPv4 and IPv6. It also defines an extensible Network Layer Reachability Information (NLRI) model for both SAFIs that allows multiple NLRI types to be defined for different use cases. Each type of NLRI contains key and TLV-based non-key fields for efficient encoding of different per-prefix information. This specification defines two NLRI types: Color-Aware Route NLRI and IP Prefix NLRI. It defines non-key TLV types for the MPLS label stack, SR-MPLS label index, and Segment Routing over IPv6 (SRv6) Segment Identifiers (SIDs). This solution also defines a new Local Color Mapping (LCM) Extended Community. HPE HPE Huawei Technologies Huawei Technologies Arrcus Cisco HPE Arrcus Deutsche Telekom AG Orange Cisco Systems Cisco Systems Steinberg Consulting

Acknowledgments Authors would like to thank Ketan Talaulikar, Jakob Heitz, Lokesh Khanna and Phil Bedard for their valuable guidance and collaboration on the numerous topics covered in this draft.

Contributors Cisco

slitkows@cisco.com