Considerations for PIM-Bidir in Multi-Planed Fat Trees

Introduction Modern Data Centers are increasingly adopting multi-plane. The Figure 3 of PDF version of gives a very good illustration of the multi-plane architecture, and the best textual description can be found in the last three paragraphs of its Section 5.2.2. This document dicusses the challenges and considerations in multi-plane fat tree networks with RP-based multicast solutions like PIM-Bidir . is a RIFT-signaled multicast solution extended from the PIM-Bidir concept, so it is also discussed.

PIM-Bidir RP Placement With PIM-Bidir, the source/receiver routers join towards an RP, forming a tree rooted the RP. Traffic can enter the tree at any tree node; at each node (source or transit), traffic goes upward towards the RP and downward towards the receivers. The RP can be a "Phantom RP", i.e., an address (RPA) on a LAN (RPL, or RP Link) but not associated with any router. The joins and traffic will arrive on routers connected to the LAN, with joins stop there but traffic dumped onto the LAN. Other routers on the LAN will receive the traffic from the LAN and forward downwards if they have join states. This provides the best RP redundnancy. However, LANs are now realized by switches, so the switch becomes a single failure point. Another way to provide RP redundancy is through primary/backup RPs - some routers are provisioned with the same RPA but they advertise the RPA route differently so that at any time only of of them (the primary) is used. The rest of this document focuses on the actual/primary RP. In a fat tree DC fabric, it is intutive to have one of the Top of Fabric (ToF, or SS - Super Spine) switches serve as the RP. This works fine in the single plane case, but in a multi-plane fat tree, if enough links break, the RP may become unreachable by some leaves while still reachable by other leaves. For example, say the RP is the Super Spine 1 in plane D (SS1_D) in Figure 3 of . If the link between SS1_D and S4_1 (the Spine 4 in PoD 1) breaks, all the leaves in PoD 1 will switch to the backup RP while others pods will continue to use the primary RP SS1_D. Or, if the link between L1_1 (the Leaf 1 in PoD 1) breaks, then L1_1 will switch to the backup RP while others will continue to use the primary RP SS1_D. This is not a problem for unicast (as leaf to leaf forwarding still works via other planes), but not acceptable for PIM-Bidir. There are two solutions to this problem: Introduce ring connections between the ToF/SS switches in each plane. However, most DC fabric literatures don't include the ring connections, and it'd becomes another barrier for adopting multicast in DCs. Place the RP on a leaf switch. A leaf switch is guaranteed to be reachable from others except in the case of node or total link failure on the leaf switch, but then all other leaves will converge to the backup RP. There is one caveat with placing the RP on a leaf switch, though. If there are local receivers on the RP, it becomes a bud node for overlay traffic - multicast traffic received from a spine may need to be forwarded to another spine in the underlay, and in the meantime, it needs to be decapsulated and then forwarded to a local receiver. This works, but often requires an internal loopback in the ASIC.

Multicast in RIFT follows the PIM-Bidir concept, except that there are no explicit RPs at all. All the joins are sent towards the ToP (Top of PoD, or Spine) nodes. The mesh of ToP and ToF nodes act as the equivalent of a LAN RPL, but since it is not a LAN, the ToP nodes can't simply dump/get the traffic to/from the RPL. specifies a way for the ToP nodes with matching join states to form a spanning tree via joins through some ToP/ToF nodes. While that works for single plane, multi-plane becomes a problem, because the RPL involving just the ToP/ToF level is no longer enough - leaves need to be used to go across planes unless there are inter-plane ToF rings. Given this complexity/infeasibility (of including leaves in the PRL), or the requirement of using ToF rings, it may be better to simply use PIM-Bidir with leaf RPs, instead of implementing the PIM-Bidir concept with extensions to emulate the RPL.

PIM-ASM RP Placement Similarly, a PIM-ASM RP can only be placed on a leaf switch as well. otherwise, a Super Spine RP may become unreachable by some leaves when enough links break.