MRTE: Multipath Traffic Engineering for Reliable and Available Wireless DetNet

Introduction Deterministic Networking (DetNet) provides flows with bounded latency, bounded jitter and very low packet loss across a network in which the path from ingress to egress is explicitly managed. One of the main tools used by DetNet to achieve low loss is the set of Packet Replication, Elimination and Ordering Functions (PREOF): a packet is replicated onto two or more paths, duplicates are eliminated where the paths rejoin, and packets are put back in order at the egress. Reliable and Available Wireless (RAW) extends DetNet to wireless links. The use cases that motivate RAW are described in , and the radio technologies to which RAW applies are surveyed in . The central observation of RAW is that a wireless link is not simply a lossier wired link: its quality changes over time scales of milliseconds to seconds, as a result of fading, interference, mobility and contention. A path that was computed and provisioned minutes ago may have one or more links that are currently unusable; conversely, a link that was unusable a moment ago may now be fine. In such an environment, PREOF across a fixed pair of disjoint end-to-end paths is both wasteful (when both paths are good) and insufficient (when both paths are bad at once). The RAW Architecture therefore computes a "recovery graph" (which it equates with the 6TiSCH notion of a Track ): not a single path, nor a pair of paths, but a Directed Acyclic Graph (DAG) that is the coalescence of all the feasible paths a packet of the flow may follow from the ingress node to the egress node. Replication and elimination happen throughout the recovery graph, wherever paths diverge and converge, rather than only at the ends. RAW also recognizes that wireless links typically have their own per-hop retransmission mechanisms (Automatic Repeat reQuest, or ARQ), which complement PREOF, and that overhearing (a node receiving a packet not addressed to it) can also be exploited. Multipath Traffic Engineering (MPTE) and Multicast Traffic Engineering (MCTE) share three notions that this memo builds on: that of a traffic-engineered DAG from one or more ingresses to one or more egresses, computed subject to TE constraints; that of a junction: a junction J is a node in a DAG with previous hops (phops) and next hops (nhops); on receiving traffic from a phop, J forwards it to one or more of its nhops; and that of direct signaling from a Signaling Source (SS) to each junction in the DAG to provision the tunnel. MPTE and MCTE differ in what a junction does with a packet. In MPTE, a junction load-balances: each packet goes to exactly one nhop. In MCTE, a junction replicates: each packet goes to every nhop. Neither, as it stands, is what RAW needs. RAW needs replication, as in MCTE, but not necessarily to all nhops. RAW also needs the reverse: a junction where several copies of a packet arrive and only one goes on. This memo defines MRTE. MRTE has no expansion of its own; if one is wanted, "MPTE for RAW" will do. MRTE takes the DAG, the junction and direct signaling from MPTE, and adds three new types of junction: the bounded replication junction, the elimination junction, and the ordering junction, which restores packet order after elimination has mixed packets from different paths. In addition, MRTE allows a junction to have multiple forwarding functions: a junction may simultaneously eliminate and order packets. The result is a signaled, traffic-engineered DAG -- an MRTE DAG, or MRTED -- in which replication and elimination are interleaved so that a packet is delivered if any one path through the DAG is working, while the number of copies in flight at any stage stays bounded. An MRTED is a direct realization of a RAW recovery graph, or Track. This version of the memo does not use load balancing: every junction in an MRTED either replicates to its nhops or has a single nhop. (If load balancing is useful in the context of RAW, it can certainly be re-introduced.) In this version, ordering junctions are used only at the egress; a future version may allow ordering junctions elsewhere in the MRTED (see ). The rapid local adjustment of forwarding that the RAW Architecture assigns to the Point of Local Repair (PLR) is described in . This memo does not define a signaling protocol; the information described here is to be carried by the signaling protocols being defined for MPTE and MCTE, extended as needed. Nor does it define an algorithm for computing an MRTED.

Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. This memo uses the terminology of , in particular: DAG, junction, phop, nhop, oif, slack, constraints, traffic trunk, Tunnel Originator (TO), DAG computer (MC) and Signaling Source (SS). It also uses the terminology of and . The following terms are specific to, or of particular importance in, this memo.

MRTE:: the application of the MPTE notions of DAG, junction and direct signaling, together with the MCTE notion of a replicating junction, to RAW DetNet. The term has no expansion; "MPTE for RAW" may be used if one is needed.
MRTED:: an MRTE DAG, from one or more ingresses to one or more egresses, consisting of replication, elimination and ordering junctions. An MRTED realizes a RAW recovery graph.
MRTET:: an MRTE tunnel: the signaled (and hence, forwarding) entity associated with an MRTED.
recovery graph:: in RAW , the coalescence of all the feasible DetNet paths that a packet of a flow may be forwarded along, from one ingress node to one egress node, together with metadata describing how the paths are built and used. equates the recovery graph with the 6TiSCH Track ; this memo uses "Track" as a synonym. An MRTED realizes a recovery graph.
segment:: in RAW, a hop of a recovery graph between two adjacent nodes. In this memo, a segment is an MRTED link, identified as in by (u, v, oif).
Point of Local Repair (PLR):: in RAW , the function that performs rapid local adjustments of the forwarding tables within the path diversity of the recovery graph. This is done by expanding or contracting the replication factor at this junction, based on link quality learned via DLEP.
DLEP:: the Dynamic Link Exchange Protocol , by which a radio (modem) reports to its attached router the state and metrics of the wireless links it provides: whether a neighbor is reachable, the current data rate, latency, and relative link quality, among others. In this memo, DLEP is the primary source of link quality information at a junction.
replication junction:: a junction with two or more nhops. A replication junction forwards a copy of each packet it receives to every one of its nhops. This is the behavior of a junction in MCTE .
replication factor:: the number of nhops to which copies of a packet are sent. This is at most all nhops and at least one. A minimum replication factor may be included in a junction specification, in which case a replication junction MUST forward at least that many copies of the packet.
elimination junction:: a junction with two or more phops. An elimination junction may receive more than one copy of a packet, one from each phop; it forwards the first and discards the rest.
ordering junction:: a junction that reorders packets of a flow by sequence number before forwarding or delivering them. In this version of the memo, only an egress may be an ordering junction.
PREOF:: Packet Replication, Elimination and Ordering Functions, as defined in and used in .
ARQ:: Automatic Repeat reQuest; per-hop, link-layer retransmission of lost frames, as provided by many radio technologies . treats ARQ as a lower-layer reliability function distinct from, and complementary to, PREOF.
stage:: informally, a set of junctions at the same distance (in hops) from the ingress. In a well-formed MRTED, elimination at one stage keeps the number of copies entering the next stage bounded.

Mapping RAW onto MRTE summarizes how the elements of the RAW Architecture correspond to those of MRTE. RAW MRTE recovery graph (Track) MRTED ingress node ingress junction (pure or transit) egress node egress junction (pure or transit) segment MRTED link (u, v, oif) PCE MC recovery graph provisioning SS signaling JUNCTION messages Replication replication junction Elimination elimination junction Ordering ordering junction (at egress, this version) ARQ property of a segment (radio layer) link quality observation DLEP at each junction; NOTIFY to SS PLR local to a junction, using DLEP Two aspects of the mapping deserve comment. First, the RAW Architecture states that a recovery graph has one ingress node and one egress node. An MRTED, like an MPTED, admits several of each. This is directly useful in wireless networks: a wireless end device is often within range of more than one access point or relay, and a destination is often reachable through more than one gateway. Treating these as multiple ingresses or egresses of a single MRTED, rather than as separate recovery graphs, lets the MC optimize over the entire set of options and lets the network share forwarding state. With multiple ingresses, each ingress injects its own copy of the traffic trunk; with multiple egresses, each egress delivers whatever reaches it, and the elimination state is per egress. Second, in the RAW Architecture, replication and elimination are functions that may be placed at nodes of the recovery graph. In MRTE, they are properties of the DAG's shape: a junction with several nhops replicates, a junction with several phops eliminates, and a junction with several of each does both. The MC controls where replication and elimination happen by controlling which links are in the MRTED. There is no separate configuration of "replicate here" or "eliminate here"; the JUNCTION message tells a junction its phops and nhops, and the junction's behavior follows. (A junction does need to be told that it is an MRTE junction rather than an MPTE or MCTE one, since the forwarding behavior on receipt differs; see .) Ordering, by contrast, is not implied by the DAG's shape: a junction is an ordering junction because the MC says so. In this version, the MC may say so only of an egress.

Overview Consider the wireless mesh in . Node 0 is the ingress; node 7 is the egress; the other nodes are relays. Every link is wireless. Each link is annotated with its nominal packet delivery ratio (PDR), the fraction of transmissions that succeed on the first attempt under typical conditions.

With a metric of 1 per hop, all eight paths from 0 to 7 (0-{1,2}-{3,4}-{5,6}-7) have length 4; the shortest-path DAG from 0 to 7 is the whole mesh. The MC's job is to decide which of these links to include in the MRTED.

A Single Path If a single path is used, say 0-1-3-5-7, and link-layer ARQ brings each link's effective loss down to about 1%, the path loses about 4% of packets, and loses them in bursts when any one link fades. This does not meet any DetNet loss target of interest.

Two Disjoint Paths Classical PREOF would replicate at 0 onto two disjoint paths, say 0-1-3-5-7 and 0-2-4-6-7, and eliminate at 7. A packet is lost only if both paths lose it: roughly 0.16% under the above assumptions. However, a single fade on link 1-3 and a single fade on link 4-6, at the same time, lose everything, even though links 1-4, 2-3, 3-6 and 4-5 are all fine. In an MPTE or MCTE DAG, these two paths would be nodes 1 through 6 each having exactly one phop and one nhop; the DAG is really just two paths.

An MRTED Now let the MRTED be the whole mesh. Node 0 has two nhops; it is a replication junction. Nodes 1 and 2 each have one phop and two nhops; they are replication junctions. Nodes 3 and 4 each have two phops and two nhops; they are both elimination and replication junctions. Nodes 5 and 6 each have two phops and one nhop; they are elimination junctions. Node 7 has two phops and is the egress; it is an elimination and ordering junction. A packet entering at 0 is sent to both 1 and 2. Node 1 sends copies to 3 and 4; so does node 2. Nodes 3 and 4 each receive two copies, forward the first to both 5 and 6, and discard the second. Nodes 5 and 6 each receive two copies, forward the first to 7 and discard the second. Node 7 receives two copies, delivers the first in sequence and discards the second. At every stage, at most two copies of the packet are in flight, exactly as with two disjoint paths. But the packet is lost only if every link at some stage fails at once: both 0-1 and 0-2, or all four of 1-3, 1-4, 2-3, 2-4, or all four of 3-5, 3-6, 4-5, 4-6, or both 5-7 and 6-7. Under the above assumptions this is on the order of 0.02%, an order of magnitude better than two disjoint paths, for the same number of copies in flight, at the cost of using twelve links rather than eight. This is what the 6TiSCH Architecture calls a "Complex Track": replication and elimination interleaved throughout the DAG so that paths can diverge and converge repeatedly. The RAW Architecture builds on the same idea. MRTE realizes it with no per-node configuration beyond the junction's phops and nhops.

Bandwidth Each link in the MRTED carries one copy of every packet of the traffic trunk (ignoring losses). The bandwidth consumed by an MRTET is therefore the bandwidth of the traffic trunk times the number of links in the MRTED, and the bandwidth on each link is simply the bandwidth of the traffic trunk. This is simpler to account for than in MPTE, where each link carries a share, and the MC MUST reserve it on every link of the MRTED.

Junctions In MPTE and MCTE, a junction node is one of a small number of types based on where traffic enters and leaves: pure ingress, pure egress, transit ingress, transit (or bud) egress, and regular. [Are transit ingresses and egresses of interest to DetNet?] These types carry over to MRTE unchanged. MRTE adds a second, orthogonal classification based on what a junction does with a packet. There are three such types: replication junctions and elimination junctions, which are determined by the number of a junction's nhops and phops respectively, and ordering junctions, which are designated by the MC. A junction may be of more than one type at once.

Replication Junction A junction with one or more nhops is a replication junction. On forwarding a packet, it sends a copy to a non-empty subset of its nhops. A JUNCTION message may include a minimum replication factor (MRF), which is computed by the MC based on what it knows of link quality to the nhops. A junction with exactly one nhop forwards each packet to that nhop. It is a degenerate replication junction and needs no special treatment. A replication junction MUST NOT send more than one copy of a given packet to the same nhop.

Elimination Junction A junction with two or more phops is an elimination junction. It maintains, per MRTET and per flow, a record of recently seen sequence numbers. On receipt of a packet it checks the packet's sequence number against this record. If the sequence number has been seen, the packet is discarded; otherwise it is recorded and the packet is forwarded (to a non-empty subset of its nhops, if the junction has a non-zero number of nhops; or to the egress function, if the junction is an egress). Elimination requires that packets carry a per-flow sequence number assigned once, at the ingress, and carried unchanged by every copy. discusses which tunnel types provide one. The MC MUST NOT include an elimination junction in an MRTED whose tunnel type does not carry a sequence number. The size of the sequence number history and the handling of wraparound are as described for the DetNet Packet Elimination Function in and ; this memo adds nothing to them. A junction with exactly one phop cannot receive duplicates (except by overhearing; see ) and need not keep elimination state.

Ordering Junction An ordering junction buffers packets of a flow and releases them in sequence-number order, up to a configured maximum delay (the Ordering Delay), after which a missing packet is declared lost and release resumes with the next packet held. This is the DetNet Packet Ordering Function of . Released packets are then handled as they would be at any other junction: forwarded to the junction's nhops, or delivered to the traffic trunk's destination at an egress. Unlike replication and elimination, ordering is not implied by the shape of the MRTED. The MC designates a junction as an ordering junction in its JUNCTION message, together with the Ordering Delay. In this version of the memo, the MC MUST designate only egress junctions as ordering junctions, and an egress SHOULD be an ordering junction unless the flow tolerates reordering. A future version may permit ordering junctions elsewhere in the MRTED, for example at a junction where several stages of replication and elimination converge, so that downstream junctions see an ordered stream and the egress has less reordering to do; see . Ordering matters more for MRTE than for a single path. In an MRTED, as in any DAG with elimination, the copy that survives at an elimination junction is whichever arrived first, which may have taken a different path from the surviving copy of the previous packet. Packets of a flow can therefore arrive at an elimination junction out of order even though every individual path preserves order, and this reordering accumulates stage by stage. An ordering junction adds up to the Ordering Delay to the latency of a packet. The MC MUST account for this in the flow's latency budget.

PLR The RAW Architecture describes a Point of Local Repair (PLR) that, at a time scale faster than the PCE, makes rapid local adjustments to forwarding within the path diversity of the recovery graph, based on observed link quality, so that replication can be reduced when conditions are good and increased when they are not. This loop looks like radio -> DLEP -> junction -> local adjustment. One example of local adjustment is the choice of nhops to which to replicate. In MCTE, a replication junction MUST replicate incoming packets to all its nhops; otherwise, some leaves of the multicast tree will not receive these packets. In MRTE, however, replication serves to increase packet availability. If a link currently has high quality, replication may be redundant; on the other hand, if link quality drops, a greater degree of replication may be warranted to maintain packet availability. Which subset of the nhops to replicate to is a decision that can often be made by a PLR on the basis of link quality of the nhops, learned through the Dynamic Link Exchange Protocol (DLEP) . This achieves the PLR's function at the junction's time scale. Changes to link quality can be reported to the SS via NOTIFY messages, which can cause the MC to recompute the MRTED (). Note that the MPTE optimization of keeping the DAG shape constant even when links go down applies equally well to MRTE; what might change is the minimum replication factor.

Multiple Ingresses and Egresses An MRTED with several ingresses behaves as several MRTEDs sharing junctions: each ingress injects its own packets, and sequence numbers MUST be unique across all ingresses of the MRTET for a given flow (for example, by having each ingress carry a distinct flow, or by coordinating sequence number ranges). This is a constraint on the traffic trunk definition rather than on forwarding. An MRTED with several egresses delivers each packet at every egress it reaches. Each egress keeps its own elimination and ordering state. Whether the destination wants the packet from every egress, or from the first egress only, is a matter for the service using the MRTET and is outside the scope of this memo.

ARQ Link-layer retransmission (ARQ) is a property of a wireless segment rather than of a junction, and is governed by the radio technology in use . It affects MRTE in two ways. First, ARQ reduces the effective loss of a segment, which the MC takes into account when deciding how much diversity the MRTED needs. Second, ARQ adds latency; the MC MUST account for the worst-case ARQ delay of each segment when verifying that the MRTED meets a flow's latency bound .

Overhearing Some radio technologies allow a node to receive a packet transmitted to a neighbor. lists this "promiscuous overhearing" among the wireless capabilities RAW can exploit: if node 5 in overhears the copy that node 3 sent to node 6, it can treat it as an additional copy received. This memo permits an elimination junction to accept overheard packets of an MRTET as if they had been addressed to it; the elimination function deduplicates them against copies that were addressed to it. A junction that is not an elimination junction MUST NOT accept overheard packets, since it has no means to discard duplicates. Whether and how overhearing is enabled is radio-specific and outside the scope of this memo.

Tunnel Types for MRTE lists a number of tunnel types, each with or without an entropy field, a discriminator, and support for multiple egresses and multiple ingresses. For MRTE, two further properties matter.

Sequence number (SN):: Elimination and ordering require a per-flow sequence number that is assigned once, by the ingress, and carried unchanged by every copy of the packet.
Flow identification (FI):: DetNet flows within an MRTET must be identifiable at elimination and ordering junctions, so that sequence number history is kept per flow.

An entropy field is not needed, since there is no load balancing in MRTE. The suitable tunnel types are as follows.

SigLab and StatLab with DetNet Control Word:: MPLS-based MRTETs can carry the DetNet Control Word (d-CW) immediately below the bottom of the label stack. The d-CW carries a 28-bit sequence number. Flow identification is provided by a DetNet service label (S-Label) above the d-CW, or by the MRTET label itself when the MRTET carries a single flow. This is the RECOMMENDED tunnel type for MRTE. Signaled and static labels work as in ; with signaled labels, an elimination junction allocates one incoming label per phop, or a single label for all phops, at its discretion.
MPLS-in-UDP with d-CW:: As above, with the MPLS packet encapsulated in UDP/IP , . Useful when intermediate wireless segments are IP-only.
IP-based tunnels (IP-in-IP, GRE, GRE6, G-in-U, SRv6):: The DetNet IP data plane identifies flows by a 6-tuple but provides no sequence number. An IP-based MRTET can therefore contain replication junctions but no elimination junctions, unless the payload itself carries a sequence number that the junctions are configured to inspect (for example, an encapsulated DetNet MPLS packet ). Since an MRTED with replication but no elimination delivers multiple copies to the destination and is simply an MCTED, IP-based tunnels are NOT RECOMMENDED for MRTE.

An MRTET has a single tunnel type throughout; this memo does not define mixed-type MRTEDs.

6TiSCH Tracks The 6TiSCH Architecture defines a Track as a set of scheduled cells in a Time-Slotted Channel Hopping (TSCH) schedule, forming a path or DAG from a source to a destination, and is one of the technologies considered by RAW . In a 6TiSCH network, an MRTET maps onto a 6TiSCH Track: the MRTED gives the nodes and links of the Track, and each MRTED link requires enough cells to be scheduled between its endpoints to carry the traffic trunk's bandwidth. The scheduling of cells is performed by the 6TiSCH mechanisms and is outside the scope of this memo; the information needed to drive it (which links, how much bandwidth) is exactly what the JUNCTION message carries.

Operation The steps to create an MRTET follow those of MPTE and MCTE. Define the traffic trunk for the MRTET, including the DetNet flow requirements: maximum loss ratio, maximum latency, maximum jitter and bandwidth, as in the DetNet flow information model . Define the constraints of the traffic trunk, including: ingresses, and the bandwidth entering the DAG at each ingress; egresses; metric to minimize (perhaps with slack); criteria of acceptable nodes and links for the DAG, including link colors, shared risk groups, radio technology, nominal PDR and ARQ limits; the degree of diversity sought, for example a minimum number of links at every stage. This information is given to the Tunnel Originator (TO). The TO sends this information to the MC. The MC computes an MRTED that it expects to meet the flow's loss and latency requirements under the range of link conditions it anticipates. It verifies that the traffic trunk's bandwidth is available on every link of the MRTED, and that the latency along every path through the MRTED, including worst-case ARQ and the ordering delay at the egress, is within budget. The MRTED is expressed as a set of junctions, which are sent to the SS. The SS instantiates the MRTET by sending a JUNCTION message to each junction. When ready, the SS tells each ingress that the MRTET is ready for traffic. The ingresses map traffic matching the traffic trunk to the MRTET, assigning sequence numbers. Computation of an MRTED is done by the MC, which may be an ingress or a PCE . This memo does not specify such an algorithm. The MC is expected to recompute the MRTED when the network changes or when reports from junctions indicate that it is no longer adequate; a recomputation yields a new version of the MRTED, signaled as an update.

MRTED As in MPTE, a node is identified by its IP loopback address, a link by the loopback address of its upstream node and that node's outgoing interface index, and an MRTED by an ID assigned by the MC together with a version number. An MRTED is broken down into a set of junctions. A junction v is specified by: bandwidth (which is the same coming in to and going out of v, and the same on every link); a list of phops of v; a list of N nhops of v. an optional minimum replication factor (MRF), which MUST be at most N. v MUST send out at least MRF copies of an incoming packet. A junction's forwarding behavior is determined by this specification together with one flag: elimination if it has more than one phop, replication if it has more than one nhop, and ordering if the MC has designated it an ordering junction (in this version, only at an egress).

Signaling The signaling protocols being extended for MPTE and MCTE (RSVP-TE , PCEP , BGP , and others) carry JUNCTION, LABEL and NOTIFY messages. This section describes what these must carry for MRTE. The encoding is left to the companion signaling documents.

Message Flow As in MPTE and MCTE: provisioning messages from the SS to each junction; notifications from each junction to the SS; LABEL messages hop-by-hop from egresses to their phops when the tunnel type requires it.

JUNCTION A JUNCTION message for an MRTE junction contains the following information elements.

MRTET ID:: a unique identifier for the MRTET, including a version number, as for the MPTET ID in .
DAG Type:: an indication that this is an MRTE junction (as opposed to an MPTE or MCTE junction). This determines the junction's behavior on receipt (eliminate if more than one phop) and on transmission (replicate to all nhops). The DAG Type is the one element that distinguishes an MRTE JUNCTION message from an MCTE one with the same phops and nhops.
Tunnel Type and Tunnel Information:: as in .
Junction Bandwidth:: the bandwidth of the traffic trunk in Mbps. This is the bandwidth on every phop and every nhop link.
phop list:: the phops of the junction. An elimination junction uses this to know how many copies it may expect; with signaled labels, it allocates labels for these.
nhop list:: the nhops of the junction, with an optional MRF.
Sequence Number Parameters:: for junctions with more than one phop, and for egresses: the sequence number length (16 or 28 bits, per ) and the elimination history window size.
Flow Identification:: how flows are identified within the MRTET for the purpose of per-flow elimination and ordering state; for MPLS, an S-Label or an indication that the MRTET carries a single flow.
Ordering:: whether this junction is an ordering junction. In this version, this MUST be set only for egress junctions; a junction that is not an egress and receives this flag set MUST reject the JUNCTION message via NOTIFY.
Ordering Delay:: for ordering junctions, the maximum time in microseconds to wait for a missing sequence number before declaring it lost.
Overhearing:: for junctions with more than one phop, whether overheard packets of the MRTET may be accepted.
Link Quality Reporting Parameters:: for each nhop (and optionally phop) segment, the thresholds, absolute bounds, rate limits and hysteresis to be applied to DLEP-triggered notifications, as described in .

LABEL As in . An elimination junction MAY allocate the same label for all of its phops, since it treats packets from all phops identically.

NOTIFY As in : a junction reports to the SS the labels it has allocated and any error conditions. In addition, an MRTE junction SHOULD be able to report per-phop and per-nhop statistics useful to the MC: packets received per phop, duplicates eliminated, packets with no duplicate received (indicating that the other paths to this junction are losing packets), and ARQ retransmission counts per nhop. The signaling documents are also to define a Link Quality notification, by which a junction reports DLEP events and metrics for a segment of the MRTET to the SS, subject to the thresholding of . It carries the MRTET ID, the segment, the event type, the DLEP metrics and values, and an indication of whether notifications for this segment were suppressed since the last report. The MC uses all of these at its (slow) time scale to decide whether the MRTED still provides adequate diversity and, if not, to recompute it.

Forwarding State An ingress identifies the traffic trunk, encapsulates each packet in the signaled tunnel type with a sequence number, and forwards a copy to every nhop. A non-ingress junction v has a FIB entry whose route identifies the MRTET from v's phops (one label per phop, or a common label, for MPLS). The action on a match is: if v has more than one phop, check the sequence number against the elimination history and discard if seen; if v is an ordering junction, hold the packet until it is next in sequence or the Ordering Delay expires; then, if v has nhops, forward a copy to each of them (performing a label swap per nhop for signaled labels); and, if v is an egress, deliver the packet to the traffic trunk's destination.

Bounded Latency Considerations describes how DetNet bounds latency along a path by accounting for queuing at each hop. In an MRTED, the surviving copy of a packet may have taken any path through the DAG, so the latency bound for the flow is the maximum over all paths in the MRTED. Replication does not increase the latency of the first copy to arrive; elimination adds a small, bounded per-hop processing time; ordering adds up to the Ordering Delay at the egress. The MC MUST include these in its latency computation. Since every path in the MRTED carries every packet, the MC MUST also reserve queuing resources for the traffic trunk on every link of the MRTED. A path through the MRTED that is within the slack with respect to the metric is not necessarily within the latency budget, if the metric is not latency. The MC MUST verify the latency bound independently of the slack.

Link Quality and DLEP An MRTED is computed on the basis of what the MC knows about the wireless links in the network. For the MRTED to remain adequate, the MC needs to learn when that knowledge is out of date. On a wireless link, the entity that knows first is the radio. The Dynamic Link Exchange Protocol (DLEP) is a protocol between a router and the radio (modem) attached to it. Through DLEP, the radio tells the router which neighbors are currently reachable (Destination Up and Destination Down messages) and reports metrics for each neighbor and for the link as a whole (Destination Update and Session Update messages), including maximum and current data rate, latency, resources, and a relative link quality (RLQ) indicator. Radio-specific metrics may be added through DLEP extensions. DLEP thus gives a junction an authoritative, radio-level view of each of its wireless segments, updated as conditions change, without the junction having to infer link quality from traffic.

Reporting DLEP Information to the SS An MRTE junction that receives DLEP information for a segment that is an nhop (or phop) of an MRTET SHOULD report significant changes in that information to the SS in a NOTIFY message. The MRTE signaling documents are to define a notification for this purpose, carrying at a minimum: the MRTET ID, the segment (as (u, v, oif)), the type of event (destination up, destination down, or metric update), and the relevant DLEP metrics with their values. The SS passes these reports to the MC. On receiving a report, the MC decides whether the MRTED still meets the flow's requirements. If a segment has gone down, or its latency or data rate has moved outside what the MC assumed, the MC MAY recompute the MRTED -- excluding the degraded segment, adding diversity elsewhere, or both -- and signal the new version to the junctions. Conversely, a report that a previously excluded segment has recovered may prompt the MC to recompute a leaner MRTED. In this way, the loop radio -> DLEP -> junction -> NOTIFY -> SS -> MC -> recompute -> JUNCTION achieves, at the MC's time scale, part of what the RAW Architecture assigns to the PLR: adjusting the set of paths in use in response to observed link quality. The faster loop radio -> DLEP -> junction -> PLR is described in .

Thresholding DLEP can generate updates frequently: a radio may report a changed data rate or RLQ many times a second on a link whose quality is fluctuating. Forwarding every such update to the SS would overwhelm the SS and the MC, and would cause the MC to recompute the MRTED for changes that do not matter. A junction MUST therefore threshold its DLEP-triggered notifications. At a minimum: A notification MUST be sent for a Destination Down event on a segment of the MRTET, and SHOULD be sent for a Destination Up event on a segment that the MC has previously been told is down. A metric update MUST NOT be reported unless the metric has changed by more than a configured threshold since the last report for that segment, or has crossed a configured absolute bound (for example, latency exceeding the segment's share of the flow's latency budget, or data rate falling below the traffic trunk's bandwidth). Notifications for a given segment MUST be rate-limited, so that no more than a configured number are sent per unit time regardless of how often the thresholds are crossed. A Destination Down event SHOULD be exempt from rate limiting, or rate-limited separately with a higher limit. A junction SHOULD apply hysteresis, so that a metric oscillating around a threshold produces one notification, not a stream of them. The thresholds, bounds and rate limits are provided by the MC in the JUNCTION message (), since the MC knows what deviations matter for the flow; defaults are a matter for the signaling documents and local policy. A junction that has suppressed notifications SHOULD indicate, in the next notification it does send for that segment, that intermediate changes occurred, so that the MC knows the segment has been unstable. A junction MAY also aggregate the DLEP information for several segments into a single notification, and MAY include a summary of DLEP state in periodic NOTIFY messages sent for other reasons.

Backward Compatibility A node that implements MPTE or MCTE junctions but not MRTE will not recognize the MRTE DAG Type and MUST reject the JUNCTION message via NOTIFY, so that the MC can route around it. Such a node can still be used in an MRTED in a position where it has exactly one phop and one nhop, since there it behaves identically as an MPTE, MCTE or MRTE junction; the MC MAY signal such a node as an MPTE or MCTE junction. An MCTE-capable node can additionally be used in a position where it has one phop and several nhops, since the behavior there is pure replication. The MPTE capability advertisement is to be extended to indicate MRTE capability, so that the MC knows which nodes may be elimination junctions.

Future Work This version of the memo provisions a static MRTED: every link in the DAG carries every packet, and the MC adapts only by recomputing the DAG, prompted by DLEP-derived notifications (). This version restricts ordering junctions to the egress. A future version will consider ordering junctions at interior points of the MRTED. Placing an ordering junction where several stages of replication and elimination converge would bound the reordering seen downstream, allow the egress a shorter Ordering Delay, and may be necessary when the MRTED is long or when intermediate consumers of the flow (such as transit egresses) require ordered delivery. The cost is buffering and added latency at each ordering junction, which the MC would have to budget. Other topics for future versions include: protection of junctions that have a single phop or nhop; multi-access wireless links, where a single transmission reaches several nhops; and graceful restart.

IANA Considerations None. The companion signaling documents will have IANA requirements, in particular for the DAG Type.

Security Considerations The security considerations of , of DetNet and of the RAW Architecture apply. Elimination junctions keep per-flow state. A compromised node within the MRTED that injects packets with novel sequence numbers could cause downstream elimination junctions to forward them and to churn their history; the history window size signaled by the MC bounds the state consumed. Only the ingress assigns sequence numbers, so this attack requires compromise of a node within the MRTED. Overhearing, where enabled, means that a junction accepts packets that were not addressed to it. The elimination function ensures that this cannot cause duplicate delivery, but a junction accepting overheard packets SHOULD verify that they belong to the MRTET (by label, for MPLS) before processing them. The DLEP-derived notifications of create a path by which the radio environment influences the control plane. A radio that is compromised, or an attacker able to manipulate the radio environment, can cause a junction to send notifications and thereby cause the MC to recompute the MRTED. The thresholding and rate limiting required by bound the rate at which this can happen; the MC SHOULD additionally apply its own damping to recomputation, and SHOULD treat a segment that is reported as repeatedly flapping as unreliable rather than following each report. The security considerations of apply to the router-radio interface.

Acknowledgments Many thanks to the authors of the RAW Architecture and RAW Technologies documents, whose careful framing of the wireless problem made the correspondence with MPTE and MCTE evident, and to the MPTE group at HPE Networking for continuing discussions.