| Internet-Draft | taxonomy | November 2026 |
| Joung, et al. | Expires 5 May 2027 | [Page] |
This draft is to facilitate the understanding of the data plane enhancement solutions, which are suggested currently or can be suggested in the future, for deterministic networking. This draft provides criteria for classifying data plane solutions. Examples of each category are listed, along with reasons where necessary. Strengths and limitations of the categories are described. Suitability of the solutions for various services of deterministic networking are also mentioned. Reference topologies for evaluation of the solutions are given as well.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 5 May 2027.¶
Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
This draft is to facilitate the understanding of the data plane enhancement solutions, which are suggested currently or can be suggested in the future, for deterministic networking.¶
An enhancement solution can be a combination of multiple data plane functional entities, such as regulators, queues, and schedulers. A solution can also include functional entities across network nodes, e.g. traffic enforcement or regulation functions at the edge. A regulator, or equivalently a shaper, is defined as a functional entity that makes the arrival process of a flow conform to a predefined process. A packet scheduler, or simply a scheduler, is a functional entity that determines when a packet is transmitted.¶
The term taxonomy refers to a systematic method for classifying the sets of various solutions, how they are related, using certain criteria. A criterion is a principle or standard by which a solution can be judged or decided to be put into a certain category. A category is a subset of solutions classified into a single group using a criterion. This draft provides several criteria for classifying data plane solutions. These criteria are orthogonal to each other.¶
Examples of the categories are listed, along with reasons where necessary. Strengths and limitations of the categories are described.¶
Suitability of the solutions for various services of deterministic networking are also mentioned. The services can be classified according to the flow characteristics and the performance requirements. For example, Requirements for Reliable Wireless Industrial Services [I-D.ietf-detnet-raw-industrial-req] characterizes the services by the latency bound, the burst size, the burst transmission period, the number of nodes, etc. This document adopts this characterization rule, and classifies the services into one of tight/loose latency, large/small burst, periodic/non-periodic, and large/small scale services. For example, the display information service defined in Section 4.4. of [I-D.ietf-detnet-raw-industrial-req] is a loose latency, large burst, non-periodic, and small scale service.¶
The taxonomy described in this draft can be applied for the solutions of other standardization bodies, such as IEEE 802.1 TSN TG.¶
In this draft, the candidate solutions currently being proposed in DetNet WG are simply listed without any descriptions. The details of the solutions are intentionally omitted. Interesting readers may refer to the corresponding drafts. When necessary, the solutions from IEEE TSN TG or existing popular ones are used as examples to better understand the taxonomy and the derived categories.¶
The solutions raised in the DetNet WG are not entirely new concepts but rather variations of existing solutions. These deliberate approaches aim to address the scalability requirements defined in [I-D.ietf-detnet-scaling-requirements] while ensuring a degree of continuity and compatibility with the current practices. The taxonomy in this draft reflects how new solutions extend existing ones to address scalability challenges.¶
For instance, Cycle Specified Queuing and Forwarding (CSQF) [I-D.chen-detnet-sr-based-bounded-latency], Tagged Cyclic Queuing and Forwarding (TCQF) [I-D.eckert-detnet-tcqf], IEEE 802.1Qdv Enhanced CQF (ECQF) are enhancements built upon the foundation of Cyclic Queuing and Forwarding (CQF). Similarly, Work Conserving Stateless Core Fair Queuing (C-SCORE) [I-D.joung-detnet-stateless-fair-queuing] is an extension of Fair Queuing (FQ). Timeslot Queuing and Forwarding (TQF) [I-D.peng-detnet-packet-timeslot-mechanism] is an extension of IEEE 802.1Qbv, also known as Time Aware Shaper (TAS). Earliest Deadline First (EDF) [I-D.peng-detnet-deadline-based-forwarding] proposed to DetNet WG is a variation of the well-known solution that has the same name. Other well-known solutions that could provide bounded latency are also covered, for example Deficit Round Robin (DRR) and Asynchronous Traffic Shaping (ATS) [IEEE_802.1Qcr].¶
The suitable categories for achieving the goal of deterministic networking are defined. A suitable category is defined as a category in which solutions can be effective and efficient at realizing the goals of deterministic networking. The criteria defined in Section 5 are used for the decision of the suitable categories.¶
Reference topologies (RTs) are also listed in this document. An RT consists of a network topology and flows' characteristics. The RTs listed in this document cover various topologies such as ring, mesh, hybrid etc. The purpose of listing the reference topologies (RTs) is to evaluate the dataplane solutions how they perform in real networks, in terms of E2E latency bound and jitter bound.¶
The following terms are used in the context of deterministic networking in this document:¶
solution¶
A set of data plane functional entities, such as queue, scheduler, and regulator; which can guarantee a certain level of latency and jitter performance to a flow¶
regulator¶
A functional entity that makes the arrival process of a flow conform to a predefined process. Note that the term regulator is equivalent to shaper.¶
scheduler¶
A functional entity that determines when a packet is transmitted. Note that the term scheduler is equivalent to packet scheduler.¶
taxonomy¶
A systematic method by which a solution is put into a category¶
category¶
A set of solutions that is put together by one or more criteria¶
criterion¶
A principle or standard by which a solution can be judged or decided to be put into a certain category¶
level of category¶
The number of criteria that are used to determine the category¶
suitable category¶
A category within which solutions can be suitable for deterministic networking¶
time bound¶
The earliest or latest transmission completion time explicitly specified for a packet¶
DetNet: Deterministic Networking¶
CQF: Cyclic Queuing and Forwarding¶
TAS: Time Aware Shaper¶
FQ: Fair Queuing¶
DRR: Deficit Round Robin¶
ATS: Asynchronous Traffic Shaping¶
RT: Reference Topology¶
E2E: End to End¶
GCL: Gate Control List¶
IR: Interleaved Regulator¶
FIFO: First In First Out¶
RSpec: Requested Specifications¶
TSpec: Traffic Specifications¶
IVN: In-Vehicle Network¶
CC: Command and Control¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
Taxonomy based on the performance, such as E2E latency bounds and jitter bounds, is helpful to understand the solutions. The performance should be exhibited as a mathematical expression with the network and traffic parameters.¶
One possible criterion would be based on the per hop dominant factor for the latency bound. The dominant factor is defined as the largest sum term in the expression, when the network and traffic conditions are the worst. The worst condition typically means high network utilization, large packet and burst sizes, and large number of hops. Any existing solution can be put into one of three categories.¶
Category 1 (Max Packet Length/Service Rate): FQ and its variations like C-SCORE fall into this category, where the latency bound is primarily influenced by the ratio of a flow's maximum packet size to its allocated service rate. This category emphasizes individual flow isolation. The consequence is that the variation of E2E latency bound for a flow is minimized with the other flows' join and leave. Therefore, this category performs well with dynamic flows. This category also fits well to services with large bursts, since the burst sizes of flows are not the dominant factor of the latency bound.¶
Category 2 (Sum of Max Packet Lengths/Capacity): Solutions like DRR belong here, where the dominant factor is the sum of maximum packet lengths of all DetNet flows over the total allocated bandwidth. This category typically has less implementation complexity than Category 1 but can impact individual flow isolation. The other flows' max packet lengths affect the latency bound, which can be altered as flows join and leave.¶
Category 3 (Sum of Max Burst Sizes/Capacity): CQF, TAS, their variations (including CSQF, TCQF, ECQF, TQF), and EDF fall into this category. The key influence on latency here is the total burst sizes of all DetNet flows relative to the network capacity. This category prioritizes bounded latency guarantees but may require tighter burst control mechanisms. Once the burst is controlled, for example by an extremely strict regulation, into a packet length level, then this category may be indistinguishable with Category 2. This category fits well to the services for static flows with small bursts.¶
As an example, assuming the capacities and maximum packet lengths are identical in all the links along the path of a flow under observation, the E2E latency bound of the flow by FQ is given as the following [STILIADIS-LRS].¶
(B-L)/r + H(Lh/Rh + L/r), (1)¶
where B, L, and r are the maximum burst size of, the maximum packet length of, and the allocated service rate to the flow, respectively; H is the number of hops; Lh and Rh are the maximum packet length and the capacity of all the links.¶
In this example, the term (Lh/Rh + L/r) can be seen as the per hop latency, because the max burst size, B, appears only once. The service rate of a flow, r, is likely to be much less than the link capacity, Rh, while the maximum lengths L and Lh would not differ too much. Therefore, the dominant factor here is L/r.¶
The dominant factor determines the level of flow isolation, as well as the level of E2E latency bound value.¶
Taxonomy based on the functional characteristics is the key to understanding the solutions. The criteria listed in this section are orthogonal to each other, if not stated explicitly.¶
If a solution has a set of consecutive time slots that is repeated periodically, and the time slots are assigned to packets based on a predefined rule, then the solution is periodic. Otherwise, the solution is non-periodic.¶
The set of consecutive time slots are called a period. Note that here we use the term period to avoid confusion with the term cycle used in CQF, which is equivalent to the time slot defined in this draft.¶
According to the above definition, IEEE 802.1Qbv TAS is a periodic solution. A finite Gate Control List (GCL) of TAS contains multiple gate control entries. Each entry represents a time slot with an assigned set of flows. A set of consecutive time slots forming a GCL is repeated periodically. Time slots can be overlapped with each other, as in ECQF.¶
TAS based solutions and CQF based solutions belong to periodic solutions, for example CSQF, TCQF, ECQF, TQF and so on.¶
Note that a periodic solution requires network synchronization. A further classification based on the level of required synchronization would also be possible. A solution could be classified as either phase synchronous, frequency synchronous, or asynchronous. However, this document does not specify this classification based on the synchronization, since only a frequency synchronization is possible in large scale networks.¶
Periodic solutions may fit well to periodic services, and vice versa.¶
This document categorizes data plane solutions based on the granularity of their traffic control target, which refers to the size and specificity of the traffic entity they handle. Three granularity levels exist.¶
Flow level: Each packet is controlled based on its specific flow, which can be identified usually by the 5-tuple. Examples include FQ and its variations such as C-SCORE, which offer precise service differentiation but require potentially complex implementation.¶
Flow aggregate level: Flows are grouped by shared characteristics like traffic specification, service requirement, or routing path. This coarser level simplifies control but may offer less precise differentiation. Examples include interleaved regulators in ATS.¶
Class level: Flows are further grouped by similar service requirements, regardless of specific path or traffic details. This coarsest level simplifies control and accommodates traffic fluctuations but provides the least individual flow differentiation. Typically, time or time based information could be used for classification, such as in EDF, CQF and its variations.¶
For each level solution, packets within the same traffic entity receive the same treatment. For example, if a solution is flow aggregate level, then the packets within the same flow aggregate are treated identically, regardless of the flows they belong to.¶
There are cases in which a single solution consists of multiple functional entities that treat packets according to multiple traffic entities of different granularities. In such cases, it is defined that the functional entity with the coarsest granularity is dominant, thus the whole solution belongs to the coarsest granularity category.¶
For example, ATS consists of interleaved regulators (IRs) and a strict priority scheduler. An IR has a queue dedicated to a flow aggregate having the same class and the same input port. The regulation function itself is based on a flow. According to the definition above, IR is a flow aggregate level solution. On the other hand, the strict priority scheduler in ATS is class-based. Therefore, ATS as a whole is class level.¶
A finer granularity level solution has a benefit of a more accurate service differentiation among flows. Its limit is the larger implementation complexity. It fits to services with flows having various independent latency bound values.¶
Periodic solutions can further be categorized based on the traffic granularity. A time slot can be assigned per flow, per flow aggregate, or per class.¶
Note that TAS in 802.1Qbv is a scheduling mechanism defined in an output port with eight queues. The queues are controlled by GCL and its gate control entries. Each queue can serve a class. In an entry, queues can be either open or closed. Thus, TAS can be seen as a class level solution. However, in many cases TAS is understood as a scheduling mechanism, where the number of queues are not limited to 8. There could be a natural extension, such as TQF, which enables Qbv to allocate one queue to each flow or a flow aggregate.¶
Finer granularity periodic solutions have more strengths in jitter control. They also fit services with many periodic flows of independent period values.¶
A solution can schedule a packet such that its transmission is completed within a specified interval of time. That interval can be either bounded, left-bounded, right-bounded, or unbounded. A left-bounded interval has a minimum time bound only. A right-bounded interval has a maximum time bound only. A bounded interval has both minimum and maximum time bounds. An unbounded interval does not have any finite bound.¶
Similarly, a solution can be categorized by this interval of allowed transmission completion time.¶
Bounded: A packet's transmission completion is allowed after a minimum time bound and before a maximum time bound.¶
Left-bounded: A packet's transmission completion is allowed after a minimum time bound.¶
Right-bounded: A packet's transmission completion is allowed before a maximum time bound.¶
Unbounded: A packet's transmission completion is allowed at any moment.¶
Note that unbounded solutions can also guarantee an E2E and a nodal latency bounds. They just do not have an explicitly specified nodal time bound.¶
FIFO, round robin schedulers, FQ and its variations like C-SCORE are examples of the unbounded solutions. TAS and CQF and their variations are bounded solutions, for example CSQF, TCQF, ECQF, and TQF. ATS and their variations with traffic shapers are left-bounded solutions. EDF can be operated either as an unbounded or a right-bounded solution, depending on whether the deadline is strictly kept or not.¶
Unbounded solutions have strengths in terms of average delay. They usually show smaller observed maximum latencies than the theoretical latency bound expressions suggest. They also benefit from the statistical multiplexing gain without any wasted capacity, thus more room for best effort traffic.¶
Bounded solutions have strengths to avoid burst accumulation and are also beneficial for jitter control. The burst size in a network of a flow can be kept similar or the same with the initial burst size. Therefore, the buffer size necessary typically is less than those in unbounded solutions.¶
The packet service order within a single flow must be maintained. Data plane solutions decide the packet service order from different flows using various decision rules, categorized as follows.¶
Rate-based: Packets are ordered based on the allocated service rate of their flows or flow aggregates. Examples include FQ and its variations like C-SCORE, and DRR.¶
Arrival-based: Packets are ordered based on their arrival to a node. FIFO is an example.¶
Priority-based: Packets are ordered based on an assigned priority, other than the service rate or the arrival.¶
A rule for the service order may also be constructed with a combination of these characteristics.¶
According to its primary service order decision rule, a solution can be categorized into either rate-based, time-based, arrival-based, or priority-based. Any solution can also use the packet arrival time as a secondary decision rule.¶
The strict priority scheduler uses primarily the priority of a packet according to its class. It also uses the arrival times among packets of the same priority. In this case it is categorized as priority-based.¶
ATS has IRs and a strict priority scheduler. The service order among packets at an IR is arrival-based. The order among packets from different input ports are decided at the strict priority scheduler. Thus, ATS is priority-based.¶
Rate-based solutions have a simple admission condition check process that is dependent only on the service rates of flows. They benefit from the "pay burst only once" property, by which the maximum burst size of a flow contributes to the E2E latency bound only once, without being multiplied by the hop count. Rate-based solutions typically fit well to services with large burst and large scale services, without a need for overprovisioning, or additional burst control mechanisms.¶
Priority-based and arrival-based solutions benefit from the implementation simplicity. The latency and jitter differentiation among flows can be coarse, however. The services with loose latency, small burst, and non-periodic services may fit this category.¶
This section specifies the suitable categories of solutions for deterministic networking. A category is a set of solutions that are put together by one or more criteria. A suitable category is a category in which solutions can be effective and efficient at realizing the goals of deterministic networking.¶
The criteria defined in Section 5 are used for the decision of the suitable categories. The seven suitable categories are defined, with the logic stated in the following.¶
There are four criteria in the functional taxonomy. They are the time bound, the service order, the periodicity, and the traffic granularity.¶
One of the most significant criteria for defining the characteristics of a solution is the time bound. There are four categories under this criterion: unbounded, left-bounded, right-bounded, and bounded.¶
For solutions in the right-bounded category, a packet has only a maximum time bound. In this category, the maximum bounds of packets directly decide the scheduling, or equivalently the service order of packets. No other criteria is important. No more categorization is necessary for this category. If a proper mechanism guarantees that a maximum time bound can be kept for a packet, then a solution in this category can guarantee an E2E latency bound. The category of right-bounded solutions is a suitable category.¶
Similarly, the bounded solutions can also guarantee E2E latency upper and lower bounds. Thus, jitter is naturally reduced. However, the existence of mechanisms to keep the time bounds for packets should be looked into.¶
The bounded solutions can be better described by the criterion of periodicity. The service order within an interval between time bounds is not important.¶
Based on the criterion of periodicity, the bounded solutions category can be further divided into the periodic bounded and the non-periodic bounded.¶
Periodic bounded solutions define a set of time slots, which periodically is repeated. Flows or flow aggregates can be scheduled into these slots. The slot scheduling should be executed over all the nodes in a path, and requires collaboration among nodes. Usually this process can be performed at the central control entity.¶
Periodic bounded solutions can be further categorized by the traffic granularity. Either flow level or class level subcategories is suitable. Note that in Section 6, the flow aggregate level category is merged into the flow level category, for simplicity. The flow level periodic bounded category is a suitable category.¶
As stated in Section 6.2, the class level periodic bounded category is also a suitable category.¶
Non-perodic bounded solutions, as was the case with the right-bounded solutions, require a mechanism to guarantee the minimum and maximum bounds to be kept for a packet. Provided that this mechanism is present, the non-periodic bounded category is suitable. Either flow level or class level subcategories of this category is suitable. The flow level non-periodic bounded category is a suitable category.¶
As stated in Section 6.4, the class level non-periodic bounded category is also a suitable category.¶
Solutions without maximum time bounds cannot be described by the periodicity. They include the solutions of the unbounded and the left-bounded categories.¶
These two categories can be further categorized by the service order, into rate based, priority based, and arrival based. However, priority based (e.g. the strict priority scheduler) and arrival based solutions (e.g. FIFO scheduler) cannot meet tight latency bounds in large scale networks. Rate based solutions only are suitable. Therefore, the level two categories, rate based unbounded and rate based left-bounded are suitable for DetNet.¶
The above level two categories can be further divided with the criterion of traffic granularity. However, the class level rate based solutions are not suitable because the interference due to the burst intermix within a class is troublesome to flow with small bursts. Therefore, the flow level rate based unbounded category is a suitable category.¶
As with the same reasoning stated in Section 6.6, the flow level rate based left-bounded category is also suitable.¶
The purpose of defining reference topologies (RTs) is to evaluate the performance of data plane solutions in real networks, particularly in terms of E2E latency and jitter bounds. For a given data plane solution and its implementation parameters, the E2E latency and jitter bounds of each flow are required to be calculated exactly.¶
In addition, statistical performance results, such as the average E2E latency or the E2E latency distribution, are recommended. Scalability in both the data plane and the control plane is also recommended to be demonstrated. Other recommended performance metrics include the implementation complexity of the data plane solution and the complexity of the admission control and slot scheduling procedures in an environment where flows dynamically join and leave.¶
An RT consists of a network topology and a set of flow characteristics. The network topology specifies the locations of source, destination, and relay nodes and their interconnections. A flow characteristic consists of its path, requested specification (RSpec), and traffic specification (TSpec). The RSpec includes the E2E latency and jitter bounds. The TSpec includes the maximum burst size and average rate, assuming a token-bucket shaping model. Alternatively, traffic can be specified by its period, phase, and maximum burst size. In this case, the maximum burst is assumed to be transmitted at a fixed phase within each period.¶
From this information, other parameters, such as the network diameter and maximum network utilization, can be derived.¶
The RTs defined in this document cover various topologies, including ring, mesh, and hybrid topologies. Some aspects of the RTs are derived from use cases associated with current or expected network deployments.¶
The RTs can also be used to determine whether a data plane solution can address scalability issues, such as those specified in [I-D.ietf-detnet-scaling-requirements]. The network diameter and utilization levels in the RTs are selected to examine scalability.¶
A major concern in deterministic networking is the worst case latency, or equivalently, the latency bound. An RT specifies not only a network topology but also a traffic scenario, which defines the number of flows that are distributed over the network and the traffic specification of each flow. A latency bound can be derived for every RT. Such a bound is not necessarily the highest possible for the network topology, since higher bounds can be produced by adding flows to those included in the traffic scenario of the RT.¶
A dataplane solution can either be prepared for the worst case scenario from the outset or adapt dynamically as flows join and leave. For a flexible solution that adapts to changing traffic conditions, the latency bounds with more flows can also be specified.¶
A reference network topology, the grid, is shown in Figure 1. It represents a general network of partial mesh or grid topology, without considering a specific use case. A partial mesh is a common topology that can be seen in many real deployments, including datacenter networks.¶
Src 1 Src 2 Src 3
| | |
+------+ +------+ +------+
Dst 1-|Node 1|<--|Node 2|-->|Node 3|-Dst 4
+------+ +------+ +------+
| ^ |
V | V
+------+ +------+ +------+
Dst 2-|Node 4|-->|Node 5|<--|Node 6|-Dst 5
+------+ +------+ +------+
^ | ^
| V |
+------+ +------+ +------+
Dst 3-|Node 7|<--|Node 8|-->|Node 9|-Dst 6
+------+ +------+ +------+
| | |
Src 4 Src 5 Src 6
In Figure 1, arrowed links indicate the directions to follow for any traffic route. For example, from Node 2, only Node 1 and Node 3 are the next possible route.¶
The capacity of all the links in the topology is 1Gbps. While real deployments easily exceed 1Gbps link capacity, this RT represents a rather scaled down example in terms of the link capacity and the number of nodes.¶
In-vehicle network (IVN) is an example network which demands deterministic networking. [Buffered_Network] summarizes the flows that require deterministic networking services in IVNs as in Table 1.¶
| Flow type | Maximum burst size | Maximum Packet length | Arrival rate | Required maximum latency |
|---|---|---|---|---|
| Audio | 2Kbit | 2Kbit | 1.6Mbps | 5ms |
| Video | 360Kbit | 12Kbit | 11Mbps | 10ms |
| Command and Control | 2.4Kbit | 2.4Kbit | 480Kbps | 5ms |
To simplify performance analysis, the flows in Table 1 are abstracted as shown in Table 2. Flows of the same type are aggregated into a single flow. For example, ten command and control (CC) flows that share the same E2E path can be considered to be a single type C flow as in Table 2. The maximum burst size and the average rate of a type C flow are about ten times those of one CC flow, in this case. Each flow type has specific destination nodes. For example, type A flows are destined only to destination 1 or 6.¶
| Flow type | Maximum burst size | Maximum Packet length | Arrival rate | Required maximum latency | Destination in Figure 1 |
|---|---|---|---|---|---|
| A | 20Kbit | 2Kbit | 20Mbps | 5ms | Dst 1, Dst 6 |
| B | 4000Kbit | 10Kbit | 100Mbps | 10ms | Dst 3, Dst 4 |
| C | 20Kbit | 2Kbit | 5Mbps | 5ms | Dst 2, Dst 5 |
A source creates one flow to each destination for a total of 6 flows. 36 flows are created throughout the network. Table 2 describes characteristics of the three different flow types used in the RT.¶
The links are unidirectional as specified in Figure 1. All the flows must follow the direction of the arrows in every link. For example, a flow from source 1 to destination 5 follows the path of Src1-1-4-5-2-3-6-Dst5.¶
If there are more than one possible route to the destination, then the shortest path is selected. If there are more than one shortest path, then the following rules are applied.¶
Note that there are at most two outgoing links from a node to select. If both choices give the same distance to the destination, the node closer to the destination is selected as the next node. For example, from Src 5 to Dst 4, the selection from node 8 is to node 9, not to node 7, because node 9 is closer to Dst 4. When the above rule does not break the tie, i.e. the possible next nodes are within the same distance to the destination, then the node closer to the source is selected as the next node. For example, from Src 4 to Dst 5, the selection from node 5 is to node 8, not to node 2, because node 8 is closer to Src 4.¶
The above rules generate a unique route for every source and destination pair.¶
The reason for introducing unidirectional links is to make the network diameter large. With this configuration, the network diameter is 7 hops, which is relatively large considering a small number of nodes.¶
The destination of a flow decides the flow type. For example, all the flows destined to node 1 are of type A. There are 6 flows for each destination. There are 12 flows for each type. The flows with longest paths within the same flow type are of interest. Table 3 shows the path of the flows with longest paths for each flow type. For all the flow types, the number of hops in the longest paths is the same. The utilizations may differ for different links.¶
| Flow type | Longest path |
|---|---|
| A | Src5-8-7-4-5-2-1-Dst1 |
| A | Src2-2-3-6-5-8-9-Dst6 |
| B | Src5-8-9-6-5-2-3-Dst4 |
| B | Src2-2-1-4-5-8-7-Dst3 |
| C | Src3-3-6-5-2-1-4-Dst2 |
| C | Src6-9-6-5-8-7-4-Dst2 |
| C | Src1-1-4-5-2-3-6-Dst5 |
| C | Src4-7-4-5-8-9-6-Dst5 |
Network utilization is defined as the maximum link utilization over all the links. The RT achieves network utilization is 53%. The bottleneck links, e.g. the link 2-3, have one type A flow, five type B flows, and two type C flows. The scalability of a solution can be properly evaluated with this topology.¶
Another RT, the hierarchical ring-mesh, is illustrated over Figure 2 and Figure 3. The core network of the RT is depicted in Figure 2. A backbone node in the core network can be connected to one or two leaf network groups. A leaf network group consists of multiple leaf networks. The number of leaf networks in a group is a design parameter, but is recommended to be from 2 to 10. A leaf network of the RT is depicted in Figure 3.¶
This RT represents a wide area network, e.g. a state wide backbone network, having multiple subsidiary regional networks. The core network is a partial mesh with unidirectional links. Some of the nodes in the core network have the links to the leaf networks. A leaf network is a unidirectional ring with eight nodes. One of the nodes in the leaf network is linked to the core network.¶
Leaf NW Leaf NW Leaf NW
group 0 group 1 group 2
| | |
Leaf NW +------+ +------+ +------+ Leaf NW
group 11-|Node 1|<--|Node 2|-->|Node 3|-group 3
+------+ +------+ +------+
| ^ |
V | V
Leaf NW +------+ +------+ +------+ Leaf NW
group 10-|Node 4|-->|Node 5|<--|Node 6|-group 4
+------+ +------+ +------+
^ | ^
| V |
Leaf NW +------+ +------+ +------+ Leaf NW
group 9 -|Node 7|<--|Node 8|-->|Node 9|-group 5
+------+ +------+ +------+
| | |
Leaf NW Leaf NW Leaf NW
group 8 group 7 group 6
The link capacity within the core network is another flexible design parameter. It can be set such that the maximum link utilization is 50%~80%. The capacity of connecting link between a leaf network and the core network is 1Gbps.¶
In Figure 2, arrowed links indicate the directions to follow for any traffic route. For example, from the leaf network group 0 to the group 6, the nodes 1, 4, 5, 8, 9 have to be visited.¶
To/From
Core NW
|
+------+ +------+ +------+
Src/Dest-|Node 7|-->|Node 0|-->|Node 1|-Src/Dest
+------+ +------+ +------+
^ |
| V
+------+ +------+
Src/Dest-|Node 6| |Node 2|-Src/Dest
+------+ +------+
^ |
| V
+------+ +------+ +------+
Src/Dest-|Node 5|<--|Node 4|<--|Node 3|-Src/Dest
+------+ +------+ +------+
|
Src/Dest
The capacity of all the links in the leaf network is 1Gbps.¶
In Figure 3, arrowed links indicate the directions to follow for any traffic route. For example, for a flow to travel from the node 1 to the core network, every node in the leaf network should be visited.¶
In this RT, the flows specified in Table 1 are used again. There are three types of flows, which are the audio, video, and command & control. Let's call these flows A, V, and C.¶
According to [Buffered_Network], the proportion of these three types in in-vehicle networks are such that A:V:C = 7:7:32. This RT lets seven A, seven V, thirty two C flows share the same path from source to destination. Let's define these flows collectively as a flow set. A flow set's total arrival rate is 103.56 Mbps.¶
Every flow in a leaf network travels from node i to node (i+7)mod8. Every flow travels 7 hops before leaving the leaf network. Exactly a single flow set enters into and leaves from the network by a node in the leaf network. The node 0, which is the connecting node to the core network, also has one flow set coming in from and another flow set going out to the core network.¶
The links in a leaf network are unidirectional as indicated by the arrows in Figure 3. All the flows follow the direction of the arrows in every link.¶
All the links in Figure 3 are passed by seven flow sets. This is the maximum number of flow sets in a link, over all the links. Any flow from a leaf network travels seven hops before leaving. This is the maximum number of hops in the leaf network.¶
In the core network, each leaf network group i sends n flow sets to the leaf network group (i+6)mod12, where n is the number of leaf networks in a leaf network group. These n flow sets are distributed to the destination leaf networks within the group evenly.¶
The links in the core network are unidirectional as indicated by the arrows in Figure 2. All the flows follow the direction of the arrows in every link.¶
At a crossroad, where a flow can choose one of two possible paths, the flow always turns left. For example, a flow from the leaf network group 1 travels to nodes 2, 3, 6, 5, 8, and then the leaf network group 7.¶
The link between the nodes 4 and 5 in Figure 2 are passed by (n times six) flow sets. This is the maximum number of flow sets in a link, over all the links. After joining the core network, a flow from the leaf network group 0 travels four hops before leaving. This is the maximum number of hops in the core network.¶
The maximum hop count in the leaf and core networks are 7 and 4, respectively. However, there are one hop each to/from the core network. The maximum hop count in this RT is 20.¶
Network utilization is defined as the maximum link utilization over all the links. A leaf network has the network utilization of (seven times 103.56 Mbps) over 1 Gbps, which is around 72.5%. The utilization of the core network can be of any value, because the link capacity in the core network can be chosen to be any value greater than the (n times six times 103.56 Mbps), where n is the number of the leaf networks in a leaf network group. For example, if n equals to 2, then the link capacity is required to be larger than 1242.72 Mbps. If the link capacity is 1.5 Gbps, then the utilization is around 82.85%.¶
Another reference topology (RT), the hybrid leaf-spine and tree (HLST), is illustrated in Figure 4 and Figure 5.¶
+--+ +--+ +--+ +--+ +--+
LAN 1---|G1|---| |---| |---| |---|CA|---a1*, a2*, a3*, a4*, hAC, hAD
+--+ |M1| |P1| |M3| +--+
+--+ | | | | | | +--+
LAN 2---|G2|---| |\ /| |\ /| |---|CB|---b1*, b2*, b3*, b4*, hBC, hBD
+--+ +--+ X +--+ X +--+ +--+
+--+ +--+/ \+--+/ \+--+ +--+
LAN 3---|G3|---| | | | | |---|CC|---c1*, c2*, c3*, c4*, hCA, hCB
+--+ |M2| |P2| |M4| +--+
+--+ | | | | | | +--+
LAN 4---|G4|---| |---| |---| |---|CD|---d1*, d2*, d3*, d4*, hDA, hDB
+--+ +--+ +--+ +--+ +--+
+---+ +---+ +---+ +---+ +--+ |S11|-----|S12|-----|S13|-----|S14|-----|G1|---Core domain +---+ +---+ +---+ +---+ +--+ | | | | +---+ +---+ +---+ +---+ |A11|-----|A12|-----|A13|-----|A14| +---+ +---+ +---+ +---+ | | | | a11,b11 a12,b12 a13,b13 a14,b14 c11,d11 c12,d12 c13,d13 c14,d14 r11,f11 r12,f12 r13,f13 r14,f14
HLST is a multi-domain topology in which multiple local area networks (LANs) with traditional tree topology are connected to a leaf-spine core network. A leaf-spine network is a two-layer Clos network. The RT includes four LAN segments in the edge domain and four leaf switches (M), two spine switches (P), and four controller servers (C) in the core domain.¶
Each LAN segment is connected to the core through a gateway G attached to a leaf switch. In the core, each leaf switch is connected to both spine switches. Therefore, traffic between two different leaf switches can traverse either of the two spine switches. The controller servers C host the virtualized controllers for the different types of devices attached to the LAN switches and aggregate their flows by type. For example, CA hosts the controllers of all type-a devices (axy) and therefore terminates the flows exchanged with these devices. The edge domain consists of four distinct hierarchical LAN trees comprising access and distribution layers. Each LAN contains four aggregators (Axy), as illustrated for one LAN segment in Figure 5.¶
In Axy, the x index identifies the LAN segment and the y index identifies the aggregator within the LAN segment.¶
Each aggregator collects the flows to and from its attached end devices and is connected to six device types. Devices axy, bxy, cxy, and dxy exchange data with their corresponding remote controllers in the core. Devices fxy and rxy exchange data with devices attached to adjacent aggregators within the same LAN. An fxy device generates a forward flow destined for the next aggregator, whereas an rxy device generates a reverse flow destined for the previous aggregator (the sequence of aggregators wraps around so that the forward flow of aggregator Ax4 ends at Ax1 and the reverse flow of aggregator Ax1 ends at Ax4). These flows remain within the LAN segment and therefore do not traverse the gateway G. Consequently, G carries only inter-domain traffic between the edge and core domains.¶
The complete connectivity between the aggregators, intermediate LAN switches, and gateway G is shown in Figure 5. Similarly, the mapping of the four LAN segments to the four leaf switches and of the controller servers to the leaf switches is shown in Figure 4.¶
The symbols used in Figure 4 and Figure 5 are described in Table 4. All links are bidirectional. Upstream and downstream inter-domain flows are mutually isolated and never share link resources. Thus, utilization and admission control are evaluated independently for the two directions. This RT represents an enterprise network. For example, it can connect campus networks or legacy office LANs, represented by the trees, to a corporate data center core, represented by the leaf-spine network, that hosts an AI inference application such as a corporate chatbot or data-analytics service. The trees provide user access, while the leaf-spine core provides low and predictable latency between the users and the application servers. Another use case is an industrial automation network, in which the edge domain represents physical production lines and the core servers host virtualized controllers.¶
| Symbol | Description |
|---|---|
| G1 | Gateway to/from LAN 1 |
| M | Leaf switch |
| P | Spine switch |
| CA | Central server A, which hosts virtual controllers |
| a1* | Controller that exchanges flows with a11 ~ a14 |
| hAC | Controller that exchanges local flows with hCA |
| S12 | Switch 2 in LAN 1 |
| A12 | Aggregator 2 in LAN 1 |
| a11 | Device linked to A11 which exchanges flows with a1* |
| r11 | Device linked to A11 which exchanges local flows in reserve direction, with f14 |
| f11 | Device linked to A11 which exchanges local flows in forward direction, with r12 |
Links between aggregators and LAN switches have a capacity of 2.5 Gbps. All other links have a capacity of 10 Gbps. These rates are lower than those typically found in practical networks. They were chosen to keep the performance evaluation experiments scalable, assuming that the experiments are mostly executed in simulation environments.¶
The characteristics of the flows between devices are summarized in Table 5. Each entry represents a flow between the source and destination endpoints specified in the table. Packet generation at the application endpoints is periodic. The maximum burst size of each flow equals its maximum packet length, i.e., each source emits at most one packet per period.¶
| Flow | Arrival rate | Max packet length | Required max latency |
|---|---|---|---|
| axy--ax* | 100 Mbps | 250 B | 5 ms |
| bxy--bx* | 100 Mbps | 1000 B | 5 ms |
| cxy--cx* | 50 Mbps | 250 B | 5 ms |
| dxy--dx* | 50 Mbps | 1000 B | 5 ms |
| fxy--rx(y%4+1) | 25 Mbps | 1000 B | 5 ms |
| h[AB]--h[CD] | 100 Mbps | 1500 B | 5 ms |
For inter-domain traffic, devices axy, bxy, cxy, and dxy exchange flows with their respective controllers in the core. All instances of a given device type communicate with the controller server associated with that type. For example, all axy flows exchanged with the core terminate at CA.¶
For intra-LAN traffic, fxy and rxy flows are exchanged only between adjacent aggregators and remain entirely within their corresponding LAN segment. They therefore consume capacity on the applicable links within the edge domain but do not consume capacity on the gateway or core links.¶
Flow paths in the edge domain are uniquely determined by the tree topology, which provides a single path between any two nodes. Consequently, no path selection is required within a LAN segment.¶
For an inter-domain flow, the edge domain portion of the path follows the unique tree path between the end device and gateway G. From G, the flow enters the core through the leaf switch to which that gateway is attached and follows the required path toward its destination controller server. For every gateway-controller server pair, two leaf-spine-leaf paths are available, one through each spine switch.¶
For each such flow, one of the two possible leaf-spine-leaf paths is selected to balance link utilization. Once selected, the path of a flow remains fixed for the duration of its lifetime.¶
Intra-LAN fxy and rxy flows follow the unique tree path between their source and destination aggregators and do not enter the core domain.¶
Network utilization is defined as the maximum link utilization over all links. Link utilization is computed independently in each direction because upstream and downstream inter-domain traffic never share link resources. The utilization of the RT is controlled by varying the number of flows of each type while preserving the flow count ratios described below.¶
For example, a 96% utilization of a 2.5 Gbps uplink or downlink between an aggregator and its LAN switch is obtained with 4 axy + 4 bxy + 8 cxy + 8 dxy + 16 fxy + 16 rxy = 2.4 Gbps. Thus, the aggregate traffic rate on the link is 2.4 Gbps, corresponding to 96% of its 2.5 Gbps capacity.¶
Similarly, a 96% utilization of the 10 Gbps link between leaf switch M3 and controller server CA is obtained as follows:¶
The flow types are identical in the upstream and downstream directions. When the total network utilization is varied, the number of flows of each type is scaled while preserving the flow count ratios used above. This preserves the relative traffic composition and simplifies admission control planning across utilization levels.¶
For example, a 48% utilization of a 2.5 Gbps aggregator-to-LAN switch link is obtained by halving the number of flows in the 96% configuration: 2 axy + 2 bxy + 4 cxy + 4 dxy + 8 fxy + 8 rxy = 1.2 Gbps. The resulting aggregate traffic rate is 1.2 Gbps, corresponding to 48% of the link capacity.¶
In an E2E path of a flow, there can be multiple networks, or equivalently technological domains, in which different solutions are deployed. The detailed definition of technological domain is given in [I-D.ietf-detnet-multi-domain-framework], as well as the other types of domains. The technological domain is the major interest of this document, and will be referred to as the domain.¶
It is desirable to serve the flow with the promised service level without significant intervention in the middle. In this regard, it is necessary to consider the level of interoperability between the solutions of the neighboring domains, when selecting a particular solution within a domain.¶
A more interoperable solution can be defined by the degree to which disparate domains can operate without specialized treatments both for data plane and control plane. In Section 8, the interoperability between solutions in different categories are considered.¶
The measure of the interoperability between solutions can be quantified by the following factors:¶
1) Operability: The ease of functional integration of two distinct domains to achieve seamless control plane operations¶
2) Performance: The feasibility of consistently meeting the requested service levels across the entire data plane path¶
These measures will be called the Operability and the Performance, respectively. The high values for these metrics are desirable.¶
The control plane governs the operability, managing complex tasks such as end-to-end admission control and cross-domain network configurations. This aspect is covered in [I-D.ietf-detnet-multi-domain-framework].¶
For Performance, or for meeting the E2E service level, there are intervention methods that can be universally applied to gateways between domains.¶
Flow reshaping can be such an intervention in the data plane, especially between non-periodic solution domains. Note that a flow reshaping, or damping, may require per-flow state maintenance.¶
For domains of periodic bounded solutions, it is required the allocation of per-cycle/per-slot resources and, at a domain boundary, the alignment or mapping of cycles/slots and their phases between adjacent domains. [I-D.ietf-detnet-multi-domain-framework]¶
The E2E latency or jitter bound, which is the critical performance metric in DetNet, can be just the simple sum of the bounds provided by the domains. However, in some solutions, the cooperation between domains can further improve the bounds. For example, rate-based solutions over domains can benefit from the pay-burst-only-once (PBOO) property, thus providing a better latency bound than the sum. These solutions over domains can also benefit by a unified rate-based admission control, thus having a low GC.¶
Table 6 is an exemplary illustration of interoperability measures between solutions that belong to different suitable categories. For example, flow/class level periodic bounded solutions should realign their cycle/slot schedules, which requires significant complexity to be done in real-time. However, once the slot scheduling is completed, the packets in the data plane can pass through smoothly over slots, just like within the same domain. Therefore, the operability is low, while the performance is high.¶
For another example, flow level periodic and rate-based solutions require significantly different admission processes and configuration mechanisms. Their operability is low, since they need independent network configurations as well as the coordination between Domain Controllers. The performance, however, can be medium since they both work at the flow-level.¶
| Class level non-periodic bounded | Flow level rate based left-bounded | |
|---|---|---|
| Flow level periodic bounded | Operability: Low, Performance: High | Operability: Low, Performance: Meduim |
| Flow level rate based unbounded | Operability: Low, Performance: Low | Operability: High, Performance: High |
When interconnecting different data plane solutions, operational planning must consider the trade-offs between performance and management overhead. From a performance perspective, combinations of solutions rated "high" produce explicit end-to-end latency bounds which are smaller or comparable to the simple sums of latency bounds of consisting domains.¶
In comparison, combinations rated "medium" or "low" performance may introduce boundary shaping delays and non-linear jitter accumulation. Interworking Functions at these boundaries should provide adequate buffers sized to accommodate the worst-case incoming burst size plus local alignment delays.¶
From an operability perspective, "high" combinations of solutions may support unified signaling and end-to-end monitoring. In contrast, "medium" and "low" operability combinations may require boundary nodes to map flows, such as mapping explicit flow identifiers to periodic time slots, which impacts the scalability of the solutions. Lower operability metrics may require multi-domain orchestrators to perform sequential admission control across multiple control planes and may also have fragmented performance monitoring. Consequently, combinations scored with "low" operability are best suited for static, long-lived deterministic paths rather than dynamic fast-changing approaches.¶
The recommended network for interoperability test is composed of three cascaded domains. Each domain can use the grid topology defined in Section 7.1. If one solution is placed in the middle, and the other domains at the sides should use a different solution.¶
In this test network, the flows specified in Table 1 can be used. There are three types of flows, which are the audio, video, and command & control.¶
There are currently seven suitable categories. Any combination of these seven categories can be evaluated. The solutions in the same categories are assumed to be highly interoperable.¶
There might be matters that require IANA considerations associated with metadata. If necessary, relevant text will be added in a later version.¶
This section will be described later.¶