Export of RoCEv2 Base Transport Header (BTH) Information Using IP Flow Information Export (IPFIX)

Remote Direct Memory Access (RDMA) is a network technology that allows a computer to read from or write to the memory of another computer directly, without involving the operating system. This zero-copy and kernel-bypass feature greatly reduces CPU overhead and communication latency. InfiniBand and RDMA over Converged Ethernet (RoCE) are two mainstream RDMA implementations that bypass the operating system kernel and achieve zero-copy data transfer. RoCE technology has become a key component of high-performance data center networks, especially in low-latency, high-throughput scenarios such as artificial intelligence training, distributed storage, and financial transactions. RDMA over Converged Ethernet version 2 (RoCEv2) runs on top of UDP (port 4791) and inherits the transport layer protocol of the InfiniBand Architecture (IBA). The IP Flow Information Export (IPFIX) protocol is extensible. The IANA IPFIX Information Elements registry does not define dedicated Information Elements (IEs) for the RoCEv2 Base Transport Header (BTH) fields specified here. This document defines IEs for selected BTH fields to provide transport context for RoCEv2 traffic in IPFIX reports. These IEs do not, by themselves, measure Priority Flow Control (PFC) pauses, queue occupancy, packet loss, or path delay, nor do they pinpoint the congestion root cause. Root-cause analysis is expected to combine these exports with per-priority counters and complementary methods such as In-situ Operations, Administration, and Maintenance (IOAM) and Alternate Marking .

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. The following terms are used as defined in :

IPFIX
IPFIX Information Elements
Template and Options Template
Data Record
Flow Key
Observation Point
Observation Domain
Metering Process
Exporting Process

The following terms are used in this document:

QP: Queue Pair, a communication endpoint in RDMA architecture.
BTH: Base Transport Header, the RDMA transport header containing QP information.
P_Key: Partition Key.
PSN: Packet Sequence Number.

This document defines the following IPFIX IEs:

rocev2BthOpcode
rocev2BthPartitionKey
rocev2BthDestinationQp
rocev2BthPacketSequenceNumber
rocev2BthOctetAtOffset1
rocev2BthOctetAtOffset4
rocev2BthOctetAtOffset8

Byte offsets in this document start at zero from the first octet of the BTH. Within an octet, bit 7 is the most significant bit. Multi-octet BTH fields use network byte order. The three octet-copy IEs preserve the exact observed wire values, including reserved bits. An exporter MUST NOT clear or rewrite a reserved bit before export. The octet-copy IEs keep fields that share one BTH octet together. This preserves the wire image and avoids assigning independent semantics to bits whose meaning depends on the transport service or packet type. defines each IE and its applicability. This document does not define a Source QP IE. Source QP is not a BTH field. For Unreliable Datagram packets, Source QP is in the Datagram Extended Transport Header (DETH) specified by the "Datagram Extended Transport Header Fields" table and "Datagram Extended Transport Header (DETH)" section in Volume 1 of . Export of DETH fields is outside the scope of this document. The IE semantics in this document apply to the RoCEv2 packet format in Annex A17 of . An exporter MUST use configured encapsulation context or other out-of-band knowledge to establish that this format applies; in particular, UDP destination port 4791 and a zero BTH transport-header version are not sufficient by themselves. Export of Multipath Reliable Connection packets specified in Annex 21 is outside the scope of this document.

This document defines a packet-observation profile and an aggregate-Flow profile. Each Template MUST be used in exactly one profile. The IE composition rules below identify that profile. A collector MUST NOT treat a packet-observation Data Record as an aggregate Flow Record. These profiles use the IPFIX Information Model in and the IE and Template guidance in . A packet-observation Template contains one observation-time IE and does not contain aggregate counters. An aggregate-Flow Template contains the start and end times and counters specified in . An exporter MUST NOT use one Template as both profiles.

A packet with UDP destination port 4791 is a RoCEv2 candidate. The port number alone does not prove that the UDP payload is a valid RoCEv2 packet. Before it exports a BTH field, the exporter MUST therefore establish the Annex A17 RoCEv2 encapsulation context as specified above. An exporter MUST locate the UDP header at its configured observation layer and MUST verify that the complete 12-octet BTH is available and that the BTH transport-header version is zero before it exports a BTH field. This document does not define export of BTH fields for another transport-header version. The exporter MUST apply the following rules:

For an IPv4 or IPv6 fragment, the exporter MAY parse a BTH from the first fragment only when that fragment contains the complete UDP header and the complete BTH. Alternatively, it MAY parse the BTH after successful IP reassembly. It MUST NOT infer a BTH from a non-initial fragment.
The exporter MUST process the IPv6 extension-header chain far enough to locate UDP. If it cannot do so, it MUST NOT export a BTH field.
Tunnel decapsulation is an Observation Point function. The exporter MUST therefore state which header layer it observes. It MUST NOT search arbitrary payload bytes for a BTH. If it cannot reach the configured inner packet, it MUST NOT export a BTH field.
If capture truncation, an inconsistent IP or UDP length, a UDP payload shorter than 12 octets, a nonzero BTH version, malformed input, or a local parsing-resource limit prevents access to a complete BTH, the exporter MUST NOT export a BTH field.
A reserved, unassigned, or unsupported opcode does not make an available common BTH invalid. The exporter MUST export the raw opcode value without change, but it MUST NOT infer or parse an opcode-specific extended transport header that it does not support. If a required extended header is absent or truncated, the exporter MAY export the complete common BTH, but it MUST NOT export a value from the unavailable extended header.

IPFIX Data Records cannot omit one field from a Template. Therefore, an exporter MUST use a full-BTH Template only when every BTH field in that Template is available. It MUST use a different Template that omits an unavailable IE. It MUST use separate Templates for the common BTH, for an applicable extended transport header, and for records that do not contain RoCEv2 fields. An exporter MUST NOT insert zero or another in-range value for an unavailable field. This document does not define a parsing-status IE; an implementation that reports a failure reason can use an enterprise-specific status IE in a separate Template.

One Data Record in this profile represents one observed RoCEv2 packet that satisfies the recognition and parsing rules above. It is not an aggregate and has no Flow Keys. The record reports the observation time, Observation Point, direction, IP and UDP context, and the BTH fields that are available under its Template. The IP addresses and UDP ports come from the headers that directly encapsulate the reported BTH at the configured observation layer. Source and destination have their packet meanings. flowDirection gives the packet direction relative to the Observation Point. The following table is the RECOMMENDED full-BTH Template. Separate Templates are required for IPv4 and IPv6 addresses. The length of each 24-bit BTH field is three octets under the reduced-size encoding rules in .

If packet selection is active, the exporter MUST use the extended Packet Report model in Section 6.4.2 of . Each Data Record includes selectionSequenceId. For every selectionSequenceId, the exporter MUST provide the Selection Sequence Report Interpretation, the Selector Report Interpretation for each Selector, the Selection Sequence Statistics Report Interpretation, and the Accuracy Report Interpretation specified in Sections 6.5.1 through 6.5.4 of . These reports identify the Observation Point, give the ordered Selector list and configuration, report statistics such as selectorIdTotalPktsObserved and selectorIdTotalPktsSelected from , and report the accuracy of observed quantities. The reports provide selection context and totals. They do not classify an individual PSN gap as selection or packet loss. When no Selection Process applies, the exporter uses a Template that omits selectionSequenceId. For records from one Observation Point and one Selection Sequence, the exporter MUST preserve observation order. The collector uses the observation time and the IPFIX Message Sequence Number and transport information defined in to identify missing or reordered export data. The following values form an illustrative full-BTH Data Record. The time is shown in text form for readability; its wire encoding follows the registered IE data type.

This subsection defines a unidirectional aggregate-Flow profile. One Data Record represents packets that match the stated Flow Keys during the interval from flowStartMilliseconds through flowEndMilliseconds. The exporter MUST use the following Flow Keys:

source and destination IPv4 or IPv6 addresses;
sourceTransportPort and destinationTransportPort;
protocolIdentifier;
flowDirection;
rocev2BthPartitionKey;
rocev2BthDestinationQp; and
selectionSequenceId when packet selection is active.

The exporter SHOULD identify these Flow Keys with the Flow Keys Options Template specified in . An exporter MAY also use rocev2BthOpcode as a Flow Key when the application needs an opcode-specific aggregate. It MUST NOT include rocev2BthPacketSequenceNumber, rocev2BthOctetAtOffset1, rocev2BthOctetAtOffset4, or rocev2BthOctetAtOffset8 in an aggregate-Flow Template. These values can change for each packet and one value would not describe the aggregate. The RECOMMENDED aggregate Template contains the Flow Keys above, observationPointId, flowStartMilliseconds, flowEndMilliseconds, packetDeltaCount, and octetDeltaCount. The flowDirection IE states whether the Observation Point measured ingress or egress traffic. An exporter uses postPacketDeltaCount and postOctetDeltaCount only when it measures the values after a middlebox function, as specified by the registered definitions of those IEs. Egress direction alone does not give the post-processing semantics. The Observation Domain ID in the IPFIX Message Header scopes the record. The exporter MUST state its active and idle timeout values in Options Data Records described by an Options Template that contains flowActiveTimeout and flowIdleTimeout and is scoped to the applicable Observation Domain or Metering Process. If packet selection is active, the record includes selectionSequenceId as a Flow Key and the exporter supplies the Selection Process Options Data Records required by . Counters then describe the packets represented under that Selection Process; they do not prove the number of packets present on the wire. The following ingress example omits rocev2BthOpcode because the Flow is not split by opcode:

rocev2BthPacketSequenceNumber contains only the raw 24-bit PSN from one packet. A PSN gap does not, by itself, prove packet loss or duplication. A sequence analysis must identify the applicable transport and QP context, quantify packet selection, preserve observation order, handle wrap modulo 2^24, and account for retries, retransmissions, reordering, capture loss, and exporter overload. A derived loss or duplication metric is outside the scope of this document. A Destination QP is local to the destination endpoint. It is not a globally unique Flow or connection identifier, and the number can be reused after a QP is destroyed. A collector correlates it with the Exporting Process identity, Observation Domain ID, observationPointId, flowDirection, outer source and destination IP addresses, UDP source and destination ports, Partition Key, and packet observation time or aggregate Flow interval. The tuple identifies an observation context; it does not create a protocol-level connection identifier. This document defines only a unidirectional aggregate profile. For bidirectional analysis, a collector correlates two unidirectional Data Records, one for each direction, by using the context above and overlapping Flow intervals. Each record keeps its own Partition Key and Destination QP as Flow Keys. The collector MUST NOT infer a Source QP by reversing a Destination QP value. An exporter can instead use the biflow model in . Such export is outside the aggregate profile defined in this document. A biflow definition for these IEs must use the normal and Reverse Information Elements for the Partition Key and Destination QP as Flow Keys, and for directional counters. It must not use one Destination QP value for both directions, and it must not aggregate a PSN or any of the three BTH control octets. An opcode can be included only as a Flow Key.

The exporter needs to parse RoCEv2 BTH information, which may affect the exporter's performance. Implementers SHOULD consider measures to mitigate this impact, such as sampling rate limiting or hardware acceleration.

The Security Considerations for IPFIX apply to this document as well. When exporting RDMA BTH information across security domains, to prevent information security risks caused by the leakage of sensitive network topology details such as QP numbers, it is recommended to implement comprehensive protection measures, such as using the encrypted transmission options of the IPFIX framework (such as the Datagram Transport Layer Security (DTLS) protocol ) to ensure the confidentiality and integrity of data during transmission. To defend against Denial-of-Service (DoS) attacks that may be caused by maliciously crafted RoCEv2 packets and to prevent the exporter from being overloaded by processing a large amount of invalid traffic, it is recommended to adopt certain protection strategies, such as configuring reasonable rate limiting policies to prevent the monitoring system from being overwhelmed by massive data packets and to ensure its stable operation.

This document requests IANA to assign Element IDs from the IPFIX Information Elements registry for the following IEs.

Name:: rocev2BthOpcode
Element ID:: TBD1
Description:: The exact 8-bit value copied from BTH byte offset 0, bits 7 through 0. The value identifies the transport service and operation and determines which extended transport headers follow the BTH. Assigned values and the associated packet layouts are specified by the "OpCode" section and "OpCode field" table in Volume 1 of ; RoCEv2-specific values are specified in Annex A17 of that document. The range is 0 through 255. An exporter exports a reserved, unassigned, or unsupported wire value without change. A collector retains an unrecognized value and does not map it to zero. This IE has packet scope. In an aggregate-Flow Template, it can be present only as a Flow Key. It is absent when the complete BTH is unavailable.
Abstract Data Type:: unsigned8
Data Type Semantics:: identifier
Range:: 0-255
Status:: current
Reference:: [IBTA-SPEC] and [this document]

Name:: rocev2BthPartitionKey
Element ID:: TBD2
Description:: The exact 16-bit Partition Key copied in network byte order from BTH byte offsets 2 and 3. The field and its use are specified by the "Base Transport Header Fields" table and "Base Transport Header (BTH)" section in Volume 1 of . The range is 0 through 0xFFFF. This IE identifies the packet's partition context. In the packet-observation profile it describes one packet. In the aggregate-Flow profile it MUST be a Flow Key. It is absent when the complete BTH is unavailable.
Abstract Data Type:: unsigned16
Data Type Semantics:: identifier
Range:: 0-65535
Status:: current
Reference:: [IBTA-SPEC] and [this document]

Name:: rocev2BthDestinationQp
Element ID:: TBD3
Description:: The exact 24-bit Destination QP value copied in network byte order from BTH byte offsets 5 through 7. The field is specified by the "Base Transport Header Fields" table and "Base Transport Header (BTH)" section in Volume 1 of . The range is 0 through 0xFFFFFF. The unsigned32 type can be encoded in three octets by reduced-size encoding. If encoded in four octets, the high-order octet is zero. A Destination QP is local to the destination endpoint identified by the outer packet context; it is not a global connection identifier. In the aggregate-Flow profile this IE MUST be a Flow Key. It is absent when the complete BTH is unavailable.
Abstract Data Type:: unsigned32
Data Type Semantics:: identifier
Range:: 0-16777215 (0xFFFFFF)
Status:: current
Reference:: [IBTA-SPEC] and [this document]

Name:: rocev2BthPacketSequenceNumber
Element ID:: TBD4
Description:: The exact 24-bit Packet Sequence Number copied in network byte order from BTH byte offsets 9 through 11. The field is specified by the "Base Transport Header Fields" table and "Base Transport Header (BTH)" section in Volume 1 of . The range is 0 through 0xFFFFFF. The unsigned32 type can be encoded in three octets by reduced-size encoding. If encoded in four octets, the high-order octet is zero. This IE has packet scope and MUST NOT be included in the aggregate-Flow profile. It is absent when the complete BTH is unavailable. Its value alone does not identify loss, duplication, reliability, or ordering.
Abstract Data Type:: unsigned32
Data Type Semantics:: default
Range:: 0-16777215 (0xFFFFFF)
Status:: current
Reference:: [IBTA-SPEC] and [this document]

Name:: rocev2BthOctetAtOffset1
Element ID:: TBD5
Description:: The exact value copied from BTH byte offset 1. Bit 7 is Solicited Event, bit 6 is Migration Request, bits 5 and 4 are Pad Count, and bits 3 through 0 are Transport Header Version. These fields are specified by the "Base Transport Header Fields" table and "Base Transport Header (BTH)" section in Volume 1 of . The range is 0 through 255. The exporter preserves all bits without change. This IE has packet scope, is absent when the complete BTH is unavailable, and MUST NOT be included in the aggregate-Flow profile.
Abstract Data Type:: unsigned8
Data Type Semantics:: default
Range:: 0-255
Status:: current
Reference:: [IBTA-SPEC] and [this document]

Name:: rocev2BthOctetAtOffset4
Element ID:: TBD6
Description:: The exact value copied from BTH byte offset 4. Bit 7 is the BTH F bit, bit 6 is the BTH B bit, and bits 5 through 0 are reserved. The base BTH layout is specified by the "Base Transport Header Fields" table and "Base Transport Header (BTH)" section in Volume 1 of . The range is 0 through 255. For a RoCEv2 port, Annex A17 of that document states that the BTH F and B congestion-control bits are unused and ignored. A collector MUST NOT interpret these bits as IP Explicit Congestion Notification (ECN) or as proof of a congestion event. The registered ipClassOfService IE carries the IP octet whose two low-order bits contain ECN. A rocev2BthOpcode value of 0x81 identifies a RoCEv2 Congestion Notification Packet according to Annex A17. The exporter preserves all eight observed bits, including reserved bits, without change. This IE has packet scope, is absent when the complete BTH is unavailable, and MUST NOT be included in the aggregate-Flow profile.
Abstract Data Type:: unsigned8
Data Type Semantics:: default
Range:: 0-255
Status:: current
Reference:: [IBTA-SPEC] and [this document]

Name:: rocev2BthOctetAtOffset8
Element ID:: TBD7
Description:: The exact value copied from BTH byte offset 8. Bit 7 is Acknowledge Request and bits 6 through 0 are reserved. These fields are specified by the "Base Transport Header Fields" table and "Base Transport Header (BTH)" section in Volume 1 of . The range is 0 through 255. The exporter preserves all eight observed bits, including reserved bits, without change. A collector ignores the reserved bits for protocol semantics. This IE has packet scope, is absent when the complete BTH is unavailable, and MUST NOT be included in the aggregate-Flow profile.
Abstract Data Type:: unsigned8
Data Type Semantics:: default
Range:: 0-255
Status:: current
Reference:: [IBTA-SPEC] and [this document]

The authors thank Chongfeng Xie for his review and helpful comments.