| Internet-Draft | Export of RoCEv2 BTH Using IPFIX | October 2026 |
| Lin, et al. | Expires 30 April 2027 | [Page] |
This document defines a new set of IP Flow Information Export (IPFIX) Information Elements (IEs) for exporting Base Transport Header (BTH) information for Remote Direct Memory Access (RDMA) over Converged Ethernet version 2 (RoCEv2) traffic. These extensions enable network monitoring systems to collect and analyze the characteristics of RDMA traffic widely used in high-performance computing, storage, and artificial intelligence applications.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 30 April 2027.¶
Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
Remote Direct Memory Access (RDMA) is a network technology that allows a computer to read from or write to the memory of another computer directly, without involving the operating system. This zero-copy and kernel-bypass feature greatly reduces CPU overhead and communication latency. InfiniBand [IBTA-SPEC] and RDMA over Converged Ethernet (RoCE) are two mainstream RDMA implementations that bypass the operating system kernel and achieve zero-copy data transfer.¶
RoCE technology has become a key component of high-performance data center networks, especially in low-latency, high-throughput scenarios such as artificial intelligence training, distributed storage, and financial transactions. RDMA over Converged Ethernet version 2 (RoCEv2) runs on top of UDP (port 4791) and inherits the transport layer protocol of the InfiniBand Architecture (IBA).¶
The IP Flow Information Export (IPFIX) protocol [RFC7011] is extensible. The IANA IPFIX Information Elements registry [IANA-IPFIX] does not define dedicated Information Elements (IEs) for the RoCEv2 Base Transport Header (BTH) fields specified here. This document defines IEs for selected BTH fields to provide transport context for RoCEv2 traffic in IPFIX reports. These IEs do not, by themselves, measure Priority Flow Control (PFC) pauses, queue occupancy, packet loss, or path delay, nor do they pinpoint the congestion root cause. Root-cause analysis is expected to combine these exports with per-priority counters and complementary methods such as In-situ Operations, Administration, and Maintenance (IOAM) [RFC9197] and Alternate Marking [RFC9341].¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
The following terms are used as defined in [RFC7011]:¶
The following terms are used in this document:¶
This document defines the following IPFIX IEs:¶
Byte offsets in this document start at zero from the first octet of the BTH. Within an octet, bit 7 is the most significant bit. Multi-octet BTH fields use network byte order. The three octet-copy IEs preserve the exact observed wire values, including reserved bits. An exporter MUST NOT clear or rewrite a reserved bit before export.¶
The octet-copy IEs keep fields that share one BTH octet together. This preserves the wire image and avoids assigning independent semantics to bits whose meaning depends on the transport service or packet type. Section 7.1 defines each IE and its applicability.¶
This document does not define a Source QP IE. Source QP is not a BTH field. For Unreliable Datagram packets, Source QP is in the Datagram Extended Transport Header (DETH) specified by the "Datagram Extended Transport Header Fields" table and "Datagram Extended Transport Header (DETH)" section in Volume 1 of [IBTA-SPEC]. Export of DETH fields is outside the scope of this document.¶
The IE semantics in this document apply to the RoCEv2 packet format in Annex A17 of [IBTA-SPEC]. An exporter MUST use configured encapsulation context or other out-of-band knowledge to establish that this format applies; in particular, UDP destination port 4791 and a zero BTH transport-header version are not sufficient by themselves. Export of Multipath Reliable Connection packets specified in Annex 21 is outside the scope of this document.¶
This document defines a packet-observation profile and an aggregate-Flow profile. Each Template MUST be used in exactly one profile. The IE composition rules below identify that profile. A collector MUST NOT treat a packet-observation Data Record as an aggregate Flow Record. These profiles use the IPFIX Information Model in [RFC7012] and the IE and Template guidance in [RFC7013].¶
A packet-observation Template contains one observation-time IE and does not contain aggregate counters. An aggregate-Flow Template contains the start and end times and counters specified in Section 4.3. An exporter MUST NOT use one Template as both profiles.¶
A packet with UDP destination port 4791 is a RoCEv2 candidate. The port number alone does not prove that the UDP payload is a valid RoCEv2 packet. Before it exports a BTH field, the exporter MUST therefore establish the Annex A17 RoCEv2 encapsulation context as specified above. An exporter MUST locate the UDP header at its configured observation layer and MUST verify that the complete 12-octet BTH is available and that the BTH transport-header version is zero before it exports a BTH field. This document does not define export of BTH fields for another transport-header version.¶
The exporter MUST apply the following rules:¶
IPFIX Data Records cannot omit one field from a Template. Therefore, an exporter MUST use a full-BTH Template only when every BTH field in that Template is available. It MUST use a different Template that omits an unavailable IE. It MUST use separate Templates for the common BTH, for an applicable extended transport header, and for records that do not contain RoCEv2 fields. An exporter MUST NOT insert zero or another in-range value for an unavailable field. This document does not define a parsing-status IE; an implementation that reports a failure reason can use an enterprise-specific status IE in a separate Template.¶
One Data Record in this profile represents one observed RoCEv2 packet that satisfies the recognition and parsing rules above. It is not an aggregate and has no Flow Keys. The record reports the observation time, Observation Point, direction, IP and UDP context, and the BTH fields that are available under its Template. The IP addresses and UDP ports come from the headers that directly encapsulate the reported BTH at the configured observation layer. Source and destination have their packet meanings. flowDirection gives the packet direction relative to the Observation Point.¶
The following table is the RECOMMENDED full-BTH Template. Separate Templates are required for IPv4 and IPv6 addresses. The length of each 24-bit BTH field is three octets under the reduced-size encoding rules in [RFC7011].¶
+--------------------------------------+--------+------------------+ | Information Element | Length | Use | +--------------------------------------+--------+------------------+ | observationPointId | 8 | observation | | observationTimeNanoseconds | 8 | observation | | sourceIPv4Address/sourceIPv6Address | 4/16 | outer IP | | destinationIPv4Address/ | | | | destinationIPv6Address | 4/16 | outer IP | | sourceTransportPort | 2 | outer UDP | | destinationTransportPort | 2 | outer UDP | | protocolIdentifier | 1 | value 17 (UDP) | | ipClassOfService | 1 | includes IP ECN | | flowDirection | 1 | observation | | selectionSequenceId | 8 | if selected | | rocev2BthOpcode | 1 | BTH byte 0 | | rocev2BthPartitionKey | 2 | BTH bytes 2-3 | | rocev2BthDestinationQp | 3 | BTH bytes 5-7 | | rocev2BthPacketSequenceNumber | 3 | BTH bytes 9-11 | | rocev2BthOctetAtOffset1 | 1 | BTH byte 1 | | rocev2BthOctetAtOffset4 | 1 | BTH byte 4 | | rocev2BthOctetAtOffset8 | 1 | BTH byte 8 | +--------------------------------------+--------+------------------+¶
If packet selection is active, the exporter MUST use the extended Packet Report model in Section 6.4.2 of [RFC5476]. Each Data Record includes selectionSequenceId. For every selectionSequenceId, the exporter MUST provide the Selection Sequence Report Interpretation, the Selector Report Interpretation for each Selector, the Selection Sequence Statistics Report Interpretation, and the Accuracy Report Interpretation specified in Sections 6.5.1 through 6.5.4 of [RFC5476]. These reports identify the Observation Point, give the ordered Selector list and configuration, report statistics such as selectorIdTotalPktsObserved and selectorIdTotalPktsSelected from [RFC5477], and report the accuracy of observed quantities. The reports provide selection context and totals. They do not classify an individual PSN gap as selection or packet loss.¶
When no Selection Process applies, the exporter uses a Template that omits selectionSequenceId.¶
For records from one Observation Point and one Selection Sequence, the exporter MUST preserve observation order. The collector uses the observation time and the IPFIX Message Sequence Number and transport information defined in [RFC7011] to identify missing or reordered export data.¶
The following values form an illustrative full-BTH Data Record. The time is shown in text form for readability; its wire encoding follows the registered IE data type.¶
observationPointId = 12 observationTimeNanoseconds = 2026-10-26T10:15:30.125Z sourceIPv4Address = 192.0.2.10 destinationIPv4Address = 198.51.100.20 sourceTransportPort = 49152 destinationTransportPort = 4791 protocolIdentifier = 17 ipClassOfService = 0x03 flowDirection = 0 selectionSequenceId = 9001 rocev2BthOpcode = 0x04 rocev2BthPartitionKey = 0xffff rocev2BthDestinationQp = 0x000123 rocev2BthPacketSequenceNumber = 0x00abcd rocev2BthOctetAtOffset1 = 0x00 rocev2BthOctetAtOffset4 = 0x00 rocev2BthOctetAtOffset8 = 0x80¶
This subsection defines a unidirectional aggregate-Flow profile. One Data Record represents packets that match the stated Flow Keys during the interval from flowStartMilliseconds through flowEndMilliseconds. The exporter MUST use the following Flow Keys:¶
The exporter SHOULD identify these Flow Keys with the Flow Keys Options Template specified in [RFC7011].¶
An exporter MAY also use rocev2BthOpcode as a Flow Key when the application needs an opcode-specific aggregate. It MUST NOT include rocev2BthPacketSequenceNumber, rocev2BthOctetAtOffset1, rocev2BthOctetAtOffset4, or rocev2BthOctetAtOffset8 in an aggregate-Flow Template. These values can change for each packet and one value would not describe the aggregate.¶
The RECOMMENDED aggregate Template contains the Flow Keys above, observationPointId, flowStartMilliseconds, flowEndMilliseconds, packetDeltaCount, and octetDeltaCount. The flowDirection IE states whether the Observation Point measured ingress or egress traffic. An exporter uses postPacketDeltaCount and postOctetDeltaCount only when it measures the values after a middlebox function, as specified by the registered definitions of those IEs. Egress direction alone does not give the post-processing semantics. The Observation Domain ID in the IPFIX Message Header scopes the record. The exporter MUST state its active and idle timeout values in Options Data Records described by an Options Template that contains flowActiveTimeout and flowIdleTimeout and is scoped to the applicable Observation Domain or Metering Process. If packet selection is active, the record includes selectionSequenceId as a Flow Key and the exporter supplies the Selection Process Options Data Records required by [RFC5476]. Counters then describe the packets represented under that Selection Process; they do not prove the number of packets present on the wire.¶
The following ingress example omits rocev2BthOpcode because the Flow is not split by opcode:¶
sourceIPv6Address = 2001:db8:1::10 destinationIPv6Address = 2001:db8:2::20 sourceTransportPort = 49152 destinationTransportPort = 4791 protocolIdentifier = 17 flowDirection = 0 rocev2BthPartitionKey = 0xffff rocev2BthDestinationQp = 0x000123 observationPointId = 12 flowStartMilliseconds = 2026-10-26T10:15:00.000Z flowEndMilliseconds = 2026-10-26T10:15:29.999Z packetDeltaCount = 125000 octetDeltaCount = 20000000¶
rocev2BthPacketSequenceNumber contains only the raw 24-bit PSN from one packet. A PSN gap does not, by itself, prove packet loss or duplication. A sequence analysis must identify the applicable transport and QP context, quantify packet selection, preserve observation order, handle wrap modulo 2^24, and account for retries, retransmissions, reordering, capture loss, and exporter overload. A derived loss or duplication metric is outside the scope of this document.¶
A Destination QP is local to the destination endpoint. It is not a globally unique Flow or connection identifier, and the number can be reused after a QP is destroyed. A collector correlates it with the Exporting Process identity, Observation Domain ID, observationPointId, flowDirection, outer source and destination IP addresses, UDP source and destination ports, Partition Key, and packet observation time or aggregate Flow interval. The tuple identifies an observation context; it does not create a protocol-level connection identifier.¶
This document defines only a unidirectional aggregate profile. For bidirectional analysis, a collector correlates two unidirectional Data Records, one for each direction, by using the context above and overlapping Flow intervals. Each record keeps its own Partition Key and Destination QP as Flow Keys. The collector MUST NOT infer a Source QP by reversing a Destination QP value.¶
An exporter can instead use the biflow model in [RFC5103]. Such export is outside the aggregate profile defined in this document. A biflow definition for these IEs must use the normal and Reverse Information Elements for the Partition Key and Destination QP as Flow Keys, and for directional counters. It must not use one Destination QP value for both directions, and it must not aggregate a PSN or any of the three BTH control octets. An opcode can be included only as a Flow Key.¶
The exporter needs to parse RoCEv2 BTH information, which may affect the exporter's performance. Implementers SHOULD consider measures to mitigate this impact, such as sampling rate limiting or hardware acceleration.¶
The Security Considerations for IPFIX [RFC7011] apply to this document as well.¶
When exporting RDMA BTH information across security domains, to prevent information security risks caused by the leakage of sensitive network topology details such as QP numbers, it is recommended to implement comprehensive protection measures, such as using the encrypted transmission options of the IPFIX framework (such as the Datagram Transport Layer Security (DTLS) protocol [RFC9147]) to ensure the confidentiality and integrity of data during transmission.¶
To defend against Denial-of-Service (DoS) attacks that may be caused by maliciously crafted RoCEv2 packets and to prevent the exporter from being overloaded by processing a large amount of invalid traffic, it is recommended to adopt certain protection strategies, such as configuring reasonable rate limiting policies to prevent the monitoring system from being overwhelmed by massive data packets and to ensure its stable operation.¶
This document requests IANA to assign Element IDs from the IPFIX Information Elements registry for the following IEs.¶
+======+===============================+============+============+ | ID | Name | Data Type | Semantics | +======+===============================+============+============+ | TBD1 | rocev2BthOpcode | unsigned8 | identifier | +------+-------------------------------+------------+------------+ | TBD2 | rocev2BthPartitionKey | unsigned16 | identifier | +------+-------------------------------+------------+------------+ | TBD3 | rocev2BthDestinationQp | unsigned32 | identifier | +------+-------------------------------+------------+------------+ | TBD4 | rocev2BthPacketSequenceNumber | unsigned32 | default | +------+-------------------------------+------------+------------+ | TBD5 | rocev2BthOctetAtOffset1 | unsigned8 | default | +------+-------------------------------+------------+------------+ | TBD6 | rocev2BthOctetAtOffset4 | unsigned8 | default | +------+-------------------------------+------------+------------+ | TBD7 | rocev2BthOctetAtOffset8 | unsigned8 | default | +------+-------------------------------+------------+------------+ Table 1: New IEs in the IPFIX Information Elements Registry¶
The authors thank Chongfeng Xie for his review and helpful comments.¶