Using BMP over QUIC connection

The BGP Monitoring Protocol (BMP) defines standard mechanisms for obtaining route views by monitoring BGP sessions. BMP operation uses TCP as its transport protocol to provide reliable communication. BMP establishes connection relationships between monitored router and monitoring station using a TCP session. In BMP message communication, in order to simplify the implementation, only the monitored router reports messages to the monitoring station, and the station does not send messages to the router . In other words, the BMP communication is actually unidirectional (from router to station). As a consequence, the direction from the monitoring station to the monitored router may be used as an interface for malicious attacks on the router. As BMP supports more and more types of routes to be reported, the number of reported BMP messages is also increasing, which also brings huge challenges to TCP data transmission pressure. QUIC is a UDP-based multiplexed and secure transport protocol that provides connection-oriented and stateful interaction between a client and server. It can provide low latency and encrypted transport with resilient connections. QUIC uses multiple simultaneous streams to carry data in one direction. Each stream is a separate unidirectional or bidirectional channel consisting of an ordered stream of bytes. In addition, each stream has its own flow control, which limit bytes sent on a stream, together with flow control of the connection. Among them, the unidirectional stream is very consistent with the message transmission mechanism of BMP. TCP-based BMP works well in stable networks, but QUIC offers advantages in challenging environments. When collectors experience processing delays from database operations or analytics, QUIC's multiplexing prevents head-of-line blocking, maintaining throughput. For mobile agents or dynamic networks, QUIC's connection migration ensures uninterrupted monitoring sessions during network changes. Therefore, QUIC is a proper transport protocol for the message transmission mechanism of BMP. This document specifies how to use QUIC as the secure transport protocol for BMP.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. In this document, the terms "client" and "server" are used to refer to the two ends of the QUIC connection. The client actively initiates the QUIC connection. The terms "monitored router" and "monitoring station" are used to refer to the two ends of the BMP session. The router sends BMP messages to the station. The station does not respond to the router, except in the Function Streams with Control Stream selection (), where the station sends control response messages over the control channel as described there.

Client: The endpoint that initiates a QUIC connection, typically a BMP monitored router.
Server: The endpoint that accepts a QUIC connection, typically a BMP monitoring station.

QUIC connection establishment is described in . During establishing connection, BMP over QUIC (BMPoQUIC) support is indicated by selecting the Application-Layer Protocol Negotiation (ALPN) token "BMPoQ" specified in during the TLS handshake. The monitored router MUST also act as the client while the monitoring station MUST also act as the server. Early Data (also known as 0-RTT) provides weaker security than standard TLS, lacking forward secrecy and replay attack protection, while BMP information contains sensitive routing information. Therefore, BMPoQUIC implementations MUST NOT support Early Data. Servers MUST reject "early data" extensions in ClientHello messages, and clients MUST NOT attempt to send 0-RTT data.

The typical QUIC connection termination process is described in .

When a BMP session is implemented based on a QUIC connection, the idle timeout SHOULD be disabled or the QUIC max_idle_timeout SHOULD be set appropriately in order to keep the QUIC connection persistent even if the BMP session is idle. When a BMP monitoring station receives a termination message, it will gracefully close the BMP session. The station SHOULD close the associated QUIC connection. When a BMP monitored router intends to close the BMP session, it SHOULD send a termination message to the BMP monitoring station before closing the QUIC connection, so that the station can gracefully close the BMP session. When a BMP monitored router detects the interruption of the QUIC connection (e.g., because the connection is idle-timed out or reset), it cannot send a termination message over the interrupted connection. In this case, the monitored router SHOULD attempt to re-establish a new QUIC connection as specified in , and a new BMP session starts on the new connection, beginning with an Initiation message as described in .

There are seven kinds of BMP messages sent from the monitored router to the monitoring station, namely Route Monitoring message, Statistics Report message, Peer Down Notification message, Peer Up Notification message, Initiation message, Termination message and Route Mirroring message . The seven kinds of BMP messages need to be mapped into QUIC streams. QUIC uses a Stream ID to identify each stream. The least significant bit (0x01) of the stream ID identifies the initiator of the stream (client-initiated with the bit set to 0). The second least significant bit (0x02) of the stream ID distinguishes between bidirectional streams (with the bit set to 0) and unidirectional streams. No BMP message is ever sent from the monitoring station to the monitored router, except in the Function Streams with Control Stream selection (), where the station sends control response messages over the control channel as described there. The monitored router MAY take steps to prevent the monitoring station from sending data or it MAY silently discard any unrecognized data sent by the monitoring station. So BMP messages from monitored router (as a client) SHOULD be mapped into unidirectional stream whose stream type is 0x2, or mapped into bidirectional stream whose stream type is 0x0, according to the above. For clarity, the figures and examples in this section label the streams with sequential ordinal numbers ("Stream 0", "Stream 1", ..., "Stream n"), where "Stream 0" denotes the first stream created on the BMPoQUIC connection, "Stream 1" the second, and so on. These ordinals are not QUIC Stream IDs. The actual QUIC Stream ID of a client-initiated stream follows from the stream type and the number of streams of the same type created before it, as specified in . In particular, the first client-initiated bidirectional stream has QUIC Stream ID 0x0, and the first client-initiated unidirectional stream has QUIC Stream ID 0x2. Which stream type is used for a given BMP channel is specified in the corresponding stream selection subsections.

When a router has many peers and a large number of routes, if the related BMP messages are reported through an independent stream, the communication pressure of this stream will be very large and the efficiency will be very low. In order to reduce the communication pressure and improve the communication efficiency, multiple streams can be allocated to carry BMP messages according to the number of peers or Address Family Identifier/Subsequent Address Family Identifier (AFI/SAFI). This document introduces three multi-stream selections (, and ), which can be freely chosen by network operator or user according to the level of complexity involved in configuring BGP. And the extent of complexity in BGP configuration is also decided by the network operator. To avoid a fragmented ecosystem where implementations support different subsets of the stream selections, a baseline mode is defined. Implementations MUST support the single-stream mode, in which all BMP messages are carried over Stream 0 (the first stream created on the connection; see for the ordinal to QUIC Stream ID mapping). The single-stream mode corresponds to the default configuration of the Peer Stream selection (one stream for all peers) and provides behavior equivalent to TCP-based BMP . The multi-stream selections are optional extensions: an implementation MAY support the Peer Stream selection, the Per-AFI/SAFI Stream selection, or the Function Streams with Control Stream selection, in any combination, selected by configuration. The following operational guidance may assist in choosing among the selections:

Single-stream mode: the simplest to implement, operate, and debug. Suitable when the BMP message volume is moderate or the monitoring station processes messages sequentially.
Peer Stream: distributes the load across a small number of streams. Suitable when a monitored router has many peers with significant per-peer message volume.
Per-AFI/SAFI Stream: organizes route data per address family. Suitable when the route volume differs significantly across address families, and in deployments where BGP itself runs over QUIC, for symmetry with the BGP stream structure.
Function Streams with Control Stream: separates low-volume state control messages from high-volume route data and allows the monitoring station to acknowledge session state transitions. It is the most complex option and is suitable for large-scale deployments where head-of-line blocking between state and route data must be avoided.

According to , the Initiation message MUST be sent as the first message after the BMP session comes up, meaning that the monitoring station MUST firstly receive the Initiation message after the BMP session comes up. So when using peer stream or per-AFI/SAFI stream to send BMP messages, it MUST be ensured that the Initiation message is the first message sent on each BMP stream, meaning that an Initiation message MUST be sent first for every BMP stream establishment. When using function streams with control stream, it also MUST guarantee that message ordering by design implementation. In the future, the information TLV of the Initiation message may be expanded to carry the multi-stream type (peer stream, per-AFI/SAFI stream, function streams with control stream, etc.), allowing the server (collector) to know, based on the multi-stream type, what kinds of BMP messages will be conveyed in the current stream. This extension does not replace the per-stream Initiation requirement above: in the peer stream and per-AFI/SAFI stream selections, the Initiation message sent on each stream would carry its own multi-stream type TLV, while in the function streams with control stream selection, the Initiation message on the control channel would indicate the selection for the whole connection. In addition, a Termination message can be sent on any stream, which terminates the entire BMP session and closes the QUIC connection along with all of its opened streams.

In order to reduce the communication pressure and improve the communication efficiency, multiple unidirectional streams can be allocated by router according to the number of BGP peers of the router, and each stream is used to transmit the BMP message of the specified peers, as shown in .

| Stream 0 | | Peer 2 | | | +------------+ +--------------+ ~ ~ ~ ~ +------------+ +--------------+ | Peer x | | | | |---------------------->| Stream n-1 | | Peer y | | | +------------+ +--------------+ ]]> The number of streams can be configured as needed, and the corresponding relationship between peer and stream can be matched through configuration. For example, if there are five peers, when three streams are created, stream 0 could carry the BMP messages of peer 1 and peer 2, stream 1 could convey the BMP messages of peer 3 and peer 4, and stream 2 could send the BMP messages of peer 5. The default number of streams is 1, meaning that by default, stream 0 is created for all peers. All of these are client-initiated unidirectional streams.

When multiple peer streams are used, each stream may carry different type (AFI/SAFI) routes from router, which may make the information seem a bit messy. So a unidirectional QUIC stream can be created for Per-AFI/SAFI to carry the routes of the specific AFI/SAFI, as shown in the figure below. The first stream created (Stream 0 in ) is always a common stream, and in this stream selection it is a client- initiated unidirectional stream (QUIC Stream ID 0x2), carrying BMP messages which may or may not include AFI/SAFI information. If the common stream is used to carry the peer-related BMP messages, the Peer Up Notification message MUST be sent prior to the other peer-related BMP messages.

| Stream 0 | +----------------+ +--------------+ +----------------+ +--------------+ | AFI/SAFI 1 |---------------------->| Stream 1 | +----------------+ +--------------+ ~ ~ ~ ~ +----------------+ +--------------+ | AFI/SAFI n |---------------------->| Stream n | +----------------+ +--------------+ ]]> In order to minimize resource occupation, it is recommended to create a stream for the AFI/SAFI when any BGP peer of the AFI/SAFI is in the up state. Of course, after creating the AFI/SAFI stream or common stream, an Initiation message MUST be sent first before sending the Peer Up Notification message. When per-AFI/SAFI streams are used to transmit BMP messages, it is necessary to distinguish which BMP messages can be categorized by per-AFI/SAFI and determine how to send BMP messages for non-per- AFI/SAFI.

For Route Monitoring message, it is per-AFI/SAFI route message, and it SHOULD be carried over corresponding AFI/SAFI stream.
For Route Mirroring message, it SHOULD always be carried over the common stream (stream 0), regardless of whether the AFI/SAFI information can be extracted from the BGP PDU. Route Mirroring is mostly used for troubleshooting scenarios, so keeping it on a single stream keeps the behavior simple and predictable.
For Statistics Report message, if the stat type is based on per-AFI/SAFI, it includes the AFI/SAFI. If not, it does not include the AFI/SAFI. So it SHOULD be carried over the common stream (stream 0).
For Peer Up Notification message, it could include open message containing multiple AFI/SAFIs, and it SHOULD be carried over all corresponding AFI/SAFI streams, and MUST be sent before sending other BMP messages which include peer information.
For Peer Down Notification message, as it does not include AFI/SAFI information, it SHOULD be carried over all corresponding AFI/SAFI streams, or over the common stream that carries the Peer Up Notification message.

If BGP still uses TCP as the transport protocol, the Per AFI/SAFI Stream structure can be used selectively. If BGP uses QUIC as the transport protocol , the Per AFI/SAFI Stream structure is RECOMMENDED in a BMPoQUIC connection because of the implementation that per-AFI/SAFI streams (function channels) are also used to carry routing information in one BGP over QUIC (BoQ) connection.

In order to ensure the timing of BMP messages, the above two solutions ( and ) must send an Initiation message or Peer Up/Down message on each stream, which may also increase the network load. Therefore, the solution with control stream is designed. Like the design of BGP over QUIC (BoQ) , the per-peer or per-AFI/SAFI streams (function channels) and the associated control stream (control channel) for the BMP session are called "BMP channels". In one BMPoQUIC connection, one control channel and one or more function channels are used to carry BMP information, as shown in the figure below.

| Stream 0 | +------------------------+ +--------------+ +------------------------+ +--------------+ | Function Channel 1 |-------------->| Stream 1 | +------------------------+ +--------------+ ~ ~ ~ ~ +------------------------+ +--------------+ | Function Channel n |-------------->| Stream n | +------------------------+ +--------------+ ]]> On a BMPoQUIC connection, the BMPoQUIC client first establishes a bidirectional stream for the "BMP control channel". The control channel is used to send the associated status control information, such as BMP session state control messages (Initiation message and Termination message) and BGP peer relationship state control messages (Peer Up Notification message and Peer Down Notification message). The control channel is also used to carry the associated control response information for keeping the order of BMP messages. In addition, the control channel is used to carry the periodic and low-load message, such as Statistics Report message. When the BGP route association information with high load is needed to send, the function channel is created to deliver the key non- status BMP messages by monitored router, such as Route Monitoring message, and Route Mirroring message. The Control channel is always the first stream created (Stream 0 in ) and is a client- initiated bidirectional stream (QUIC Stream ID 0x0). The Function channels, which are initiated by a monitored router, are unidirectional streams (the first one having QUIC Stream ID 0x2, the next 0x6, and so on).

In QUIC layer, BMPoQUIC message are carried by QUIC STREAM frames. In BMPoQUIC layer, the two BMPoQUIC Frame types are defined, namely Data and Control Data, according to the BMP channels. Data frames have following format: Control Data frames have following format:

Type:: one octet, identifying the frame type.
Length:: a 24-bit unsigned integer that describes the length in bytes of the frame payload.
Sequence Number:: a 32-bit unsigned integer that indicating the sequence number of sending the control data by monitored router. After the sequence number increases to 2^32-1 by monitored router, the next sequence number returns to 0.
Frame Payload:: BMP messages.

When the monitoring station needs send response message in control channel, the Length is set to 0 and the Sequence Number is same with corresponding BMP message from the monitored router.

After the control channel is created, the Initiation message is first sent over control channel. Only when the response message corresponding to the initialization message is received by the monitored router, the other BMP messages are allowed to be sent by the monitored router. When the Peer Up Notification message is sent over the control channel, and its response message from the monitoring station MUST be received, the monitored router can create function channels by using unidirectional QUIC streams. Or before the monitored router needs send other peer-related BMP messages over function channels, the corresponding function channel MUST be created. These function channels are used to carry the relevant non-status BMP messages with high load for specific peer or AFI/SAFI. So these function channels include one Per-Peer common function channel and several Per-Peer Per-AFI/SAFI function channels according to the AFI/SAFIs of the OPEN information contained in the Peer Up Notification message . The Per-Peer common function channel can be used to carry messages related to peer and need to be sent on the same channel, such as Route Mirroring message. For Route Mirroring message (regardless of whether it can get AFI/SAFI or not), it SHOULD be carried over the Per-Peer common function channel to prevent impacting the transmission efficiency of Route Monitoring messages in the Per-Peer Per-AFI/SAFI function channel, because Route Mirroring message may be extremely verbose. For Statistics Report message (regardless of whether it can get AFI/SAFI or not), since it may be a periodic message with low load, and different statistics may have some correlation. So it SHOULD always be carried over the control channel to maintain better atomicity of all statistics. The Per-Peer Per-AFI/SAFI function channel SHOULD be used to transmit messages only related to AFI/SAFI for a specific peer, such as Route Monitoring message. As a Peer Down Notification message is sent over the control channel, the router SHALL close all the related function channels. After the Peer Down Notification message was sent, the response message to the previous Peer Up Notification message SHOULD be ignored. In the future, the Peer Down Notification message may be expanded to include AFI/SAFI information, allowing the router to notify peer down event of the specified AFI/SAFI for a peer.

In Monitored Router, the IPv4 unicast peer 192.0.2.1 is monitored. The Monitored Router and Monitoring Station establish a BMPoQUIC connection, then the relevant processing flow is shown in .

| Monitoring | | Router | | Station | +-----------+ +-----------+ 1. Connection | | Establishment | 3. Send Initiation | 2. Create CC |--------------CC----------->|4. Receive (Stream 0) | | Initiation | 5.Send Initiation Response | 6. Receive |<-------------CC------------| Initiation | | Response | | | 8.Send Peer Up Notification| 7. Peer 192.0.2.1 |--------------CC----------->|9. Receive Peer Up | | Up Notification | 10.Send Peer Up Response | 11. Receive Peer |<-------------CC------------| Up Response | | | 12a.Create FC-1 (Stream 1) | | Send Related Message | | (RMI) | |--------------FC-1--------->| | | | 12b.Create FC-2 (Stream 2) | | Send Related Message | | (RMO) | |--------------FC-2--------->| | | | 12c.Send SR Message | |--------------CC----------->| | | 13. Peer 192.0.2.1| 14.Send Peer Down | Down | Notification | |--------------CC----------->| 15. Close FC-1/2 | | | | 16. Close | 17.Send Termination | Connection |--------------CC----------->| ]]> Similarly, for IPv6, the IPv6 unicast peer 2001:db8::1 is monitored. The processing flow is shown in .

| Monitoring | | Router | | Station | +-----------+ +-----------+ 1. Connection | | Establishment | 3. Send Initiation | 2. Create CC |--------------CC----------->|4. Receive (Stream 0) | | Initiation | 5.Send Initiation Response | 6. Receive |<-------------CC------------| Initiation | | Response | | | 8.Send Peer Up Notification| 7. Peer 2001:db8::1|------------CC------------->|9. Receive Peer Up | | Up Notification | 10.Send Peer Up Response | 11.Receive Peer |<-------------CC------------| Up Response | | | 12a.Create FC-1 (Stream 1) | | Send Related Message | | (RMI) | |--------------FC-1--------->| | | | 12b.Create FC-2 (Stream 2) | | Send Related Message | | (RMO) | |--------------FC-2--------->| | | | 12c.Send SR Message | |--------------CC----------->| | | 13.Peer 2001:db8::1| 14.Send Peer Down | Down | Notification | |--------------CC----------->| 15.Close FC-1/2 | | | | 16.Close Connection| 17.Send Termination | |--------------CC----------->| ]]> In and , CC is the Control Channel, and FC is the Function Channel. RMI is the Route Mirroring message, and RMO is the Route Monitoring message. SR is the Statistics Report message.

BMPoQUIC uses QUIC which uses TLS version 1.3 or greater. Therefore, the TLS handshake process can be used for BMPoQUIC endpoint authentication. A third-party authentication mechanism can also be applied for BMPoQUIC endpoint authentication, such as a TLS client certificate.

The decision to use BMPoQUIC instead of the TCP-based mechanism in is an operational decision, and an implementation MUST provide a configuration mechanism to enable BMPoQUIC on the BMP session. Some connectivity problems (such as blocking UDP) could result in a failure to establish a QUIC connection. In this case, the monitored router MUST NOT automatically fall back to a TCP-based BMP session. BMP is a provisioned mechanism: whether the BMP session uses QUIC or TCP is determined by configuration at both ends. Automatic fallback would silently downgrade the transport security of the BMP session, since TCP-based BMP is not encrypted, and could surprise operators. If a QUIC connection fails after it has been established, the monitored router SHOULD attempt to re-establish a new QUIC connection rather than fall back to a TCP-based BMP session. On the new connection, a new BMP session starts with an Initiation message, as described in . This document does not define any stream-level error recovery mechanism. If a monitoring station receives a BMP message that it cannot parse on any stream, it SHOULD terminate the entire BMP session, consistent with the behavior of TCP-based BMP . The station MAY reset the individual stream on which the malformed message was received before terminating the session. Stream-local error handling (e.g., resetting a single stream while keeping the rest of the BMP session operational) would require mechanisms for stream identity negotiation and message re-synchronization that BMP does not currently define; this is left for future revisions of this document. When using multiple streams, a configuration MAY be implemented to select which multi-stream selection to use. Debugging BMPoQUIC sessions is more complex than debugging TCP-based BMP sessions, because QUIC encrypts all transport-layer information. Protocol analysis tools such as Wireshark can decode QUIC streams and analyze each stream separately, but only when the TLS session keys are provided (e.g., through a TLS key log file). Operators are therefore advised to capture the TLS key log file together with the packet trace when debugging BMPoQUIC sessions.

This document creates a new registration for the identification of BMPoQUIC in the "Application Layer Protocol Negotiation (ALPN) Protocol IDs registry established in . The "BMPoQ" string identifies BMPoQUIC:

Protocol: BMPoQUIC
Identification Sequence: 0x42 0x4d 0x50 0x6f 0x51 ("BMPoQ")
Specification: This document

The security considerations described throughout and apply here as well. This document replaces the transport protocol layer of BMP from TCP to QUIC. The basic protocol specification of BMP is not modified, and therefore the new security risks are not introduced to the basic BMP protocol. BMPoQUIC enhances transport-layer security for BMP session according to . This document does not require to support third-party authentication (e.g., backend Authentication) due to the fact that TLS does not specify this way of authentication. If third-party authentication is needed, TLS client certificates are recommended to be used here. A BMPoQUIC connection is typically long-lived and may stay idle for extended periods, which widens the window for off-path ICMP-based attacks, as studied for TCP in . A single successful injection can affect the connection for its entire remaining lifetime. Implementations MUST follow the ICMP handling requirements of and ; in particular, ICMP messages whose embedded payload cannot be validated as belonging to the QUIC connection MUST be ignored, ICMP Packet Too Big messages indicating a size below the 1200-byte minimum datagram size MUST be ignored, and the maximum datagram size MUST NOT be increased based on ICMP. A BMPoQUIC implementation MUST NOT close the QUIC connection on the basis of ICMP destination or port unreachable errors alone. ICMP unreachable errors are unauthenticated and can be forged. Liveness of the connection is determined by QUIC's own loss detection and timeout machinery as required by . Since a spoofed PMTU reduction is not reversed without new probing, BMPoQUIC implementations SHOULD implement DPLPMTUD with periodic probing so that a successfully spoofed PMTU clamp can recover. Long-lived, high-throughput BMPoQUIC connections will reach the AEAD confidentiality limits of , whose key update procedures MUST be supported.