AI Agent Protocols for Multi-modality

1.1. Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶

2.1. Multi-modality of AI Agent Traffic

An AI Agent traffic may consist of multiple “Part” with multi-modal data (e.g., text, file, sensor data, structured data, video, audio and image) [3]. Different “Part” of an AI Agent traffic may be delivered with different transmission modes (e.g., Streaming, Push Notification, Burst).¶

Taking Google A2A protocol as an example [1], Message constitutes a communication unit between client and server, and Artifacts represent concrete outputs of task executions. Both Messages and Artifacts transmit content composed of one or more Parts:¶

• A Part serves as a container for discrete communication content segments, which MUST contain exactly one content type:text (purely textual), raw (file-based:images, videos, etc.), url (resource locator), or data (structured blobs e.g., JSON).¶

2.2. Indistinguishable Multi-modal AI Agent Traffic

There are two distinct methods by which an AI Agent client or server, acting as the source, can encapsulate multi-modal data.¶

a) The multi-modal data of an AI Agent traffic can be encapsulated into multiple lower layers streams (e.g., HTTP streams, MoQ stream, QUIC Stream):¶

• the underlying network lacks awareness that these streams belong to the same AI Agent traffic. Consequently, while network nodes (e.g., core network user plane function, radio access network base stations) can guarantee the QoS for individual stream during data forwarding, the overall performance (e.g., latency) for the traffic may increase.¶

• Moreover, in scenarios like multi-Agent systems, it may require the multi-modal data to arrive at different receiving Agent endpoints as simultaneously as possible. This characteristic renders the existing techniques susceptible to introducing significant overall performance (e.g., latency) decrease if the underlying network lacks awareness.¶

b) The multi-modal data of an AI Agent traffic can be encapsulated within a single lower layer stream (e.g., HTTP stream, MoQ stream, QUIC Stream), this stream exhibits inherent multi-modal characteristics:¶

• This manifests as significant variance in Part sizes (e.g., text snippets vs. video files) and divergent quality of service (QoS) criticality across Parts, e.g., regarding transmission urgency and latency tolerance thresholds.¶

• Limitation: Content of different Parts are encapsulated within a single lower layer stream (e.g., as payload of HTTP protocol, MoQ protocol, QUIC protocol), the underlying network (e.g., mobile network, transport network) remains unaware of the distinct Parts and their individual requirements, such as specific QoS needs.¶

The Following figure illustrates the interaction between Agent Client and Agent Server via underlying Network:¶

+--------+                  +--------+
| Agent  |                  | Agent  |
| Client |                  | Server |
+--------+                  +--------+
    |                            |
    |       +------------+       |
    +-------| Underlying |-------+
            |  Network   |
            +------------+

Figure 1. Interaction between Agent Client and Agent Server via underlying Network.¶

3.1. Quality of Service (QoS)

Multi-modal AI Agent traffic should be transmitted appropriately by underlying network. Taking 3GPP mobile network as an example of underlying network, QoS framework controls and manages transmission of traffic to ensure performance. QoS framework governs how diverse traffics are treated to meet specific performance requirements (e.g., latency, reliability, data rate, packet loss) [5], [6], [8]. It operates primarily through QoS Flows, which are granular data streams bound with specific QoS parameters and characteristics. These parameters and characteristics indicate the packet forwarding treatment (e.g., scheduling weights, admission thresholds, queue management) applied throughout the network, from the User Equipment (UE) to the mobile network (e.g., core network, radio access network). Multi-modal data of an AI Agent traffic should be transmitted with differentiated QoS to increase the overall performance and resource utilization.¶

3.2. Awareness of AI Agent Traffic Multi-modality

The underlying network should be aware of the AI Agent traffic characteristics. Different transmission treatment policies may be configured and bounded to the different multi-modal data of an AI Agent traffic based on their different traffic characteristics (e.g., delay budget, priority, error rate). Taking mobile network as an example, the Application Server may influence QoS treatment by notifying service requirements to the mobile network functions. Based on this, the mobile network provisions and configures the corresponding QoS enforcement policies. Subsequently, the mobile network maps incoming packets of application traffic to the appropriate QoS Flows by matching against the enforcement policies and the traffic information (e.g., typically IP 5-tuple). Similarly, the AI Agent client and server need to notify necessary information on its AI Agent traffic to the underlying network for better performance.¶

4.1. Multiple Streams for Multi-modal Agent Traffic

+------------------------------------------------------+
|                    Agent Protocol                    |
|               {Correlation Information}              |
+------------------------------------------------------+   --+
| e.g., HTTP Protocol       | e.g., MoQ                |     |
| {Correlation Information} | {Correlation Information}|     |
+------------------------------------------------------+     |-- Lower Layers
|                   Transport Protocol                 |     |
|               {Correlation Information}              |     |
+------------------------------------------------------+   --+

+-------------------------------------------------+
| POST /rpc/ HTTP/2.0                             |
+-------------------------------------------------+
| Host: agent.example.com                         |
| Content-Type: application/json                  |
| Content-Length: xxx                             |
| Correlation Information: Relationship indictor  |
+-------------------------------------------------+

4.2. Single Stream with Multi-modality

Using method b) mentioned in 2.2, the source Agent can encapsulate multi-modal data of an AI Agent traffic (e.g., as separate Parts) within a single lower layer stream. One of the limitations of this method is that the underlying network remains unaware of individual Parts and their associated QoS requirements.¶

To address this limitation, the source Agent is required to perform Part segmentation during encapsulation: it should separate different Parts with clear boundaries, and embed supplementary metadata (e.g., traffic information, QoS requirements) corresponding to each Part within the packet header or sub-headers. This supplementary metadata enables the underlying network to identify assistance information on specific Parts (e.g., their different traffic characteristics and respective QoS requirements).¶

a) In one implementation, supplementary metadata of each Part can be encapsulated in the packet header (e.g., Agent protocol header, or lower layer header) as in Figure 4. For example, the header contains fields such as:¶

• the total number of Parts,¶

• sequence number of each Part and the corresponding traffic information (e.g. size of a Part, modality of a Part, and transmission mode of a Part), QoS requirements (e.g., latency, error rate, priority, privacy), etc.¶

The contents of different Parts will be encapsulated in the payload in sequence with the same order of the supplementary metadata in the header.¶

+----------------------------+-------------+-------------+-------------+
|Header{                     |             |             |             |
|   Number of Parts,         |             |             |             |
|   {Part 1,                 |             |             |             |
|      {Traffic information, |             |             |             |
|       QoS requirements},   |   Payload   |   Payload   |     ...     |
|    Part 2,                 |   (Part1)   |   (Part2)   |             |
|      {...},                |             |             |             |
|   ...                      |             |             |             |
|   }                        |             |             |             |
|}                           |             |             |             |
+----------------------------+-------------+-------------+-------------+

Figure 4. Supplementary metadata in the header.¶

b) In another implementation, supplementary metadata of each Part can be encapsulated in the sub-header of the sub-PDU as in Figure 5. For example, each sub-header contains fields such as:¶

• sequence number of each Part and the corresponding traffic information (e.g. size of a Part, modality of a Part, and transmission mode of a Part), QoS requirements (e.g., latency, error rate, priority, privacy), etc.¶

The contents of different Parts, on the other hand, will be encapsulated in the payload following the corresponding sub-header. In this case, a common header (e.g., in front of the subPDUs) can contain the field to indicate the total number of Parts of an AI Agent traffic.¶

                    +------------------------+
                    |Sub-header{             |
                    |   Part ID / SN,        |
                    |   Traffic information, |
                    |   QoS requirements     |
                    |}                       |
                    +------------------------+
                    |                       /
                    |                    /
                    |                 /
                    |              /
                    |           /
                    +------------------+  +------------------+
                    |Sub-header|Payload|  |Sub-header|Payload|
                    +------------------+  +------------------+
                    |                 /  /                  /
                    |                / /                   /
                    |               //                    /
+---------------------------------------------------------------+
|       Header      |               |                     |     |
| number of subPDUs |     subPDU    |       subPDU        | ... |
+---------------------------------------------------------------+

Figure 5. Supplementary metadata in the sub-headers.¶

Moreover, the source Agent may also inform the underlying network in advance about the supplementary metadata (e.g., traffic information, QoS requirements) corresponding to each distinct Part. The underlying network then provision QoS parameters for each Part. Upon receiving the stream, the underlying network can distinguish these multi-modal Parts based on their specific characteristics and associated QoS requirements.¶

Upon data arrival at the receiving endpoint, the entity of underlying network can perform reassembly of data packets associated with an AI Agent traffic prior to delivery to the receiving Agent client/server. Or, the entity of Agent protocol layer in the receiving Agent client/server can reassemble the data packets. Which one is better can be left for further study.¶