]> Alibaba Inc.

miaoji.lym@alibaba-inc.com

Alibaba Cloud

max.ldp@alibaba-inc.com

Network Working Group agentproto live agent MOQT dialog management use cases Live AI interactions combine ongoing observations, overlapping responses, and changes to the devices through which a user participates. This document contributes three use case extensions to the AgentProto use cases and requirements work: live multimodal calls, AI glasses with companion devices, and embodied systems that converse while performing delegated actions. For each scenario, the document identifies the existing requirements it builds on, the interoperability questions it raises, and the requirement refinements motivated by its interaction flow. The focus is on independently changing observation sources, response state, participation, and authority within a continuing dialog. The scenarios are independent of transport selection. An appendix uses Media over QUIC Transport (MOQT) to illustrate questions for subsequent binding and application-profile work. About This Draft This is an individual contribution to the Agent Communication Protocols (agentproto) Working Group. It proposes additions to the shared use cases and requirements work, using draft-agentic-ai-usecases-requirements-05 as the comparison baseline. It does not represent working group consensus. Discussion takes place on the agentproto mailing list (agentproto@ietf.org).

Introduction This document develops generalized scenarios informed by emerging industry use cases, to help ensure that IETF standardization work covers typical live agent interactions and their interoperability requirements. A live agent, as used here, is an AI application that participates in an ongoing interaction with overlapping input and output. It can listen while speaking, observe a changing scene while explaining it, and report on work while continuing to receive user instructions. Users can interrupt a response, change the available sensors, or continue the interaction on another device. The term does not mean a human customer-service representative. This document contributes live interaction use cases and derived requirement refinements to the work represented by . It uses the existing client, agent, tool, and mediator roles; a device category alone does not introduce a new protocol role or requirement. The proposed contribution is the coordination needed when observation, response, device participation, and action state change independently. For example, a camera can stop while the connection remains available, a user can silence speech while a delegated action continues, and a new device can join while output for an earlier device is still in transit. Implementations need to agree on the affected scope and resulting state across these changes. The charter provides the boundary for this work: dialog correlation and lifecycle, their propagation across intermediaries, and bindings to existing protocols. The requirements below distinguish that common work from application semantics for media, device permissions, and action outcomes. Sensor content and model-facing state remain application data. Physical control, local safety, and the interface used to obtain user consent remain with the relevant application or device specifications. presents the scenarios, maps proposed refinements to their motivating use cases and existing requirement identifiers, and discusses relationships to existing work and open questions. The MOQT example in supports later binding evaluation; accepting the use cases does not depend on choosing that transport.

Terminology and Deployment Assumptions This document uses dialog and dialog context as defined in and . Dialog context concerns correlation and lifecycle; it is not a container for observations, response content, or model memory.

Live Agent session:

The application interaction, including participants, authorized modalities, and interaction state. It can use multiple transport sessions and outlive a connection. This term does not require an identifier additional to the AgentProto dialog identifier.

Observation source:

A particular source of input to the application, such as a camera or microphone. A restarted or replaced source can require a distinguishable instance within the application even when the dialog continues.

Application signaling:

Exchanges that establish or change application state, including source availability and scoped response control. The selected application protocol or profile defines these exchanges and their relationship to dialog changes.

The scenarios assume independently implemented device clients, agent services, and, where applicable, intermediaries or device controllers. A companion phone can host the client and mediate access to glasses; alternatively, the glasses can host their own client. A device or sensor is not necessarily a separate AgentProto participant. The application needs a way to associate its sources and outputs with the participant that represents them. The client and service can be operated by different providers, and a delegated agent or controller can introduce another administrative boundary. The interoperability question is what those implementations can infer from an exchange, including when a peer is unavailable.

Use Case Extensions The comparison baseline is revision -05 of . The multimodal framework in , particularly Sections 2.1.1 and 3.3.2 through 3.3.4, provides additional background. Each use case below includes a description, an interaction flow, and an analysis of requirements and existing coverage. UC1-UC3 identify the use cases; S1-S7 in identify the shared requirement refinements they motivate.

UC1: Live Multimodal Calls

Description A user asks an agent to explain an object shown by a camera. A client streams observations while the agent returns speech and visual annotations. The client, agent service, and optional speech or media intermediaries can come from different vendors.

Interaction Flow

1. The client and service establish a dialog and agree on active audio and video. The application associates the camera source with the dialog.
2. The agent begins an explanation based on an identified observation while further input continues to arrive.
3. The user interrupts that explanation and points the camera at another object. The client stops local presentation and requests the intended remote effect on the identified response.
4. The user disables video while continuing audio. Output associated with the earlier observation can still arrive after the change.

Requirements and Existing Coverage A1-1, A1-4, A1-5, and A1-10 in provide the streaming and multimodal basis; A1-2 and CMN-11 cover task cancellation and dialog changes. The added question is which observation and response a change affects while other exchanges continue. Without an agreed scope, one implementation can treat an interruption as local silence, another as cancellation of a task, and another as termination of the dialog. A shared dialog identifier alone also does not identify the observation to which a delayed annotation applies. The application needs to make these relationships and operation outcomes interoperable, with dialog correlation maintained across any intermediaries. , Section 3.4, already proposes exchange correlation and interruption for speech-processing chains. This case adds changing visual observations and modality availability during an overlapping response. Shared interruption and correlation requirements should be consolidated with that work. The refinements below are , , , and .

UC2: AI Glasses and Companion Devices

Description A person uses AI glasses for assistance while handling everyday objects. The glasses provide audio and, when supported and authorized, video or pointing information. A companion phone hosts the client, provides connectivity and consent controls, and connects to an agent service.

Interaction Flow

1. The client establishes authorized participation and associates the glasses' input and output with the dialog.
2. The wearer stops camera capture through a local control. The phone's connection remains available, and audio interaction can continue.
3. The wearer removes the glasses and continues the interaction on the phone. The application changes the active source and the recipient of private output, with authorization for the affected roles.
4. The glasses reconnect or send delayed data. The application distinguishes the old source and its former permissions from the current state.

Requirements and Existing Coverage CMN-11, CMN-12, and B1-6 through B1-8 in supply lifecycle, authorized modification, and resumption requirements. Hardware mediation already appears in its Section 4.6.1. , Section 3.3, also addresses continuation on another channel. Here, a physical source and its output availability can change without a client or network disconnection. A service that observes only connectivity cannot distinguish active capture from locally stopped capture or an unavailable camera. Similarly, successful dialog resumption does not determine which device may now receive private output or whether an earlier sensor grant is still applicable. Device continuity can be a change within one client or an authorized change of participant; those cases need to remain distinguishable. Battery or connectivity constraints can reduce video availability while audio remains usable. The protocol-visible information is the effect on available modalities, not detailed battery telemetry or a prescribed consent interface. These conditions motivate , , , , , and .

UC3: Embodied Systems with Concurrent Conversation and Action

Description A service robot carries an object under an authorized task while explaining its progress. Cameras and microphones provide observations; an agent handles the conversation; a controller exposes operations through a tool or mediator. The agent and controller can be independently implemented.

Interaction Flow

1. The agent delegates an authorized action and associates its application operation with the ongoing dialog.
2. The person asks it to stop explaining. The application interrupts speech without treating that request as a cancellation of the physical action.
3. A later request to change the physical task requires its own authorized operation and an outcome reported by the controller.
4. Connectivity fails before that outcome arrives. During the interruption, both the physical scene and the camera source can change.
5. On recovery, the application obtains the operation status and current observations before reporting the situation or deciding whether to retry.

Requirements and Existing Coverage B1-5, B2-5, B2-6, and Section 4.2.1 of provide cancellation, scoped authorization, and handling of uncertain operation outcomes. Its mediator model can already represent access to the controller. The additional condition is that perception, explanation, and action have independent lifetimes. Neither stopped speech nor a lost connection establishes that an action stopped. Recovering an action result also does not establish that a cached observation depicts the current scene. Interoperability requires separate scopes and outcomes for these operations, together with their association to the continuing dialog. The proposed refinements are , , , , and . Motor commands, servo timing, and safety interlocks remain device and domain functions. A dialog protocol does not replace local safety control or an emergency-stop mechanism.

Candidate Requirement Refinements The identifiers S1-S7 are local to this contribution. They identify proposed clarifications or additions to the requirements in , rather than seven wholly unaddressed protocol capabilities. Common requirements from that document continue to apply. Each refinement describes an observable interoperability outcome. It is expressed in ordinary language, not implementation conformance language. The mapping below identifies the motivating use cases, related existing requirements, and specification areas for discussion. The use-case column records the examples developed in , rather than restricting where a refinement can apply. It does not assign all of the state described here to the AgentProto dialog context. Use Case and Existing Requirement Mapping

ID	Motivating use cases	Related requirements in AGENTIC-UC	Proposed specification area
S1	UC2	CMN-11, CMN-12	Dialog lifecycle and application participation
S2	UC1, UC2	A1-4, A1-5, CMN-11, CMN-12	Dialog changes and modality profile
S3	UC1, UC2, UC3	A1-10, B5-3	Dialog binding and application associations
S4	UC1, UC3	A1-2, B1-5, CMN-11	Application operation control
S5	UC1, UC2, UC3	A1-3, A1-6, CMN-11	Application availability signaling
S6	UC2, UC3	B1-6 to B1-8, B5-2, CMN-12	Dialog recovery and application recovery
S7	UC2, UC3	CMN-10, CMN-12, B2-5, B2-6	Existing authorization mechanisms and profiles

S1: Participation Independent of Connectivity For the glasses case, a live interaction needs to distinguish a transport connection remaining available, a device ceasing to provide input, and a participant leaving the dialog. An accepted participation change needs an identifiable scope and outcome. Retaining dialog state during an interruption does not imply that a device remains active or authorized. The dialog protocol handles participant and lifecycle changes. An application profile handles changes among devices represented by a single participant. The observable result is that removing the glasses does not silently end the phone's participation or leave the glasses recorded as an active source.

S2: Modality Changes During an Interaction A live interaction needs a shared account of active input and output modes when they change after establishment. Hardware capability, present availability, and permission are distinct: a video-capable device can be available only for audio or permitted to send only requested still images. A proposed change needs an explicit outcome and a defined point at which the new state applies. When camera capture stops while audio interaction continues, the interoperability question for discussion is how to communicate the affected source and its changed availability to the peer while preserving the continuity of other exchanges. The application profile would define how these changes are represented, as well as fallback or failure when no acceptable mode remains. Existing media-format descriptions should be reused where applicable.

S3: Source and Response Associations Within a Dialog An application needs to associate each relevant flow with its dialog, source instance, modality, and direction. A restart or replacement needs to be distinguishable where delayed data could otherwise be mistaken for current input. The association can be scoped to a flow or exchange; it need not add a dialog-management exchange for each sample or use a global device identifier. The profile also needs to associate an output with the observation or exchange it depends on and define the interpretation of time references and discontinuities. A binding preserves the applicable dialog association across transport boundaries. Correlation inside a dialog can then distinguish a late annotation for an earlier view from output for the current view without requiring a new dialog for each observation.

S4: Scoped Interruption and Operation Outcomes An interruption needs an identified target and intended effect. Stopping local presentation, stopping remote generation, canceling a delegated task, stopping a physical action, and ending a dialog have different outcomes. An application profile needs to distinguish the operations it supports and expose whether a remote request took effect, failed, or remains unresolved. The profile also needs to define races with completion and the treatment of late or superseded output. A response may be superseded while the dialog and other work continue. Deciding that an observation invalidates an explanation is an application function; communicating the resulting scope and state is the interoperability issue. Local silence is not evidence of remote cancellation. Any common exchange-correlation mechanism should be aligned with , Section 3.4.

S5: Availability and Observation Gaps Peers need to distinguish a capture pause, a revoked permission, an unavailable source, and a delivery interruption to the extent necessary to interpret the ongoing interaction. Lack of media alone establishes none of these causes. Application signaling needs to identify the affected scope and make unavailable or unknown observation state explicit. When the signaling path also fails, the profile needs a rule for treating last-known state as stale or unknown. It needs bounded retention and a way to handle repeated or delayed state updates. The observable result is that a cached camera view is not represented as current merely because the dialog or a companion connection remains active. Detailed hardware telemetry is unnecessary when an availability change conveys the required information.

S6: Recovery with Device and Source Changes After dialog recovery, participants need to reconcile the active sources, modalities, output recipients, and unfinished application operations before claiming that the live interaction has resumed in its prior state. Replacing a failed path and replacing a participating device are different events. A replacement device is admitted through an authorized change; it is not assumed to be the same participant solely because it presents a dialog identifier. The application needs to establish which source and control associations remain valid and reject delayed requests that refer to retired authority or state. If continuity cannot be established, restart or termination is explicit. The embodied case combines existing uncertain-operation handling with refreshed observations; if either remains unknown, the application preserves that uncertainty. Recovery does not itself authorize an action retry or reactivation of an earlier sensor.

S7: Authority for Sources, Recipients, and Actions Authorization needs to distinguish consuming a sensor flow, receiving private output, and invoking a controller operation. A device change or revocation needs a defined effect on the affected scope, including already active flows and operations as determined by their application policy. Device ownership or possession of a dialog identifier alone establishes no such permission. Existing identity, delegation, and authorization mechanisms should supply these decisions. The contribution is the live-device example and its scope mapping, not a new credential format or authorization protocol. Consistent with Section 5 of , coordination with OAuth and WIMSE is needed. The dialog layer preserves the appropriate relationship; the application and controller enforce the source, recipient, and operation permissions.

Relationship to Existing Work and Open Questions These use cases provide live-interaction examples for the patterns in and . relates the S1-S7 refinements to existing requirements and to the use cases that motivate them. UC1 and UC2 relate to the single-agent pattern in Section 4.1 of . UC3 combines conversation with delegation and mediated controller access, relating to the patterns in Sections 4.2 and 4.6. These relationships identify common protocol interactions across different device and deployment settings. , Sections 3.3 and 3.4, provides related requirements for cross-channel recovery, exchange correlation, and speech interruption. The live-agent cases additionally examine changing observations, input availability, output recipients, and independently changing action state. The following questions remain for discussion:

1. Which refinements are already supported by the referenced requirements and existing application protocols? Which require clarification or expose a distinct remaining interoperability gap?
2. Which parts of interruption, exchange correlation, and recovery can use common semantics across live-agent and contact-center cases?
3. How are clients, companion devices, observation sources, intermediaries, and controllers represented in the reference architecture? Which state belongs to dialog management, a binding, an application profile, or existing authorization and device specifications?

The live-call and glasses cases provide direct examples of dialog continuity and authorized changes. The embodied case additionally tests the boundary between dialog management and domain-specific operations. Its inclusion does not request that AgentProto standardize physical control or safety behavior. For each refinement, review should establish whether existing application and transport mechanisms already provide the required outcome across independently implemented components, and identify the remaining gap within the charter. The open question is the missing interoperable agreement, not merely whether a particular product can implement the behavior internally. Transport selection and mechanism design are subsequent work. Once the requirements and their allocation are agreed, candidate bindings can be evaluated against camera-off with continuing audio, interruption of one response, source restart, authorized device continuity, and recovery with an unknown action outcome. gives one MOQT-based example; it does not request selection or adoption of a binding in this contribution.

Security Considerations Continuous audio, video, and environmental observations can reveal private information about users and bystanders. Authorization needs to identify permitted sources, recipients, and duration. Local capture control is a device or application function. addresses how changes in source availability are communicated to peers while other exchanges continue. User, device, gateway, and agent identities have different meanings. Reusing a connection, wearing a device, or possessing a dialog identifier is not sufficient proof of permission. Authorization changes need scoped effects, including revocation of downstream access where applicable. Persistent identifiers and detailed capability reports can enable tracking; implementations should minimize their disclosure. Forged associations can direct a camera stream to the wrong dialog. Replayed control requests can silence a current response or restore a former controller's authority. A binding needs confidentiality, integrity, peer authentication, and replay handling appropriate to its trust boundaries, including intermediaries. Correlation identifiers should not implicitly serve as bearer credentials. In embodied systems, conversational participation and physical action authority need separate enforcement. Accepted, completed, canceled, and unknown action outcomes cannot be inferred from network delivery, a spoken acknowledgment, or disconnection. Retry behavior needs to prevent unintended duplicate actions. Local safety mechanisms remain independent of the live interaction. Untrusted media or model output must not acquire the authority of authenticated session signaling merely because it mentions a control operation. Separating these paths does not guarantee model correctness. Implementations also need limits on continuous input, generated output, retained recovery state, and repeated session changes to manage resource exhaustion.

IANA Considerations This document has no IANA actions.

Informative References Work in Progress Work in Progress Work in Progress IETF Work in Progress Work in Progress

MOQT Example for Subsequent Binding Evaluation This appendix illustrates how the use cases can inform later binding and application-profile work. It uses revision -22 of as a fixed comparison point. It does not select a transport, define a wire protocol, or require a MOQT extension.

Existing Transport Functions and Application Questions A MOQT transport session is the association between MOQT peers, with setup, requests, and object delivery governed by . It is distinct from the application interaction and the AgentProto dialog. A relay can terminate a transport session without becoming the user or the agent. MOQT specifies session establishment and termination, extension negotiation, track publication and subscription, priorities, delivery timeouts, and graceful migration. Relevant material appears in Sections 2, 3, 5, 6, and 7. Those facilities provide a delivery foundation for the scenarios above. The questions below concern the application agreements using them. Questions for a MOQT-Based Deployment

MOQT mechanism	Question exposed by the use cases
Session setup and termination	Which participant or device has joined or left the application interaction?
Tracks and publish/subscribe	Which dialog and source instance does a flow represent?
Extension negotiation	Which application profile and active modalities have been agreed?
Priorities and timeouts	Which response is interrupted, and what is its resulting state?
Subscription updates and pause	Has delivery paused, capture stopped, or permission changed?
Migration and relaying	Which application state and authority remain valid after recovery?
Authorization support	Which source, recipient, and operation scopes are authorized?

For example, restoring object delivery does not establish whether a camera remains authorized, or whether stopping playback also stopped generation. An application profile needs an agreement on those effects even when media transport is interoperable. proposes related inference-stream, turn-control, and track conventions. That proposal is relevant input to profile work; this document does not assume that it has been adopted or validated against every later MOQT revision.

Binding and Profile Responsibilities The proposed allocation for discussion is:

• AgentProto supplies dialog correlation and lifecycle semantics.
• A binding specifies carriage of dialog context and its association with the application's signaling and media flows.
• An application profile defines modality state, response relationships, scoped controls, and application recovery using existing mechanisms where suitable.
• Device and domain specifications define sensor representations, actuator operations, and local safety behavior.

The selected profile and binding together need to explain how signaling is located and associated with media, and how independent failures are handled. Application signaling might use an existing application protocol or application-defined MOQT tracks. MOQT's reserved .session namespace in Section 6.5 is for transport session extensions; it is not a general namespace for arbitrary application signaling. Evaluation should consider control delay under media load, setup overhead, ordering scopes, duplicate handling, acknowledgments, and recovery cost. Reliable dialog-control delivery and real-time media delivery have distinct needs, as described in Section 3.1 of . A candidate needs to explain how it satisfies both, including partial failure when control and media follow different paths. Any identified change to MOQT itself should be discussed with the MoQ working group. These questions follow from the scenarios and do not determine which binding the working group should select.