Dynamic Service Performance Considerations for AI Agent Selection

Introduction Agent services may be supported by different models and computing environments. Two candidate agents may declare the same capability while exhibiting different current response performance; even the same candidate service may perform differently over time. Registration and capability discovery answer which services may meet the basic requirements, whereas service selection determines which candidate is more suitable for a particular request. This document focuses on application-layer inference performance of already discovered candidate agents and on the meaning and limitations of observations such as TTFT and token generation rate. For example, lower TTFT may benefit short responses, while output token generation rate may matter more for long-form generation. Ranking candidates using only one metric may produce a result that is inconsistent with the request objective. Moreover, an agent-level average TTFT observed at a gateway cannot be attributed to a particular internal model without supporting evidence. The objective is to describe considerations for observing and using service performance information, helping existing selection systems avoid direct comparisons of values obtained using different measurement scopes, time windows, or metric semantics.

Scope and Terminology This document assumes that the requester has already obtained a candidate agent set through capability discovery, and that the candidates satisfy the core capability requirements of the request. Service selection here refers to a logical choice among candidate services, not to IP-layer routing or forwarding. Subsequent communication continues to use existing network connections and application protocols. discusses a related multi-model agent registration and discovery approach.

Candidate Agent: An agent service returned by capability discovery that meets the necessary service capability requirements.
Service Instance: A deployed instance that can receive or process requests; there need not be a one-to-one mapping among agents, models, and instances.
Model-level Observation: A performance observation explicitly attributable to a specific executing model or model instance.
TTFT: The time from a defined request start event to receipt of the first output token; measurement endpoints must be stated.
Output Token Generation Rate: The rate at which tokens are produced during the sustained generation stage, with a defined statistical method.
Observation Window: The time interval and sample population used for an aggregate measurement.
Metric Freshness: The elapsed time between observation and use, and an assessment of the measurement’s applicability.

Problem Statement In AI agent service selection, a requester may obtain candidate agents satisfying basic capability requirements through an existing registration and discovery mechanism. Because agent workloads, model inference performance, and network conditions vary, static capability information alone cannot reflect current service performance. Dynamic service performance information can therefore assist subsequent selection. Existing computing-aware service selection work considers dynamic information such as computing load and network latency. AI agent services also exhibit performance characteristics such as first-token response and sustained token generation, and their measurements may originate from different service entities and observation points. Cross-agent and cross-platform selection therefore requires consideration of metric comparability, association with observed entities, and information validity. and related metrics work.

Comparability of Dynamic Service Performance Metrics AI agent performance can be described using TTFT, token generation rate, request success rate, and other metrics. However, different agents or monitoring systems may use different measurement methods and statistical definitions, so metrics with the same name are not necessarily directly comparable. For example, client-side TTFT may include network transfer, request queueing, and model inference, while TTFT measured inside a model service may cover only model processing. Different tokenizers may also produce different token counts for the same text; directly comparing token generation rates may not reflect equivalent amounts of generated content. Accordingly, comparing candidates using dynamic performance requires awareness of the measurement scope and statistical method, to avoid misleading selection outcomes caused by differing metric semantics.

Association Between Performance Information and Candidate Services The externally visible agent service and the entity that actually executes a request may exist at different levels. For example, one agent may expose a common service endpoint while internally invoking multiple models or model instances. An agent gateway can often observe requests and responses passing through it, but may not identify which internal model executed a given request. As a result, gateway measurements may describe the entire agent, whereas measurements reported by a model service may refer to one model or instance. If a selection function cannot distinguish the observed entities, it may incorrectly attribute agent-level performance to a specific model, or treat the performance of one model instance as representative of an entire agent. The observed entity and its association with candidate services therefore need to be understood before the measurements are used.

Validity of Dynamic Service Performance Information Agent service performance changes with load, concurrency, and operating state. Information may be reported by the agent itself or collected by an agent gateway or monitoring system. These sources may use different sampling periods, measurement windows, and update mechanisms. For instance, an agent with low TTFT in the previous window may experience much higher load when a new request arrives. Without knowledge of measurement time and update status, a selector may rely on stale information. Measurements from different sources may also reflect different observation scopes. In particular, processing latency measured within a model service is not necessarily the same as end-to-end latency perceived by the client. A service selection function therefore needs to consider information source, measurement time, and validity, rather than directly comparing values collected under different conditions.

Dynamic Service Performance Metrics for Agent Selection Agent service performance depends on inference capability, request load, runtime environment, and network conditions. Once candidates meet the necessary capability requirements, dynamic performance information may assist in selecting a target agent. Compared with conventional computing services, large-language-model-based agent services often exhibit distinct first-token and sustained-generation phases. A single latency or computing-load metric may not adequately capture these characteristics. This section discusses relevant metrics and considerations for their use. discusses further characteristics of AI agent services.

Time to First Token Time to First Token (TTFT) is the interval between a defined measurement start event and the reception of the first output token. It characterizes initial response performance for generative agent services. TTFT may be affected by network transmission, request queueing, input preprocessing, and model inference. Its scope depends on the observation point: a client-side measurement generally includes network transport, whereas a measurement made inside a model service may exclude transport between the client and server. Candidate comparisons therefore need clearly defined start and end events and an understanding of which stages are included. Values observed at different points should not be compared directly without that context. TTFT can be useful for interactive question answering and short responses, but it does not describe the time required to complete an entire request.

Token Generation Rate Token generation rate describes the rate of output token production during the sustained generation phase, commonly expressed in tokens per second. It may be calculated from the number of generated output tokens and their corresponding generation duration. Measurements need to distinguish whether initial token waiting time is excluded and whether the result represents a single request or an aggregate across requests. The rate varies with model type, batching, concurrency, and runtime resources. Different tokenizers may assign different token counts to equivalent text, so rates from different models do not necessarily have identical application-level meanings. For longer outputs, the rate can help estimate sustained generation time, but it does not guarantee the future speed of a request. Output-length estimates, TTFT, and application requirements should be considered together.

Other Dynamic Service Performance Metrics In addition to TTFT and token generation rate, other dynamic metrics may assist agent selection. Concurrent Requests denotes the number of requests being processed by an observed entity at a given instant or during a measurement window. It can provide context about current load; however, higher concurrency does not necessarily imply worse performance, because service capacities differ. Request Success Rate is the fraction of relevant requests completed successfully within a specified measurement window. Its interpretation depends on the definition of successful completion and on whether timeouts and invocation failures count as unsuccessful requests. Service Availability indicates whether a candidate agent or service instance is currently able to accept and process requests. Availability may be used as a filtering condition, but does not guarantee that latency or throughput requirements can be satisfied. End-to-End Response Time is the elapsed time from the client's request to receipt of the complete response. Unlike TTFT, it reflects overall completion performance. For services involving long tool invocations, human interaction, or streaming output, the event defining completion needs to be clear. These metrics characterize different aspects of current agent service behavior. The relevant subset depends on the request requirements and the measurements available.

Considerations for Using Dynamic Metrics Requests may place different emphasis on performance metrics. Short interactive responses may prioritize TTFT, while long-form generation may consider both TTFT and token generation rate. Requests with stronger reliability needs may also consider success rate and availability. Agent selection therefore need not rely on a single metric or a uniform set of metrics for all requests. Dynamic measurements describe performance under particular conditions and within specific time intervals; they are not guarantees for future requests. Comparisons should account for the observed service, measurement location, measurement window, and information validity.

Dynamic Service Performance Information Collection and Association Capability discovery can identify a set of candidate agents that satisfy basic functional requirements. Dynamic selection additionally needs information about their operating conditions and service performance. Unlike relatively stable capability descriptions, such information may change with load, runtime resources, and network conditions. Measurements may also refer to different observed entities, locations, and time windows. Metric values alone are therefore insufficient: association and measurement context are needed to interpret them correctly. This section discusses information sources, association, and validity. provides related work on metric definitions.

Sources of Dynamic Service Performance Information Dynamic agent performance information may be provided by the agent itself, collected by an agent gateway, or obtained from a service monitoring system. These sources differ in what they can observe. An agent may report its internal operating state, such as current concurrency, availability, and status of associated models. An agent using several models or model instances may also collect model service performance using information about actual execution. An agent gateway can observe requests and responses that traverse it and derive latency, success information, and streaming-output performance. For streaming token output, TTFT and token generation rate can be measured when the gateway can identify request start, first output token, and subsequent output events. A monitoring system can aggregate periodic samples or request records over defined windows and expose the resulting measurements to a service selection function through queries or updates. Sources do not necessarily expose the same metrics. A gateway may observe traffic without knowing which internal model was invoked; a model service may observe internal inference performance accurately but not the client's complete end-to-end delay. The source and its observation limits should therefore be considered. A value not directly observed should not be inferred as a precise measurement at another level solely from available aggregates.

Association with Candidate Agents Agent services may involve several levels of entity, including an agent, a service endpoint, a model, and a model instance. Their dynamic performance can differ. For example, one agent may expose a single document-analysis endpoint and choose among internal models according to the request or local policies. Average response time observed at the gateway then describes aggregate agent behavior, not necessarily the performance of any single model. Performance information should identify its observed entity, using an agent identifier, service endpoint identifier, or another suitable identifier according to deployment conditions. When measurements describe a model or model instance, their association with the candidate agent also needs to be understood. A measurement associated with Agent-A describes that agent's overall behavior within the stated observation scope; a measurement associated with Model-1 inside Agent-A describes that model's behavior. An agent-level selection may use measurements associated with the agent or its external endpoint. Selection among models or instances additionally requires confidence that the observations correspond to those specific entities. If the exact entity cannot be identified, the known aggregation scope can still be retained, but the data should not be treated as a finer-grained measurement.

Measurement Scope and Time Dynamic performance information is collected at particular locations and over particular windows. Comparing different candidates requires an understanding of these measurement conditions. For latency metrics such as TTFT and end-to-end response time, the observer and start and end events should be clear. Client-side TTFT may include network transport, while gateway-side or model-side measurements may encompass only some processing stages. For statistical metrics such as token generation rate and request success rate, the aggregation window and sample population should be understood. Success rate may be computed from recently completed requests, while generation rate may be based on one request or an aggregate of several requests. Observations may also include the observation time, measurement window, and last update time, helping consumers determine the operating conditions to which the data applies. A selector should consider comparability where statistical definitions differ. Values collected over substantially different scopes should not be ranked by their numeric magnitude alone.

Updates and Validity of Dynamic Performance Information Because performance changes with concurrency, model state, and network conditions, stored measurements do not necessarily reflect present conditions. Information may be provided through periodic updates, event-triggered updates, or on-demand queries. Periodic updates show trends but may become stale; event-triggered updates report selected important changes but may not capture every fluctuation; and on-demand queries can obtain more recent information at the cost of additional queries. These mechanisms can be combined as deployment needs dictate. When selecting a service, the consumer can consider observation time, update time, and expected variability. For fast-changing concurrency, older information may be misleading; more stable capability information may tolerate longer update intervals. If some data is missing or no longer applicable, the selector may use other valid information or refresh the observations. Measurements of uncertain validity should not be interpreted as current performance guarantees.

Describing Dynamic Performance Information To facilitate use of performance information across different systems, its description can include the following information:

Observed Entity Identifier: identifies the agent, service endpoint, or other entity to which a measurement applies.
Metric Name and Value: identifies the particular measurement, such as TTFT, token generation rate, or concurrent request count.
Measurement Scope: describes the observation point, start and end events, or statistical conditions.
Information Source: indicates whether the data originates from an agent, gateway, monitoring system, or another source.
Measurement Time Information: specifies the observation time, measurement window, or last update time.

These elements describe the basic semantics of measurements and do not define a new common data format. Existing protocols and management mechanisms may be used for representation and exchange. Some systems may already supply part of this context without retransmitting it; where descriptions are incomplete, consumers need to account for the limits of the available data. Clear attribution and measurement conditions provide a more consistent basis for candidate selection.

Agent Selection Based on Dynamic Service Performance After obtaining a candidate set satisfying the basic capability requirements, a service selection function may use dynamic performance data to evaluate candidate suitability and determine a target agent. Different requests may favor first response, sustained output, or service reliability, so the same candidates may be selected differently according to request characteristics. This section describes the basic selection process and related considerations.

Obtaining the Candidate Agent Set A requester may obtain candidates satisfying basic capabilities through existing agent registration and discovery mechanisms. A request may describe required service capabilities, supported interaction modes, and necessary service constraints. For example, document analysis may require support for a particular document-processing capability and context length. Discovery matches the capability descriptions and returns candidate agents and service information. The selection function can then identify the dynamic metrics relevant to the request. Interactive question answering may emphasize initial response latency; longer text generation may consider both initial response and sustained token generation; and more reliability-sensitive requests may also consider availability and success rate. There is no requirement that every request use the same set of metrics.

Obtaining Dynamic Performance Information for Candidates For the candidate set, the selection function may obtain dynamic performance information from the relevant sources. Agents, gateways, and monitoring systems may provide different metrics and observation scopes. Before use, measurements need to be associated with the candidate service being evaluated. Agent-level selection can use agent- or endpoint-level performance data, whereas instance-level selection requires measurements attributable to the corresponding instance. Measurement scope, statistical window, and last update time also help determine whether the information applies to the current decision. Values with different statistical definitions should not be compared solely on their magnitudes. When information is missing or stale, the selector may refresh it or assess candidates using other suitable information, subject to the request requirements.

Request-Oriented Performance Evaluation Different performance metrics describe different characteristics, and their relevance depends on the request. For requests that emphasize rapid initial output, TTFT can be an important reference. For expected long-form generation, sustained token generation rate may also affect total completion time. Actual performance can additionally depend on load, network conditions, and model operating state. Because token counting can differ across models, generation rate alone is not a sufficient basis for comparing overall agent performance. Where reliability is important, success rate and availability can also be taken into account rather than selecting solely on response speed.

Target Agent Selection A selection function can determine a suitable target agent from capability match results, available dynamic performance information, and current request needs. It may first remove candidates that fail mandatory service constraints or are unavailable, and then evaluate remaining candidates using relevant performance information. The result can include the target agent identifier and service endpoint, after which the requester communicates using existing mechanisms. Observed performance describes behavior in a particular measurement period; selecting an agent does not guarantee that the same performance will be achieved by subsequent requests. The selected agent is not necessarily the same entity as the model or model instance eventually used internally. Where internal model choice is managed by the agent, an external selection function can select based on the exposed agent service without controlling internal model scheduling.

Applicability Considerations Deployments may differ in which performance measurements they can provide. Some agents expose detailed inference metrics, while others only expose availability or end-to-end latency. Selection can use the available metrics without requiring identical metric sets from every candidate. Where measurements are not comparable, the selection function can prioritize those with clear measurement conditions and relevance to the request, avoiding rankings based on incompatible statistics. Missing or stale measurements may be handled using available capability descriptions, basic service status, or other valid information. Missing performance data does not itself imply poor performance or an inability to serve requests. Collecting and refreshing dynamic data also adds signaling and processing overhead; deployments can balance periodic collection, on-demand queries, and reuse of still-applicable measurements based on variability and timeliness needs. In summary, selection based on dynamic service performance should consider request requirements, observed candidate behavior, and applicability of the available measurements. Together with discovery results, appropriate dynamic information provides a more informed basis for selecting a target agent.

Security and Privacy Considerations Dynamic service performance information may reveal sensitive details about service load or operational state and therefore calls for appropriate access controls. Existing authentication and integrity protection can help prevent forged or modified measurements from influencing selection decisions.

IANA Considerations This informational document does not define new protocol parameters, port numbers, message types, or registries. No IANA actions are requested.