Model inference crossed an architectural boundary in 2026.
At meaningful scale, the serving unit is no longer one process running a model behind an HTTP endpoint. It is a topology of encoder, prefill, and decode workers, tensor and expert parallel groups, KV cache tiers, request routers, policy gateways, and independent scaling loops.
The core engineering question has changed with it. Teams are no longer choosing only which engine can produce the most tokens. They are deciding where execution state should live, how requests should find it, which layer owns each decision, and which combinations can be operated safely in production.
This field report maps that system. It separates projects by responsibility, examines six architectural shifts, compares the leading execution and orchestration layers, and closes with four reference deployments and a direct assessment of production maturity.
The 2026 inference stack
The ecosystem is easier to reason about when projects are placed in layers instead of forced into one competitive category.
Read from the product contract down to execution and evidence.
- L7 Application and agent server OGX
Conversations, files, vector stores, durable jobs, moderation, and tools
- L6 AI traffic gateway Envoy AI Gateway
Authentication, quotas, protocol translation, accounting, provider routing, and policy
- L5 Inference routing contract Gateway API Inference Extension
Portable pool APIs, endpoint-selection integration, and conformance
- L4 Model-serving control plane KServe
Declarative deployment, rollout, storage, adapter reconciliation, and lifecycle
- L3 Distributed inference runtime llm-d / NVIDIA Dynamo
Phase placement, KV transport, multi-node coordination, flow control, and scaling inputs
- L2 Model execution engine vLLM / SGLang / TensorRT-LLM
Continuous batching, kernels, quantization, speculative decoding, and local KV management
- L1 Performance and conformance Inference Perf / Kubernetes AI Conformance
Workload models, SLO metrics, trace replay, and testable infrastructure requirements
The most important boundary is between model execution and inference orchestration. An engine can efficiently serve a model on one node or a statically configured cluster. A distributed runtime decides how requests, phases, state, and capacity should be coordinated across a larger deployment. KServe sits above both. It reconciles the desired serving topology but does not replace the engine scheduler or distributed cache plane. (GitHub)
A second boundary separates inference-aware routing from the broader AI gateway. The Gateway API Inference Extension primarily answers which endpoint inside an inference pool should receive a request. An AI gateway answers a wider set of questions: which model or provider is allowed, which credentials apply, which protocol should be translated, how tokens should be counted, and which quota, egress, guardrail, or failover policy should run. (GitHub)
These distinctions matter because a sound platform gives each decision one clear owner. When gateways, schedulers, control planes, and engines all attempt to own the same placement or retry decision, the system becomes difficult to explain and even harder to recover.
Six architectural shifts defining inference in 2026
1. The model replica is no longer the serving boundary
Traditional serving systems mapped one request to one replica. The replica owned execution from prompt ingestion through the final token. Large reasoning, mixture-of-experts, and multimodal models have weakened that abstraction.
Prefill and decode have different resource profiles. Prefill processes many input tokens in parallel and tends to be compute intensive. Decode repeatedly evaluates a small number of new tokens while reading a large KV state, making memory bandwidth, cache locality, and synchronization dominant.
Separating the phases lets each pool use different batch sizes, parallelism strategies, hardware shapes, scaling rules, and scheduling policies.
The architecture is expanding further into encoder, prefill, and decode disaggregation. Multimodal encoders can run on dedicated workers, pass their outputs to prefill workers, and avoid tying expensive language-model capacity to image, audio, or video preprocessing. vLLM 0.28 includes encoder, prefill, and decode work alongside continued prefill and decode hardening, while SGLang and llm-d expose increasingly mature disaggregated paths. (GitHub)
This changes what a deployment represents. A model service may contain several worker roles with independent lifecycle and capacity. Scaling the model becomes an ambiguous instruction. The system may need more prefill workers, more decode workers, more experts, or simply better placement of existing KV state.
2. KV cache is becoming shared infrastructure
Paged attention made KV memory manageable inside an engine. The 2026 challenge is managing that state across processes, nodes, execution phases, and storage tiers.
The state plane must answer distributed-systems questions:
- Which worker or tier owns a prefix?
- How fresh and complete is it?
- Is transfer cheaper than recomputation?
- Can several requests reuse it safely?
- How is tenant isolation preserved?
- What happens when a producer fails during transfer?
- Which state should be evicted when GPU, CPU, and disk have different costs?
vLLM 0.28 extends multi-tier KV offload with disk-backed tiers, external secondary-cache managers, metrics, and a canonical CPU representation. SGLang combines its radix-based reuse model with HiCache, Mooncake, and SSD-backed storage. llm-d integrates HMA and Mooncake-oriented paths into its disaggregated architecture. Dynamo uses NIXL-based transport and adds routing knowledge about cache location across replicas and datacenters. (GitHub)
The result increasingly resembles a specialized distributed database. GPU memory is the hot tier. CPU memory is the warm tier. Local or shared storage is the colder tier. The router acts partly as a query planner.
There is one important difference from a conventional database: recomputation remains an alternative to retrieval. The scheduler must compare computation, transfer time, queue delay, and expected future reuse.
3. Routing is becoming execution-state aware
Inference routing originally meant selecting a healthy replica. Current systems treat the request body and execution state as scheduling inputs.
KServe’s LLMInferenceService can route by model name and expose model availability through status. Its LoRA work moves adapter affinity into placement decisions rather than treating every replica as equivalent. llm-d supports data-parallel-aware scheduling, prefix-cache signals, latency prediction, and agent-oriented policies. Dynamo 1.4 adds cross-datacenter prefix routing, tenant-isolated cache-salt-aware routing, and session affinity synchronized across router replicas. (GitHub)
A practical routing hierarchy now looks like this:
- Resolve the tenant, credentials, API contract, and policy.
- Resolve the requested model, adapter, or capability.
- Identify candidate inference pools and regions.
- Estimate queue delay, phase capacity, and predicted latency.
- Account for reusable prefixes, session state, and transfer cost.
- Select a worker or distributed execution topology.
- Preserve streaming, cancellation, retries, and accounting through completion.
The router is becoming part of the inference runtime, not an interchangeable network proxy. This is why generic gateway implementations increasingly delegate endpoint selection to an external processor or scheduler.
4. The workload is now a session, not merely a request
A conventional benchmark sends independent prompts and measures aggregate throughput. Agentic and reasoning workloads behave differently:
- Requests arrive in correlated, multi-turn sessions.
- Large prefixes recur with small modifications.
- Tool calls introduce pauses and external dependencies.
- Reasoning budgets change output length and speculative-decoding behavior.
- Context may move between workers.
- One user operation may generate several model calls.
- Streaming interruption and recovery affect completion quality.
Inference Perf 0.6 added stronger support for multimodal and agentic workloads, including session simulation, tool replay, trace-driven execution, affinity, and context recovery. Its metrics include time to first token, time per output token, inter-token latency, request distributions, throughput, and goodput under defined objectives. (GitHub)
This shift also changes autoscaling. GPU utilization is often a lagging or misleading signal. A decode pool can be saturated by active sequences without appearing compute bound. A prefill pool can create severe time-to-first-token spikes in short bursts. Cached workloads can have radically different costs from uncached ones.
Token rates, queue depth, active sequences, predicted completion latency, cache state, and role-specific saturation are becoming more useful scaling inputs.
5. The API boundary is moving above the engine
OpenAI-compatible endpoints have become table stakes, but compatibility with one API shape no longer describes the complete application contract.
Production platforms also need Responses-style interactions, Anthropic Messages, asynchronous batches, file ingestion, vector stores, conversation state, multimodal inputs, tool execution, provider fallback, and MCP connectivity.
Envoy AI Gateway 1.0 established a stable v1beta1 control-plane API and supports an OpenAI-oriented interface across several providers, protocol translation, MCP routing, multimodal traffic, and quota-aware policy. Version 1.1 added broader token counting, credentials selected per request, stream idle handling and failover, optional OpenTelemetry generative AI tracing, and more capable MCP routing. (GitHub)
OGX occupies the layer above engines and gateways. Its 1.3 line develops provider abstraction, conversations, files, vector stores, durable file-processing jobs, reranking, and higher-level application APIs. It should be evaluated against application and agent servers, not against vLLM or SGLang kernel performance. (GitHub)
6. Validated compositions are replacing latest everything
Engine projects release rapidly because support for new model architectures and accelerator kernels is highly competitive. Distributed runtimes and platforms move more deliberately because they must validate combinations of engines, transports, schedulers, metrics, and deployment manifests.
For example, llm-d 0.9 tracks a specific upstream vLLM line and includes component versions that do not all match the umbrella release. Its router components have reached 0.10 while other components remain on 0.9-era releases. Dynamo 1.4.2 similarly packages validated versions of vLLM, SGLang, and TensorRT-LLM that trail the newest standalone engine tags. (GitHub)
The relevant production artifact is a tested bill of materials:
- engine and model implementation;
- CUDA, ROCm, XPU, or CPU runtime;
- collective-communication libraries;
- KV-transfer implementation;
- scheduler and router;
- model-serving custom resources;
- gateway and extension versions;
- observability schema;
- driver, firmware, and accelerator generation.
An upgrade to any component can alter memory layout, cache compatibility, scheduling signals, metrics, or model correctness. Current should mean the newest validated composition for a workload, not the newest independent release of every project.
Release snapshot on 2 September 2026
Ten projects across five layers
Published release lines at the time of this field report.
The version table is a snapshot, not a recommendation to upgrade each component independently. Production teams should start from the compatibility matrix published by the higher-level runtime or control plane they intend to operate.
Model execution engines
vLLM: broad platform coverage and an expanding serving core
vLLM remains the broadest general-purpose open-source inference engine in the ecosystem. Its trajectory is no longer limited to continuous batching and paged attention. The project is progressively absorbing capabilities that once belonged to an external serving system: distributed role separation, cache-tier management, request rendering, gRPC transport, reasoning controls, and lifecycle hooks for reinforcement-learning workflows.
Version 0.28 advances several major tracks:
- encoder, prefill, and decode disaggregation;
- Model Runner V2 expansion;
- tiered KV-cache offload, including disk and external secondary tiers;
- adaptive speculative-decoding budgets;
- standalone rendering and a developing Rust and gRPC frontend;
- data-parallel-rank routing and distributed lifecycle support;
- further Kimi-K3 and DeepSeek V4 kernel and parallelism optimization. (GitHub)
Model Runner V2 is strategically important. It is not merely a refactor of model code. It is intended to provide a more modular execution path across dense, mixture-of-experts, hybrid-attention, multimodal, and speculative models. The project is still moving models onto that path incrementally, so MRv2 should not yet be treated as a uniform default for every architecture.
The experimental Rust frontend reflects another 2026 pattern. Tokenization, request normalization, streaming, routing, and other latency-sensitive control work are moving away from a monolithic Python process. This allows the model engine to concentrate on scheduling and execution while frontend components scale independently.
Best fit: Teams needing broad model coverage, several hardware backends, rapid support for frontier architectures, and a path from simple serving to llm-d, Dynamo, KServe, or a custom distributed control plane.
SGLang: prefix reuse, session state, and aggressive systems optimization
SGLang’s strongest differentiation is the integration of structured generation, prefix reuse, cache-aware scheduling, and high-performance model execution. Its radix-based approach is particularly relevant to multi-turn chat, agents, few-shot workloads, and reinforcement learning, where large prompt segments recur across related calls.
Version 0.5.18 continues high development velocity across checkpoint staging, distributed communication, Blackwell and AMD optimization, and non-autoregressive modalities. SGLang is also expanding beyond language-model serving into image and video generation infrastructure. (GitHub)
The Unified Radix Cache and session-reference work make application-level session identity visible to the cache system. HiCache adds a hierarchy beyond accelerator memory, with Mooncake-oriented and SSD-oriented paths. The decode side of disaggregated serving can preserve and exploit prefix state rather than assuming that all reusable context must remain attached to the prefill worker. (GitHub)
SGLang is also developing a native Rust server with multimodal support, OpenAI-compatible APIs, and disaggregated operation. Related work on piecewise CUDA graphs, decode context parallelism, heterogeneous CPU and GPU execution, and fast restart through a weight-cache daemon demonstrates how much of 2026 inference optimization concerns startup, control flow, and data movement rather than matrix multiplication alone. (GitHub)
Best fit: Workloads with substantial prefix reuse, long-lived sessions, agent loops, reinforcement-learning generation, or teams willing to follow a fast-moving system for strong end-to-end performance.
TensorRT-LLM: the NVIDIA-specific optimized path
TensorRT-LLM remains the most vertically integrated open-source route for teams committed to NVIDIA hardware and prepared to optimize against specific GPU, CUDA, model, and kernel combinations.
The 1.3 release-candidate line introduces or develops:
- KV Cache Manager V2 as the recommended path for major model families;
- improved multimodal KV-block hashing;
- disaggregated-serving cache reuse for hybrid models;
- expanded speculative and multi-token prediction support;
- context-parallel and ring-attention work;
- additional OpenTelemetry-compatible instrumentation. (GitHub)
The qualification is operational maturity. The latest package at the time of writing is 1.3.0rc25 rather than a final 1.3 release. Its published known issues include configuration-specific hangs, shutdown failures, out-of-memory conditions, and disaggregation problems on some B200 and distributed deployments.
This does not make the engine unsuitable for production. It means adoption should be based on a narrowly validated model and hardware matrix rather than the nominal feature set of the release line. (GitHub)
Best fit: NVIDIA-standardized deployments where maximum platform-specific optimization justifies tighter coupling to supported hardware, models, and software combinations.
Distributed inference systems
llm-d: Kubernetes-native composition around inference engines
llm-d is not another model engine. It is a collection of components and deployment patterns for turning engines into a distributed inference service.
The project now spans:
- inference routing and endpoint selection;
- prefill and decode disaggregation;
- KV-cache indexing and transfer;
- workload-variant autoscaling;
- latency prediction and flow control;
- asynchronous inference and batch APIs;
- observability;
- reference model-service deployments.
The top-level 0.9 release arrived on 17 August 2026, while some router components independently advanced to 0.10. This component-level versioning is intentional, but it makes compatibility documentation and installation manifests part of the product rather than secondary packaging concerns. (GitHub)
During the 0.8 and 0.9 cycles, llm-d added production multimodal paths, batch and flow-control components, non-Kubernetes FileDiscovery, Responses API support, multi-tier KV offload, HMA and Mooncake integration, data-parallel-aware scheduling, predicted-latency signals, and initial agentic-workload routing.
Non-Kubernetes discovery is particularly important for reinforcement-learning clusters, Slurm environments, and specialized GPU fleets where Kubernetes may not own the execution layer. (GitHub)
The project is also where the original feature-rich Gateway API Inference Extension endpoint picker, latency predictor, and related scheduling components have moved. This leaves GAIE focused on portable contracts and conformance while llm-d carries the heavier production implementation. (GitHub)
Best fit: Kubernetes-oriented teams that want a composable, community-driven distributed serving architecture, close alignment with KServe and Gateway API, and the ability to replace individual scheduling, cache, benchmarking, and autoscaling components.
NVIDIA Dynamo: an integrated multi-engine inference fabric
Dynamo also sits above model engines. It supports vLLM, SGLang, and TensorRT-LLM and coordinates multi-node serving, disaggregation, KV-aware routing, cache transfer, and scaling.
The project explicitly distinguishes the simple case, where an engine alone is sufficient for one model on one GPU, from larger deployments that require an inference fabric. (GitHub)
Version 1.4 introduced capabilities that illustrate Dynamo’s direction:
- cross-datacenter and hierarchical routing;
- sequenced KV relay;
- peer reservation and replay;
- endpoint-scoped transport;
- token-in and token-out execution interfaces;
- tokenizer-side prefix caching;
- session affinity across router replicas;
- tenant-isolated, cache-salt-aware routing;
- multimodal state transfer through NIXL. (GitHub)
Version 1.4.2 is primarily a patch release, including NIXL loader fixes and initial Dynamo Enterprise artifacts. Its packaged engine versions remain deliberately behind the newest standalone releases. Dynamo is distributed as a validated system composition rather than a transparent wrapper around arbitrary engine versions. (GitHub)
Best fit: NVIDIA-centered infrastructure that needs an integrated distributed runtime across several supported engines, advanced cache movement, multi-region or multi-datacenter routing, and close alignment with NVIDIA networking and accelerator technology.
llm-d and Dynamo overlap, but they are not equivalent
Both projects orchestrate distributed inference, but they begin from different centers of gravity.
llm-d is organized as a Kubernetes-friendly, multi-project component ecosystem with close ties to KServe, Gateway API, workload autoscaling, and replaceable routing or benchmarking components.
Dynamo presents a more integrated NVIDIA-led fabric with NIXL transport, engine adapters, planner functionality, and cross-datacenter state routing.
The practical choice is unlikely to be determined by one benchmark. It depends on the surrounding platform:
- Kubernetes API alignment compared with a specialized inference runtime;
- community-standard interfaces compared with vertically integrated transport;
- replaceable components compared with a validated end-to-end distribution;
- heterogeneous accelerator goals compared with NVIDIA infrastructure depth;
- single-cluster deployment compared with a broader datacenter topology.
Kubernetes model serving and standards
KServe: model lifecycle rather than kernel execution
KServe 0.20 continues the shift from a generic prediction-serving abstraction toward a control plane capable of expressing distributed generative-model deployments.
The active center is LLMInferenceService, which now includes stronger model-name-based routing, explicit model status and routing gates, static LoRA reconciliation, adapter-affinity behavior, support for several OCI storage sources, and more flexible scheduler creation during rolling workload updates. (GitHub)
KServe’s specification is also becoming more explicit about distributed-runtime concerns. Recent work includes:
- CPU-backed KV-offload configuration;
- latency-predictor integration;
- explicit prefill and decode worker roles;
- secondary filesystem cache tiers;
- migration toward llm-d-owned routing APIs;
- confidential model-serving support. (GitHub)
KServe 0.20 was released with versions of GAIE and llm-d that preceded their later August releases. Operators should follow KServe’s tested integration matrix instead of independently upgrading every custom resource and router component.
KServe’s durable role is deployment intent: which models, adapters, storage sources, worker roles, rollout policies, and serving graphs should exist. The runtime beneath it remains responsible for execution, cache movement, and request scheduling.
Gateway API Inference Extension: a contract, not a full scheduler
GAIE 1.6 marks a significant clarification of project scope.
The original full endpoint picker, latency predictor, and associated implementation-heavy APIs have moved into llm-d. GAIE now concentrates on:
- portable inference-pool APIs;
- integration with Kubernetes Gateway API;
- a lightweight endpoint-picker reference implementation;
- conformance behavior;
- implementation-neutral extension points.
The lightweight endpoint picker primarily serves as a minimal external-processing and conformance reference. It is not intended to reproduce the complete production scheduler now developed under llm-d. Several alpha APIs, including inference objectives, model rewrites, and endpoint-picker configuration, were removed or migrated during this simplification. (GitHub)
This separation is healthy. A Kubernetes standard should define interoperable behavior without requiring every gateway to adopt one scheduler implementation. Production projects can innovate rapidly in cache scoring, predicted latency, fairness, and flow control while GAIE stabilizes the contract between gateway and scheduler.
Kubernetes AI Gateway working group: standards remain in development
The Kubernetes AI Gateway working group addresses a broader layer than GAIE. Its scope includes AI protocol awareness, egress to model providers, payload processing, token-based controls, traffic inspection, routing, caching, and guardrail integration.
The group explicitly describes its repository implementations as prototypes rather than production-ready gateways. (GitHub)
As of 2 September, the PayloadProcessor specification, reference implementation, and CEL-based vocabulary remain active work in progress. Rate-limiting and multi-stage-routing proposals are also evolving. PayloadProcessor should be described as an emerging Gateway API design, not an established standard resource. (GitHub)
AI Conformance: requirements are becoming testable
The AI Conformance effort is moving toward a Kubernetes-style requirements process based on Kubernetes AI Requirements, known as KARs, and aligned with the KEP model. The initial work targets the Kubernetes 1.36 cycle and defines how requirements can be proposed, reviewed, and attached to conformant AI infrastructure. (GitHub)
Beginning with the 1.37 process, new SHOULD and MUST requirements are expected to include automated tests as a prerequisite. This is an important transition from descriptive checklists to executable platform contracts. (GitHub)
The difficult questions remain architectural rather than syntactic. Accelerator discovery, topology, gang scheduling, model lifecycle, distributed checkpoints, inference state, agent tool access, workload identity, and failure semantics do not fit cleanly into traditional stateless workload conformance.
Inference Perf: from benchmark prompts to workload models
Inference Perf 0.6.1 is developing into a shared workload-description and measurement layer rather than another isolated load generator.
Its current capabilities include:
- server-reported token counts;
- Prometheus runtime metrics;
- per-request latency distributions;
- session-level and request-level errors;
- on-demand trace sessions;
- more faithful tool-call replay;
- shared-prefix and multi-turn workloads;
- multimodal requests;
- time to first token, time per output token, inter-token latency, throughput, and SLO-constrained goodput. (GitHub)
Version 0.6 also added OpenTelemetry-oriented and Weka-oriented trace replay, agentic-session simulation, affinity, and context-recovery behavior. This makes it increasingly useful for evaluating routers and distributed runtimes, not only engines. (GitHub)
Gateway and application plane
Envoy AI Gateway: the concrete AI traffic layer
Envoy AI Gateway reached its 1.0 general-availability line in June and released 1.1 in August. The 1.0 release stabilized control-plane APIs including AIGatewayRoute, AIServiceBackend, BackendSecurityPolicy, GatewayConfig, and MCPRoute.
It also established cross-provider API translation, multimodal handling, MCP gateway functionality, and tenant-aware or quota-aware routing. (GitHub)
Version 1.1 expands the operational layer with:
- provider-independent token counting;
- credentials selected per request;
- stream idle timeouts and failover behavior;
- MCP hostname and CEL-driven routing;
- optional OpenTelemetry generative AI traces;
- HTTP CONNECT support for controlled egress. (GitHub)
Envoy AI Gateway does not replace an inference-aware endpoint scheduler. It can route to providers, clusters, models, or inference pools and then delegate replica-level selection to GAIE-compatible or runtime-specific components.
A representative request path is:
- The AI gateway owns authentication, tenant policy, provider translation, quotas, token accounting, egress, and model or region selection.
- The inference pool and distributed router own model availability, adapter affinity, prefix locality, predicted latency, queue state, and phase-aware worker selection.
- The model engine owns scheduling, batching, kernels, and execution on the selected worker topology.
- The KV hierarchy carries reusable state across GPU, CPU, filesystem, or object-backed tiers.
The layers can share signals, but they should not obscure ownership.
OGX: an application server above the serving layer
The former Llama Stack project now ships as OGX. Its latest release is 1.3.1, published on 1 September, following the feature-bearing 1.3.0 release in August. (GitHub)
OGX’s scope includes:
- OpenAI-oriented and Anthropic-oriented application APIs;
- conversations and response state;
- files and file processing;
- vector stores and retrieval;
- asynchronous and batch jobs;
- moderation and guardrails;
- tool and agent integration;
- remote model-provider abstraction.
The 1.3 line adds provider and token-counting work, durable out-of-process file-processing jobs, classifier-based reranking, and additional vector-store integrations such as Neo4j. (GitHub)
OGX belongs in the inference landscape because application state, files, retrieval, background processing, and provider abstraction determine how model calls are produced. It remains categorically separate from the execution layer.
Replacing vLLM with SGLang changes kernels, batching, and cache behavior. Replacing OGX changes application APIs, storage semantics, provider integration, and agent orchestration.
Four reference deployment patterns
There is no universal production stack. The correct architecture follows the scale, tenancy, workload shape, state-reuse potential, and operational capability of the team.
Pattern 1: direct engine serving
- Client or application Request boundary
- Model engine vLLM / SGLang / TensorRT-LLM
- GPU node Execution and local KV state
This remains the correct architecture for one model on one node, or a small number of nodes, with moderate traffic and limited multi-tenancy. Adding a distributed runtime prematurely increases failure modes, version constraints, and observability requirements.
Dynamo’s own architecture documentation explicitly distinguishes this case from deployments that require a larger inference fabric. (GitHub)
The engine should still provide production basics: health and readiness, bounded queues, streaming cancellation, API authentication, request metrics, model warm-up, deterministic configuration, and a tested restart path.
Pattern 2: high-volume shared model
- AI gateway Tenant and traffic policy
- State-aware router Cache and predicted latency
- Prefill pool Compute intensiveDecode pool Memory intensive
- Tiered KV state GPU / CPU / storage
This pattern fits a large shared model with enough concurrency to justify independent prefill and decode scaling. The central design questions are KV-transfer cost, queue isolation, failure recovery, and how much traffic has reusable prefixes.
llm-d, Dynamo, vLLM, and SGLang all provide parts of this topology, but reliable combinations remain version specific and model specific. (GitHub)
Disaggregation is not automatically faster. It trades local execution simplicity for independent scaling and potentially higher utilization. Poor network topology, small prompts, low concurrency, excessive transfers, or insufficient cache reuse can erase the benefit.
Pattern 3: Kubernetes multi-model platform
- Envoy AI Gateway Tenant-facing traffic policy
- Gateway API and GAIE Routing contract
- KServe Model lifecycle and desired state
- llm-d Distributed scheduling and state
- Validated engine Execution and kernels
This pattern gives each platform concern a distinct owner:
- Envoy AI Gateway owns tenant-facing traffic policy.
- Gateway API and GAIE provide routing contracts.
- KServe owns model deployment and lifecycle.
- llm-d owns distributed scheduling and state-aware routing.
- The engine owns model execution and kernels.
It is the strongest open Kubernetes-native composition in 2026, but it also carries the most demanding compatibility matrix. KServe, GAIE, llm-d, the engine, custom resources, charts or Kustomize overlays, and observability conventions need to be upgraded as a tested unit. (GitHub)
Pattern 4: multi-provider agent platform
- Agent application or OGX Conversations, files, and tools
- Envoy AI Gateway Policy, credentials, and accounting
- Remote providers Managed model APIsLocal inference pools Private model capacity
- Distributed runtime llm-d or Dynamo
- Model engines Local execution
This pattern fits organizations combining self-hosted models with external APIs. The gateway normalizes policy, credentials, accounting, and provider failover. OGX or an equivalent application layer manages conversations, files, retrieval, and jobs. The distributed runtime optimizes local execution. (GitHub)
The central risk is duplicated state. Conversation history may exist in the application server, prefix state in the inference cache, request state in the gateway, and tool execution state in an agent framework.
Stable identifiers and explicit ownership are required to avoid inconsistent retries, duplicate execution, lost affinity, and cross-tenant leakage.
Production maturity in September 2026
What is ready, and what still needs proof
Architectural assessment based on current capabilities and documented limitations.
A standard engine capability, still sensitive to model and backend.
Production ready inside workload-specific topology and collective configurations.
Most effective when applications expose stable repeated prefixes.
Loading, affinity, eviction, and rollout still need explicit policy.
Benefits depend on concurrency, prompt shape, network, and transfer path.
Especially relevant for multimodal traffic, with interfaces still evolving.
Several implementations exist, but recovery and interoperability differ.
Available in specific stacks without a universal state protocol.
Depends on continuously calibrated workload models and trusted signals.
More useful than generic GPU utilization, with scale-down still difficult.
Stable implementations cover translation, credentials, quotas, tracing, and MCP.
Tool calls, pauses, retries, and session state are entering standard benchmarks.
This maturity map is an architectural assessment based on current project capabilities and their documented limitations. The strongest production baseline remains a carefully pinned engine deployment.
Distributed state, phase disaggregation, and state-aware routing are usable, but their reliability depends more heavily on the exact workload, topology, and release composition. (GitHub)
Open problems shaping the rest of 2026
Portable inference-state interfaces
Every major distributed stack is developing its own cache managers, metadata models, transports, and routing signals. vLLM has external secondary-cache managers. SGLang has HiCache and radix-based session semantics. llm-d combines cache indexing with several transfer paths. Dynamo uses NIXL and its own routing fabric. (GitHub)
The missing abstraction is not another completion API. It is a portable contract for describing reusable inference state:
- model, tokenizer, adapter, and cache-layout identity;
- token or multimodal-prefix identity;
- state location and completeness;
- ownership, lease, and isolation;
- transfer capabilities and expected cost;
- invalidation and failure semantics.
Without that contract, gateways can standardize endpoint selection while remaining blind to the most valuable state in the system.
Failure semantics for disaggregated execution
A failed monolithic request can be retried against another replica. A failed disaggregated request is harder. The encoder may have completed, prefill may have produced partial KV state, decode may have streamed tokens, and cache transfers may still be in flight.
Systems need explicit answers for:
- whether partially transferred state is addressable;
- whether decode can resume elsewhere;
- how streamed output constrains retries;
- how reservations expire;
- how cache references survive worker replacement;
- which component owns cancellation;
- how accounting distinguishes completed, resumed, and abandoned execution.
The known-issue profiles in current release-candidate and distributed-serving stacks show that startup, shutdown, failure recovery, and coordination remain at least as important as steady-state throughput. (GitHub)
Autoscaling coordinated with state movement
Scaling decode workers down can discard valuable KV state. Scaling prefill workers up may increase transfer pressure instead of reducing time to first token. Moving sessions can improve balance while destroying locality.
- 01 Queue delay +
- 02 Compute time +
- 03 State transfer +
- 04 Recomputation +
- 05 Cold start
The emerging direction is correct: role-specific metrics, workload variants, predicted latency, cache knowledge, and integration with Horizontal Pod Autoscaler or KEDA. Stable coordination between scheduler, cache plane, and autoscaler is still developing. (GitHub)
Tenant isolation below the API layer
Authentication at the gateway is insufficient once state is reusable. Prefix caches, adapter caches, multimodal embeddings, speculative state, and conversation affinity can all become cross-tenant channels.
Dynamo’s cache-salt-aware routing and OGX’s tenant-aware application storage illustrate two sides of the problem: execution-state isolation and application-state isolation.
KServe, gateways, and distributed runtimes need an end-to-end tenant identity that survives routing, caching, batching, background jobs, and observability without exposing sensitive prompt structure. (GitHub)
Reproducible goodput instead of benchmark peak
Engine benchmarks often use idealized prompt distributions, fixed output lengths, warmed caches, and homogeneous concurrency. Production traffic includes mixed tenants, tool pauses, multimodal inputs, retries, long-lived streams, and rapidly changing prefixes.
Inference Perf’s session, trace-replay, and SLO work points toward a better standard. Publish the workload distribution, cache conditions, topology, failure behavior, and percentage of requests meeting the latency objective, not only the maximum aggregate token rate. (GitHub)
The operating conclusion
The durable story of model inference in 2026 is not that one project won.
It is that execution, state, routing, lifecycle, traffic policy, application state, and performance evidence have become separate architectural concerns. The industry is learning where those boundaries belong and which signals must cross them.
Three principles are already clear:
- Start with the simplest topology that satisfies the workload. An engine on a well-operated node is better than an unnecessary distributed fabric.
- Treat reusable inference state as a first-class production concern. Placement, isolation, transfer, recovery, and accounting now shape the service objective.
- Deploy tested compositions, not a collection of latest tags. The compatibility matrix is part of the architecture.
The next durable standard in model inference is likely to be a portable contract for execution state, locality, and service objectives, not another HTTP completion schema.
For a broader view of how inference composes with agent, delivery, knowledge, and governance concerns, read the AI-Native Platform Patterns field guide or explore the complete AI Platform Pattern Library.