NEWOpen AI platform pattern libraryExplore the patterns

Open AI platform pattern library

The patterns behind
production AI
platforms.

A shared engineering language for the decisions that repeat across model serving, agent runtimes, platform delivery, knowledge systems, and production operation.

Explore all patterns Read the platform thesis
18
Open patterns
6
Platform domains
01
Shared language
PATTERN MAP / VERSION 1.0 Production AI platform
OPEN LIBRARY
Foundations 03
  1. AP-01 Thin Platform Slice
  2. AP-02 Separate Control and Runtime Planes
  3. AP-03 Paved Path with Escape Hatches
Inference 03
  1. AP-04 Governed Model Gateway
  2. AP-05 Workload-Aware GPU Scheduling
  3. AP-06 Tiered Model Cache
Agents 03
  1. AP-07 Governed Action Boundary
  2. AP-08 Durable Agent Execution
  3. AP-09 Scoped Agent Identity
Delivery 03
  1. AP-10 GitOps Model Delivery
  2. AP-11 Evaluation Before Promotion
  3. AP-12 Progressive Model Release
Knowledge 03
  1. AP-13 Retrieval Boundary
  2. AP-14 Data Freshness Contract
  3. AP-15 Behavioral Release Manifest
Operations 03
  1. AP-16 End-to-End AI Telemetry
  2. AP-17 Tenant Cost Attribution
  3. AP-18 Failure Containment by Workload
Cloud neutral Model neutral Operations first

Why a pattern language

Make architecture decisions discussable.

AI platform stacks change quickly. The underlying forces repeat. A pattern gives platform, product, security, data, and operations teams a durable way to discuss the same decision.

  1. 01

    Name the decision

    Give teams a precise term for a platform choice that will recur across workloads and environments.

  2. 02

    Expose the forces

    Make the competing needs visible so teams understand why the obvious solution is incomplete.

  3. 03

    Bind trade-offs to signals

    State the operating cost, then define the evidence that shows whether the pattern works in practice.

The library

Patterns for the complete platform path.

Start with a recurring constraint, not a product name. Open any card to examine the context, problem, competing forces, implementation moves, trade-off, and operating signals.

18 patterns shown

AP-01 Platform foundations

Foundations

Thin Platform Slice

Build the smallest complete platform path that can carry one real workload into production.

Use when

You need to move beyond prototypes without designing every future platform capability first.

Open pattern

The recurring problem

Broad platform programmes delay contact with real workloads, while narrow demos omit the operational work that makes a platform reusable.

Forces in tension

  • Speed to first value competes with platform completeness.
  • Workload specificity competes with reusable interfaces.
  • A credible slice must include production operation, not deployment alone.

Apply the pattern

  1. Choose one workload with real security, delivery, reliability, and cost constraints.
  2. Build one end-to-end path across identity, deployment, runtime, and telemetry.
  3. Generalize only after the next workload exposes a repeated need.

Signals to watch

  • Time to first production workload
  • Reuse by the second workload
AP-02 Platform foundations

Foundations

Separate Control and Runtime Planes

Keep policy, delivery, and orchestration independent from the workloads that execute models and agents.

Use when

Multiple AI workloads share platform services but need different runtimes, release cycles, and scaling behaviour.

Open pattern

The recurring problem

When control logic and workload execution are coupled, a runtime change can disrupt governance, and a control-plane failure can stop healthy inference.

Forces in tension

  • Governance needs stable central policy while execution needs independent scale and failure isolation.
  • Routing intent changes through review while per-request decisions must react in real time.
  • Shared interfaces reduce duplication but can couple runtime evolution.

Apply the pattern

  1. Place identity, policy definitions, allowable models and providers, routing weights and defaults, quotas, and desired state in the control plane.
  2. Keep per-request admission, routing, failover, batching, scheduling, inference, agent execution, and retrieval in independently scalable runtime and data planes.
  3. Define versioned interfaces between the planes and test their failure behaviour.

Signals to watch

  • Independent runtime upgrades
  • Control-plane failure tolerance
AP-03 Platform foundations

Foundations

Paved Path with Escape Hatches

Make the safe, observable path the easiest option while preserving explicit routes for exceptional workloads.

Use when

Product teams need self-service AI capabilities, but model and workload requirements differ too much for one rigid template.

Open pattern

The recurring problem

An unrestricted platform creates inconsistent risk, while a closed golden path pushes advanced teams to bypass the platform entirely.

Forces in tension

  • Consistency reduces risk while exceptional workloads need flexibility.
  • Self-service adoption depends on the default path being easier than bypass.
  • Every extension increases the supported operating surface.

Apply the pattern

  1. Automate the common path for identity, deployment, telemetry, and policy.
  2. Document supported extension points and the responsibilities attached to each one.
  3. Review escape-hatch usage and promote repeated exceptions into platform capabilities.

Signals to watch

  • Paved-path adoption
  • Exceptions converted into capabilities
AP-04 Inference runtime

Inference

Governed Model Gateway

Give applications one policy-aware interface for hosted, self-hosted, and specialist models.

Use when

Teams use several model providers and runtimes, each with different APIs, limits, credentials, and economics.

Open pattern

The recurring problem

Direct model integrations scatter credentials and routing logic across products, making policy, migration, and cost control difficult.

Forces in tension

  • Central policy competes with provider-specific capability and latency.
  • Routing needs portability while models expose different semantics.
  • A shared gateway improves attribution but can become a bottleneck.

Apply the pattern

  1. Expose a stable inference contract at the platform boundary.
  2. Centralize identity, quotas, routing, fallback, and usage records without hiding model capabilities.
  3. Keep provider adapters replaceable and make routing decisions observable.

Signals to watch

  • Models changed without product rewrites
  • Requests attributed by team
AP-05 Inference runtime

Inference

Workload-Aware GPU Scheduling

Schedule compute from latency, throughput, memory, and availability needs rather than GPU labels alone.

Use when

Interactive, batch, embedding, and fine-tuning workloads compete for a heterogeneous accelerator fleet.

Open pattern

The recurring problem

Static placement strands expensive capacity, creates noisy neighbours, and assigns the wrong hardware to the workload shape.

Forces in tension

  • Latency needs spare capacity while utilization rewards saturation.
  • Availability needs redundancy while cost rewards fewer accelerators.
  • Model memory, topology, and batching constrain where useful work can run.

Apply the pattern

  1. Describe workloads with service objectives, memory needs, and interruption tolerance.
  2. Maintain hardware-aware queues, priorities, and isolation classes.
  3. Scale from queue depth and inference demand while measuring useful GPU work.

Signals to watch

  • Useful GPU utilization
  • Queue time by workload class
AP-06 Inference runtime

Inference

Tiered Model Cache

Move model weights through registry, object, node, and accelerator tiers with explicit freshness and eviction rules.

Use when

Large models move slowly, replicas scale often, and many nodes repeatedly fetch the same immutable artifacts.

Open pattern

The recurring problem

Uncoordinated downloads extend cold starts, overload registries, and waste network and storage capacity.

Forces in tension

  • Fast startup rewards prefetching while uncertain demand makes prefetched capacity wasteful.
  • Cache locality reduces network cost while every tier adds retention and eviction work.
  • Immutable artifacts improve provenance but can preserve vulnerable weights until policy removes them.

Apply the pattern

  1. Address model artifacts by immutable digest and verify them before use.
  2. Cache popular artifacts close to compute with measured prefetch and eviction policies.
  3. Expose cache state to autoscaling and rollout decisions.

Signals to watch

  • Cold-start duration
  • Cache hit rate by tier
AP-07 Agent runtime

Agents

Governed Action Boundary

Place consequence classification, authorization, validation, approval, and audit between agents and every side effect they can create.

Use when

Agents use APIs, browsers, shells, code interpreters, filesystems, networks, and databases with consequences beyond text generation.

Open pattern

The recurring problem

Direct capability access gives low-risk reads and irreversible actions the same execution path, making unsafe effects difficult to intercept or explain.

Forces in tension

  • Autonomy increases speed while consequence demands proportional human control.
  • A uniform capability interface simplifies agents while actions carry radically different risk.
  • Broad execution power competes with least privilege, isolation, and durable audit.

Apply the pattern

  1. Publish every side-effecting capability through a typed, versioned contract with clear ownership and sandbox boundaries.
  2. Classify actions by consequence: allow low-risk work automatically, evaluate medium-risk work through policy, require approval or dual control for high-risk work, and require strongly authenticated human authorization for irreversible work.
  3. Authorize each action from agent, user, task, and environment context, then validate inputs, constrain effects, and retain durable evidence of the result.

Signals to watch

  • Every side effect crosses an attributable boundary
  • High-consequence actions receive proportional authorization
AP-08 Agent runtime

Agents

Durable Agent Execution

Persist agent state and transitions so long-running work can pause, recover, and resume without replaying unsafe actions.

Use when

Agent workflows span multiple tools, approval points, retries, and time windows longer than one process lifetime.

Open pattern

The recurring problem

In-memory loops lose state during failure and can repeat external side effects when retried from the beginning.

Forces in tension

  • Recovery needs replay while side effects must not be duplicated.
  • Durable state improves control while increasing orchestration complexity.
  • Long-running autonomy needs explicit pause, approval, cancellation, and timeout semantics.

Apply the pattern

  1. Model agent work as explicit, persisted state transitions.
  2. Assign idempotency keys and replay rules to every side-effecting step.
  3. Checkpoint before external actions and expose pause, resume, and cancellation controls.

Signals to watch

  • Recovery without duplicated actions
  • Workflow age and stuck-state visibility
AP-09 Agent runtime

Agents

Scoped Agent Identity

Give every agent run a short-lived identity whose permissions are derived from the user, task, and environment.

Use when

Agents act across systems on behalf of people and services with different permissions and accountability needs.

Open pattern

The recurring problem

Shared service credentials give agents excessive access and make it impossible to attribute or revoke individual actions.

Forces in tension

  • Agents need enough authority to complete work while every run should receive least privilege.
  • Fine-grained attribution improves control while increasing identity integration work.
  • Delegated authority must reflect the initiating user, task, and environment at the same time.

Apply the pattern

  1. Issue a unique workload identity for each agent run or bounded task.
  2. Derive permissions from the initiating principal and reduce them to task scope.
  3. Propagate identity through tool calls, traces, and audit records.

Signals to watch

  • Actions traced to one run
  • Credentials expire with task scope
AP-10 Delivery and assurance

Delivery

GitOps Model Delivery

Promote models, runtime configuration, and policy through reviewed desired state rather than direct cluster changes.

Use when

Model-serving environments change frequently and need the same repeatability and auditability as application infrastructure.

Open pattern

The recurring problem

Manual deployment creates drift, weak rollback paths, and uncertainty about which model and configuration are actually running.

Forces in tension

  • Auditability needs reviewed desired state while runtime control must react without a Git commit.
  • Immutable promotion improves rollback while AI configurations change frequently.
  • Continuous reconciliation reduces drift but must not overwrite valid runtime decisions.

Apply the pattern

  1. Represent deployment intent, model digest, runtime, and policy in version control.
  2. Reconcile environments continuously and report drift as an operational signal.
  3. Promote the same immutable artifact through environments with reviewed configuration changes.

Signals to watch

  • Running state matches declared state
  • Rollback through one reviewed change
AP-11 Delivery and assurance

Delivery

Evaluation Before Promotion

Require workload-specific quality, safety, and performance evidence before a model or prompt reaches wider traffic.

Use when

A model, prompt, retrieval, or policy change can alter product behaviour without changing application code.

Open pattern

The recurring problem

Technical health checks can pass while task quality, safety, latency, or cost regresses in ways users will notice.

Forces in tension

  • Release speed competes with the depth and cost of evaluation.
  • Deterministic checks provide certainty while behavioral quality often needs bounded judgment.
  • Representative evaluation sets decay as products, users, and failure modes change.

Apply the pattern

  1. Version representative evaluation sets with clear owners and acceptance thresholds.
  2. Run deterministic checks first, then bounded model-assisted evaluation where needed.
  3. Attach results to the exact model, prompt, data, and runtime configuration being promoted.

Signals to watch

  • Promotion tied to evidence
  • Production regressions become new evaluations
AP-12 Delivery and assurance

Delivery

Progressive Model Release

Increase exposure in measured stages and keep a fast, tested route back to the previous model behaviour.

Use when

Offline evaluation cannot capture every production prompt, traffic pattern, dependency, and cost interaction.

Open pattern

The recurring problem

A full model cutover turns unknown behavioural and operational risk into an immediate incident for every user.

Forces in tension

  • Faster exposure accelerates learning while increasing the population affected by a regression.
  • Parallel serving improves comparison while consuming additional capacity.
  • Rollback speed depends on preserving the complete previous behavioral configuration.

Apply the pattern

  1. Start with shadow traffic or a narrow tenant cohort before serving broad production traffic.
  2. Compare quality, safety, latency, errors, and cost against the current baseline.
  3. Automate hold and rollback conditions while keeping product owners in the decision loop.

Signals to watch

  • Exposure grows from measured evidence
  • Rollback completes within the service objective
AP-13 Knowledge and data

Knowledge

Retrieval Boundary

Treat retrieval as a governed platform capability with explicit authorization, citations, and quality contracts.

Use when

Many products need grounded access to private, current, or domain-specific knowledge.

Open pattern

The recurring problem

Embedding retrieval inside every application duplicates pipelines and makes access control, freshness, and quality inconsistent.

Forces in tension

  • Shared governance improves consistency while domains need control over ranking and chunking.
  • Fresh retrieval improves grounding while ingestion and indexing consume time and capacity.
  • Authorization must remain current without adding unacceptable request latency.

Apply the pattern

  1. Separate ingestion, indexing, retrieval, and generation behind versioned interfaces.
  2. Enforce source permissions at retrieval time, not only during ingestion.
  3. Return provenance and retrieval diagnostics with every result set.

Signals to watch

  • Answers retain source provenance
  • Permission changes affect retrieval immediately
AP-14 Knowledge and data

Knowledge

Data Freshness Contract

Define how current each knowledge source must be, then measure the full path from source change to retrievable evidence.

Use when

AI products depend on sources that change at different speeds and carry different consequences when stale.

Open pattern

The recurring problem

A healthy ingestion job does not prove that current, authorized content is available to the workload when it is needed.

Forces in tension

  • Lower freshness lag improves correctness while increasing processing and indexing cost.
  • Each pipeline stage needs separate visibility even when users care about one end-to-end objective.
  • Deletion and permission changes may require faster propagation than ordinary content updates.

Apply the pattern

  1. Set freshness objectives per source and use case.
  2. Measure source lag, processing lag, index visibility, and deletion propagation separately.
  3. Degrade, warn, or block when freshness falls outside the agreed contract.

Signals to watch

  • End-to-end freshness objective
  • Deletion propagation time
AP-15 Knowledge and data

Knowledge

Behavioral Release Manifest

Treat model, prompt, retrieval policy, tools, and safety configuration as one traceable behavioral release unit.

Use when

AI behaviour emerges from several independently changing components rather than one model artifact.

Open pattern

The recurring problem

Versioning components separately makes production outcomes difficult to reconstruct and rollback targets incomplete.

Forces in tension

  • Operational reconstructability needs immutable configuration while external providers, tools, and knowledge may remain mutable.
  • A complete manifest improves provenance while increasing coordination across component owners.
  • Stochastic execution allows faithful replay attempts but not guaranteed response reproduction.

Apply the pattern

  1. Create an immutable manifest that identifies every behaviour-shaping component.
  2. Attach evaluation and deployment evidence to the manifest digest.
  3. Log the active manifest for every production request and agent run, including preserved dependency versions or snapshots where available.

Signals to watch

  • Every outcome maps to its exact behavioral configuration
  • Replay is possible against preserved dependencies
AP-16 Operations and governance

Operations

End-to-End AI Telemetry

Connect product intent, model calls, retrieval, tools, infrastructure, quality, and cost in one traceable request path.

Use when

AI latency and failure emerge across application, gateway, runtime, data, and accelerator layers.

Open pattern

The recurring problem

Separate dashboards show component health but cannot explain why one user request was slow, expensive, unsafe, or wrong.

Forces in tension

  • End-to-end explainability needs shared context while services and teams remain independently operated.
  • High-fidelity telemetry improves diagnosis while increasing cost and cardinality.
  • Behavioral evidence can contain sensitive data that privacy policy must minimize and protect.

Apply the pattern

  1. Propagate one trace context across product, gateway, inference, retrieval, tools, and infrastructure.
  2. Record model, release manifest, tenant, token, cache, and evaluation context with controlled cardinality.
  3. Link service objectives to complete user-facing outcomes rather than isolated components.

Signals to watch

  • One request is explainable end to end
  • Quality, latency, and cost share context
AP-17 Operations and governance

Operations

Tenant Cost Attribution

Attribute model, accelerator, storage, and platform cost to the team, product, tenant, and workload that created it.

Use when

Shared AI infrastructure hides which use cases create value and which consume disproportionate capacity.

Open pattern

The recurring problem

Aggregate cloud spend cannot guide routing, capacity, pricing, or product decisions, so optimization becomes guesswork.

Forces in tension

  • Precise allocation improves decisions while shared platform cost is inherently approximate.
  • Lower spend can conflict with quality, latency, and availability objectives.
  • Attribution needs identity across layers that often expose different billing units.

Apply the pattern

  1. Carry tenant and workload identity through gateway, runtime, scheduler, and billing signals.
  2. Allocate shared capacity with a documented model and expose its uncertainty.
  3. Pair cost with quality and service objectives so teams optimize value rather than spend alone.

Signals to watch

  • Cost per successful workload outcome
  • Idle and shared cost made visible
AP-18 Operations and governance

Operations

Failure Containment by Workload

Bound concurrency, dependencies, retries, and resource consumption so one AI workload cannot destabilize the platform.

Use when

Models and agents produce bursty, expensive, and sometimes recursive demand across shared infrastructure.

Open pattern

The recurring problem

Without workload boundaries, retry storms, long contexts, or runaway agents can exhaust gateways, queues, GPUs, and downstream tools.

Forces in tension

  • Burst tolerance improves availability while strict limits protect shared capacity.
  • Retries can recover transient failure or amplify overload and side effects.
  • Higher shared utilization competes with isolation and predictable blast radius.

Apply the pattern

  1. Define quotas, concurrency, time, token, and tool-call budgets per workload class.
  2. Use queues, circuit breakers, and bulkheads at every shared dependency.
  3. Prefer explicit degraded modes over uncontrolled retries and hidden overload.

Signals to watch

  • Blast radius remains within one workload
  • Degraded modes activate before exhaustion

Patterns compose

One workload. Several deliberate decisions.

A production platform is not one pattern. This example path shows how a model-serving workload combines platform, runtime, delivery, and operations decisions without turning them into one coupled system.

  1. 01 AP-01 Thin Platform Slice Study pattern
  2. 02 AP-04 Governed Model Gateway Study pattern
  3. 03 AP-05 Workload-Aware GPU Scheduling Study pattern
  4. 04 AP-10 GitOps Model Delivery Study pattern
  5. 05 AP-11 Evaluation Before Promotion Study pattern
  6. 06 AP-16 End-to-End AI Telemetry Study pattern

Apply the language

Patterns are a starting point, not your architecture.

Bring a workload, a current platform, or a difficult constraint. We will identify the patterns that fit, the trade-offs that matter, and the first assumption worth proving.

Map your platform decisions