Model-service SLOs and limits: industry practice

Research date: 2026-09-10
Scope: online model serving and offline/asynchronous model training. This note distinguishes reliability objectives from enforceable limits and maps current practice to this project's prototype vocabulary. Sources are standards, official SRE guidance, cloud-provider documentation, and first-party Kubernetes/Envoy/ML-platform documentation.

Executive conclusion

Industry systems do not normally represent every concern as one kind of “limit.” They use several related but operationally different contracts:

Concern	Question answered	Typical unit/window	Typical enforcement
SLI	What outcome did users observe?	Good/valid events, latency-threshold events, freshness, completion	Telemetry calculation
SLO	How reliable must that outcome be?	Target over a rolling or calendar window	Alerts and an error-budget policy, not request rejection
Error budget	How much unreliability may be spent?	1 - SLO, measured over the SLO window	Release/priority/incident policy
Quota	How much shared or purchased capacity may a scope consume?	RPM, concurrent jobs, CPUs/GPUs, tokens, spend	Reject, throttle, or queue
Capability/system limit	What input or operating bound is supported at all?	Per request, dataset, model version, or artifact	Validate before expensive work
Admission control	May this particular unit of work start now?	Instantaneous reservation against all applicable limits	Admit with permit, queue, or decline
Technical failure	Did valid, admitted work fail unexpectedly?	Per attempt/run	Retry/recover/fail with a distinct reason

Google's SRE guidance defines an SLO as a reliability target for customers, recommends ratio-style SLIs, and derives the error budget as 100% minus the SLO. Its enforcement examples are operational actions such as prioritizing reliability or freezing releases after budget exhaustion—not blocking an individual request because the monthly objective is currently missed (Implementing SLOs, example error-budget policy). OpenSLO likewise models an SLO separately from its SLI, objective, time window, budgeting method, and alert policy (OpenSLO specification).

Google Cloud makes a similarly useful limit distinction: quotas are generally adjustable consumption restrictions, while system limits are fixed and cannot be changed. Quotas are usually scoped to a project and often a region; attempted over-consumption is usually blocked (Cloud Quotas terminology, Vertex AI quotas and limits).

How the pieces interact

flowchart TB
    subgraph Control[Control and governance plane]
        CAP[Model Capability Contract]
        POL[Deployment Limit Policy]
        SLO[SLO definitions and error-budget policy]
        DIAG[Diagnostic definitions]
        SNAP[Validated immutable snapshot]
        CAP --> SNAP
        POL --> SNAP
        DIAG --> SNAP
    end

    subgraph Online[Serving data plane]
        REQ[Request] --> PRE[Shape/capability validation]
        PRE --> ADM[Rate, quota, concurrency admission]
        ADM --> RUN[Model execution under permit]
    end

    subgraph Offline[Training data plane]
        JOB[Training request] --> READY[Dataset/capability preflight]
        READY --> QUEUE[Quota reservation and queue]
        QUEUE --> EXEC[Scheduled execution]
        EXEC --> GATE[Artifact quality/publication gate]
    end

    SNAP --> PRE
    SNAP --> ADM
    SNAP --> READY
    SNAP --> QUEUE
    Online --> OBS[Telemetry and Diagnostic Occurrences]
    Offline --> OBS
    OBS --> SLOE[SLO/error-budget evaluation]
    SLO --> SLOE

This separation is visible in mainstream infrastructure. Kubernetes LimitRange validates the shape of a new resource, ResourceQuota tracks aggregate namespace consumption, the scheduler places work according to resource requests, and the kernel enforces runtime CPU/memory limits (LimitRange, ResourceQuota, container resource enforcement). These are separate phases with different failure semantics.

Online model serving

There is no defensible universal target such as “every model API should be 99.9% available and under 500 ms.” Targets come from user tolerance, workload class, measured capability, and cost. As reference points—not templates—the current Vertex AI commercial SLA lists 99.5% monthly uptime for custom-model online prediction on two or more nodes and 99.9% for training/deployment/batch prediction; the SageMaker AI SLA covers multi-instance online inference and batch transform, with availability calculated from 500/503 responses (Vertex AI SLA, SageMaker AI SLA). Internal SLOs should normally be tighter and more user-specific than a provider's service-credit boundary; that last sentence is a design recommendation, not a claim made by either provider.

Typical SLOs and supporting signals

Serving SLOs are normally request-outcome ratios, segmented where user expectations differ:

• Availability: proportion of eligible requests that return a valid response.
• Latency: proportion of eligible requests completed below one or more thresholds; streaming systems commonly add time-to-first-token/chunk and inter-token latency.
• Quality/degradation: proportion served without a fallback or degraded result, when degraded responses are materially worse for the user.
• Correctness/validity: proportion whose result passes a meaningful validity check, when that can be measured independently.

Google SRE recommends good-event ratios and separate objectives for workload classes with different latency needs. It explicitly calls out measuring the proportion of responses served in an undegraded state when fallback behavior exists (service-level objectives, Implementing SLOs). First-party ML platforms expose the expected raw signals: SageMaker publishes invocation count, 4xx/5xx/model errors, concurrency, model latency, overhead latency, and streaming first-chunk latency; Azure ML exposes request rate plus P50/P90/P95/P99 latency (SageMaker endpoint metrics, Azure ML metrics).

Every SLI must define its eligibility denominator. Intentional rejection of an invalid request or an exhausted tenant quota is not automatically a model-execution failure. For example, Vertex AI's commercial SLA counts 500/503 responses for valid requests and excludes quota errors and invalid or misconfigured customer requests. Such denials should still have their own metrics because a high denial ratio can reveal poor capacity planning or user experience (Vertex AI SLA definitions).

QPS, concurrency, queue depth, CPU/GPU/memory/KV-cache utilization, tokens per second, replica count, and autoscaling lag are usually capacity/saturation signals. They drive scaling and admission policy, but are not user-facing SLOs unless the product explicitly promises them. SageMaker, for example, can target-track invocations per instance or concurrent requests per model; concurrency includes requests queued inside the model container and, for streaming, remains active until the last token (SageMaker scaling policy).

Typical service limits

Limit family	Examples	Scope and window	Common mechanism
Request shape	bytes, items, batch size, tokens, dimensions, timeout	Request and model/operation	Synchronous validation before admission
Rate/burst	requests or weighted units	Tenant/project, operation, model, region/cell; second/minute plus burst	Local and/or distributed token bucket
Concurrency	requests, streams, GPU work units	Endpoint/model/capacity pool and tenant	Expiring permit/lease; release or settlement
Queue	queued count, age, wait deadline	Operation/priority/pool	Bounded queue; reject or expire when bound is reached
Resource budget	CPU/GPU time, memory, tokens, bytes	Model, tenant, pool, region	Reserve estimate, then reconcile actual usage
Long-horizon quota	daily/monthly units or spend	Tenant/project/plan	Durable ledger; often looser consistency than capacity safety
Deployment bounds	endpoints, replicas, connections, bandwidth	Subscription/project and region	Control-plane validation/allocation

Real platforms combine several of these. Azure ML documents regional endpoint limits for request rate, connection rate, active connections, bandwidth, replica count, and timeout. Vertex AI documents rate quotas separately from fixed limits such as request/model/dataset bounds (Azure ML quotas and limits, Vertex AI quotas and limits).

Hierarchical scoping is normal: fleet/organization → region or capacity pool → service → model/version → operation → tenant/project → priority class. The exact hierarchy is product-owned, not client-supplied. Envoy's global rate limiter evaluates descriptors through an external service, while a local token bucket can absorb bursts before they reach the global limiter (Envoy global rate limiting). Envoy also separates rate limiting from resource-pressure overload actions such as immediately returning 503 or limiting active connections (Envoy overload manager).

Autoscaling is complementary: it changes future capacity; admission control protects the service while scaling is delayed or impossible. A cloud quota is not a capacity guarantee—Azure states this explicitly—and Vertex notes that exhausted serving compute quota can prevent further autoscaling (Azure ML quota guidance, Vertex AI release note).

Serving response and retry semantics

• Use 429 Too Many Requests when the caller has sent too many requests in a time interval. RFC 6585 says the representation should explain the condition and may include Retry-After; it does not prescribe the counting scope or algorithm (RFC 6585 §4).
• Use 503 Service Unavailable for temporary overload or service unavailability, including an admission plane that cannot provide a trustworthy decision when the route's policy is fail-closed. Retry-After may indicate expected recovery time (RFC 9110 §15.6.4, Retry-After).
• A request that exceeds a static content-size bound can use 413 Content Too Large; other capability/precondition violations need an API-specific mapping rather than being mislabeled as rate limiting (RFC 9110 §15.5.14).
• Retrying should be limited to transient outcomes, use bounded exponential backoff with jitter, and depend on operation idempotency. Permanent validation or capability violations should fail fast (Google Cloud retry guidance).

For the response body, RFC 9457 Problem Details is a good transport projection: its stable type identifies the problem; detail is occurrence-specific and should help the client correct it; machine-readable facts belong in extension fields and should not be parsed from prose (RFC 9457). This aligns closely with this project's DiagnosticOccurrence without making the diagnostic object itself HTTP-specific.

Offline/asynchronous model training

Typical SLOs and supporting signals

Training is treated like a batch/data pipeline, not a low-latency RPC. Typical objectives are:

• Control-plane availability: proportion of valid job submissions/status requests accepted.
• Start latency: proportion of admitted jobs that start within a class-specific queue deadline.
• Completion/freshness: proportion of scheduled training runs that publish an eligible artifact by a business deadline; alternatively, age of the currently servable artifact.
• Successful coverage: proportion of scheduled runs that process the required input scope.
• Correctness: proportion of records/artifacts passing independently defined correctness checks.

Google SRE describes pipeline freshness SLOs as “X% processed within Y,” “oldest data no older than Y,” or “job completed successfully within Y,” and separately discusses correctness and coverage (data-processing pipeline SLOs). The SLO should be defined per service class: an urgent refresh, ordinary retraining, and experimental tuning should not share one start/completion target.

Model quality metrics—precision, recall, NDCG, calibration, bias, or drift—are usually qualification gates or model-health objectives. They become an SLO only when the team can define a user-relevant good event, measurement, target, and window. A single run's “NDCG ≥ X” is a release criterion, not an error-budgeted service reliability objective. As one concrete platform example, SageMaker treats a target objective metric as a tuning-job completion criterion alongside runtime and job-count bounds (SageMaker tuning completion criteria).

Typical training limits and enforcement stages

1. Submission/schema validation: required fields, allowed shapes, supported framework/model configuration, and per-job min/max resources.
2. Model/data preflight: minimum examples/events/labels/history; maximum catalog, feature, vocabulary, file, or dataset size; schema and data-quality requirements. Violations are expected domain outcomes and should prevent expensive work.
3. Quota admission: concurrent jobs, total CPU/memory/GPU/accelerator, storage, trial count, and tenant/team/project allocation. A platform may reject or retain the job in a bounded queue.
4. Scheduling: reserve the complete resource vector, choose a resource flavor/topology, and wait for physical capacity. Kueue defines the workload as its admission unit, reserves CPU/memory/GPU quota, supports tenant queues, priority/fair sharing, borrowing, and preemption (Kueue concepts, ClusterQueue).
5. Runtime enforcement: CPU throttling, memory OOM termination, active/pending/wall-clock deadline, cancellation, checkpoint/restart, and retry-attempt budget. SageMaker exposes maximum pending, runtime, and total Spot wait times; Kubeflow TrainJob can terminate work at an active deadline with a DeadlineExceeded failure reason (SageMaker stopping conditions, Kubeflow TrainJob lifecycle).
6. Artifact gate: validate metrics, completeness, compatibility, and publication preconditions; a completed compute job does not imply a production-eligible model.

Queueing is a deliberate semantic choice, not a universal meaning of quota. Kubernetes ResourceQuota rejects a resource-creation request with 403 when a hard namespace quota would be violated, while Kueue holds a workload until quota reservation, capacity, and admission checks are ready; its check states distinguish retry from rejection (Kubernetes ResourceQuota, Kueue admission checks). Vertex AI also documents product-specific cases where exceeding concurrent batch-job quota queues tasks (Vertex AI quotas and limits).

Implications for this project

The prototype has the right core insight but should not turn one ConstraintDefinition into a universal policy language.

Keep

• ModelCapabilityContract: use it for immutable, model-version-bound supported operating bounds and data prerequisites. This corresponds most closely to fixed system/capability limits, not SLOs or adjustable quotas.
• DeploymentLimitPolicy: use it for environment/cell/tenant operational limits and quota allocations. The “may tighten, never relax capability” invariant is sound.
• DiagnosticOccurrence: use one stable, transport-neutral occurrence model across training and serving, then project it to HTTP Problem Details or the existing API envelope. Preserve typed observed/expected facts; do not require consumers to parse the message.
• Separate evaluators: stateless training preflight and stateful serving admission should share diagnostic vocabulary, not an enforcement engine.

Change before specification approval

1. Add a separate ServiceLevelObjective contract family. It should reference an SLI implementation, good/valid-event criteria, target, rolling/calendar window, error-budget policy, service class, owner, and version. Do not place SLO targets in ModelCapabilityContract or enforce them through AdmissionPrototype.evaluate().
2. Split the current integer-only constraint shape into typed families: static capability predicates, token/rate buckets, concurrency leases, queue/deadline limits, durable quotas, and runtime resource budgets. Common metadata can be shared; algorithms and state cannot.
3. Replace AdmissionDecision.allowed: bool with distinct Granted / Declined / Undetermined outcomes. A policy decline is not a technical failure; inability to decide is not quota exhaustion.
4. Make retry semantics part of the diagnostic/policy definition. The prototype's unconditional "retryable": true is unsafe: insufficient training events or an oversized request will not change through retry, while concurrency exhaustion may.
5. Add scopes, window/algorithm, resource vector, effective policy version, decision time, and optional retry_after to operational limit decisions. Keep trusted identity and target outside caller-controlled measurements.
6. For training, model the durable lifecycle explicitly: Blocked for unmet capability/data prerequisites, Queued/Pending for admissible work awaiting quota/capacity, Failed for an execution failure, and CompletedButIneligible (or equivalent) when artifact qualification fails. HTTP acceptance of an asynchronous job and its later run state are separate contracts.
7. Compile and publish immutable capability/policy/diagnostic snapshots through a control plane; training workers and serving cells load snapshots locally. SLO calculation consumes telemetry asynchronously and must not become a serving hot-path dependency.

Recommended contract boundary

ReliabilityContract
  ServiceLevelIndicator + ServiceLevelObjective + ErrorBudgetPolicy

ModelCapabilityContract
  immutable supported bounds and prerequisites for one model version

DeploymentLimitPolicy
  scoped operational rate, concurrency, queue, quota, resource, and deadline rules

DiagnosticDefinition
  stable code/type, retry semantics, message template, troubleshooting, ownership

DiagnosticOccurrence
  evaluated code, typed evidence, effective versions, correlation, API projection

This is not overengineering if the boundaries remain separate. The lightweight starting point is files plus schema validation and CI, compiled into immutable snapshots. A standalone registry service, generic expression language, and synchronous registry lookups are not required to adopt the contracts.