Skip to content

Proposal: shared serving admission control

Status: Initial vertical slice; production policy and distributed state are not approved or qualified.
Date: 2026-09-10
Scope: Admission control in front of the commerce recommendation API and, eventually, hundreds of independently deployed model-serving APIs.

1. Outcome and interpretation

This proposal interprets a Service Limit System as request admission control: it protects a model-serving fleet from excessive request rate, bursts, and in-flight work while allocating capacity between consumers and models. It does not make authorization or billing decisions.

The desired outcome is one reusable policy decision on the request path. A model implementation does not contain limit logic and does not trust a caller-supplied property, model, or capacity-pool identifier. Adding a model normally adds routing metadata and policy; it does not add limiter code.

This separation follows the repository literature review's requirements for trusted client scope, per-client resource isolation, bounded queues, explicit capacity pools, and end-to-end tail-latency measurement. The exact peak QPS and burst shape remain unknown, so this proposal makes no replica, latency, availability, or cost claim.

2. Canonical terms

Term Meaning
Service Limit A versioned rule that bounds request units or in-flight units for matching trusted descriptors
Admission Request Trusted descriptors plus the weighted cost of proposed work
Admission Decision Allow or deny result, policy version, stable reason, retry delay, and optional lease
Descriptor A bounded key/value derived from authenticated identity or server-side routing, such as consumer, service, operation, or capacity_pool
Capacity Pool A separately allocated and isolated portion of serving capacity
Lease An expiring, idempotently releasable reservation of concurrency units
Policy Plane Versioning, validating, publishing, and rolling back Service Limits
Admission Plane Evaluating Admission Requests against hot policy and limit state

“Quota” is reserved for a long-horizon entitlement or commercial allowance. Quota accounting can reuse descriptors and weighted units later, but it should not be confused with overload protection: the consistency, reconciliation, and audit requirements are different.

3. System shape

flowchart LR
    Caller[Caller] --> Auth[Authentication and trusted scope]
    Auth --> Proxy[Gateway / Envoy]
    Proxy --> Local[Coarse local burst limit]
    Local --> Admission[Global admission plane]
    Admission --> State[(Partitioned hot state)]
    Policy[Versioned policy plane] --> Admission
    Admission -->|allow + lease| Router[Model router]
    Admission -->|deny + retry delay| Reject[429 / 503]
    Router --> Rec[Recommendation API]
    Router --> M1[Model API 1]
    Router --> MN[Model API N]
    Rec --> Release[Release lease]
    M1 --> Release
    MN --> Release
    Release --> Admission

The gateway is the preferred fleet-wide seam. The included ASGI adapter proves that the same core can protect the current FastAPI application without changing its routes, but embedding middleware in hundreds of model processes is not the target topology. Envoy supports a generic external global rate-limit interface based on hierarchical descriptors, plus a coarse local token bucket that can absorb bursts before they reach the global limiter (Envoy global rate limiting, Envoy RLS contract).

4. Deep module interface

The framework-independent module exposes two operations:

admit(descriptors, weighted_cost) -> decision
release(lease_id)                 -> idempotent completion

Everything else—rule matching, partition keys, burst refill, all-or-nothing evaluation, concurrency recovery, retry calculation, and policy version—is hidden behind this interface.

An Admission Request for this repository might contain descriptors derived by trusted ingress:

capacity_pool = shared-commerce
consumer      = property-42
service       = commerce-recommendations
operation     = serve-global-recommendations
cost          = 1

A generative endpoint might use a higher cost based on the requested output class. Cost classes must be selected server-side from bounded request properties; accepting arbitrary client-declared cost would make bypass trivial.

All matching rules apply. Examples include:

  • capacity-pool concurrency, partitioned by capacity_pool;
  • model request rate, partitioned by service;
  • consumer fairness, partitioned by consumer;
  • consumer/model rate, partitioned by both consumer and service;
  • operation-specific protection selected by service and operation.

If any rule denies, no rule consumes capacity. An allowed decision consumes token-bucket units and returns one lease covering every matched concurrency rule. Explicit release restores capacity immediately; TTL expiry recovers capacity after process death or a lost release.

The lease TTL must exceed the model route's hard execution timeout, or the adapter must support renewal. Otherwise valid long-running requests could lose their reservation while still consuming capacity.

5. Invariants

  1. Authentication and routing establish descriptors before admission; user query parameters are not a trusted source of limit identity.
  2. A request is either acquired under every applicable enforced rule or under none of them.
  3. Unmatched traffic fails closed by default so a new route cannot silently bypass protection.
  4. Rule and descriptor inputs are bounded before state access to contain per-request key input.
  5. Concurrency leases are expiring and release is idempotent.
  6. Interactive model calls are shed before expensive work; they are not placed into an unbounded queue that can outlive the caller's deadline.
  7. Policy versions are immutable. Publication is atomic, rollback is explicit, and decisions expose the active version for diagnosis.
  8. Metrics identify the rule and service but do not place raw consumer IDs into metric labels.
  9. Limit failures and model failures remain distinguishable in logs, metrics, and client responses.

6. Algorithms and response semantics

The initial algorithms are deliberately small:

  • Token bucket: finite burst capacity with a continuous refill rate. AWS API Gateway uses a token bucket for request throttling but describes its limits as best-effort targets, so it is useful as outer protection rather than the only source of strict per-consumer semantics (AWS HTTP API throttling).
  • Weighted concurrency lease: bounds expensive in-flight work even when latency increases. A request consuming four GPU work units can count as four without changing the interface.

An enforced capacity denial maps to HTTP 429 and includes Retry-After when the next useful retry can be calculated. Missing policy maps to 503, because retrying as though a consumer merely spent its allowance would hide a deployment defect. Standard rate-limit response fields are still an evolving HTTP specification; enable them only after selecting and contract-testing a concrete version. Retry-After is sufficient for the current slice.

7. Distributed state adapter

InMemoryLimitState is deterministic, thread-safe, and appropriate only for tests or one process. The production adapter must implement the same async state interface with one atomic read/decide/write operation. Redis scripts are a practical first candidate because the script keeps concurrent updates atomic and expiring keys bound state (Redis rate limiter).

For Redis Cluster, every rule participating in one admission must be colocated. The physical keys should include the request's capacity-pool hash tag, and each consumer must map to exactly one pool. Fleet capacity is allocated among pools by policy rather than updated through one cross-cluster hot key. A genuinely strict fleet-wide limit would require its own serialization point and should be introduced only with measured need.

The production adapter must also provide:

  • server-side time or a bounded clock-skew strategy;
  • atomic acquire across token and concurrency state;
  • lease TTL and idempotent release;
  • policy-version namespacing during rollout;
  • key expiry based on bucket refill and lease TTL;
  • request timeout shorter than the remaining model-request deadline;
  • overload behavior that does not turn the limiter into the fleet bottleneck;
  • deterministic contract tests shared with the in-memory adapter.

8. Policy plane

Policy starts as reviewed, immutable JSON deployed independently of model code. The included strict loader rejects unknown fields and converts the document into immutable domain rules; the example is config/serving_limits.example.json. Its numbers are illustrative, not capacity recommendations. Before a version can become active it must additionally pass maximum-cardinality checks, dry-run examples, and a shadow comparison against the current version.

The first rollout should support one active and one shadow version. Shadow evaluation records what would have been denied but never allocates concurrency or consumes rate units. A later policy store may add staged percentage rollout and automatic rollback, but neither belongs in the initial hot path without operating evidence.

9. Availability and overload behavior

The admission call has its own small deadline. For public or costly model routes, the safe default is fail closed when no trustworthy decision is available, with a separately configured emergency local allowance for known critical fallbacks. Health probes and internal lease release are explicitly exempt; arbitrary route exemptions are not.

Large consumers can move to dedicated capacity pools. Shared-pool policy should reserve enough headroom that one consumer's Black Friday burst cannot starve every other property. Pre-scaling and load testing remain necessary: limiting controls overload shape; it does not create capacity.

10. Telemetry and qualification

Record:

  • admitted and denied weighted units by service, operation, pool, rule, and reason;
  • admission latency and state-adapter errors;
  • active concurrency units, lease expiry, and late/idempotent releases;
  • token utilization and retry delay distributions;
  • unmatched-policy count;
  • model latency, timeouts, fallback use, and response validity beside limit outcomes.

Consumer-level investigation belongs in logs or traces with access controls, not unbounded metric labels. A decision ID should connect gateway, admission, model, and fallback observations.

Qualification needs ordinary mixed traffic, the largest consumer's burst, sustained expected peak, sudden pre-autoscale traffic, a state-shard failure, cold policy/cache, model latency inflation, and synchronized policy/model rollout. Measure p50/p95/p99, admitted throughput, denial correctness, state latency, lease recovery, fallback validity, and per-pool fairness together.

11. Delivery slices

Slice Deliverable Exit evidence
1 — Core Versioned hierarchical token and concurrency rules, weighted costs, expiring leases, optional ASGI adapter Unit and current API contract tests
2 — Policy config Strict serialized policy and generic/model-specific examples are present; shadow evaluation remains Config contract, dry-run, shadow, and rollback tests
3 — Distributed state Async Redis adapter with atomic script and key expiry Shared adapter suite, race tests, Redis failure tests
4 — Gateway Envoy RLS-compatible transport, trusted descriptor mapping, deadlines, exemptions End-to-end tests with recommendation API and a synthetic slow model
5 — Qualification Multi-pool load harness, dashboards, alerts, operational runbook Agreed traffic matrix passes stated SLOs

12. Decisions still needed before production

  • Peak and sustained QPS, burst duration, and geographic distribution.
  • Whether “Service Limit” also includes daily/monthly commercial quotas or spend budgets.
  • Required fairness: per consumer, per model, per operation, priority class, or a combination.
  • Weighted-cost source for GPU, token, batch-size, and response-size variation.
  • Maximum acceptable admission-plane latency and the end-to-end deadline budget.
  • Which routes fail closed, fail open under an emergency local allowance, or are exempt.
  • Whether a consumer can use more than one capacity pool or region simultaneously.
  • Policy ownership, approval, emergency override, and audit retention.

Until those are answered and the distributed slices are qualified, the current code is a maintainable foundation—not a claim of production-scale enforcement.